FedML-AI / FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
https://TensorOpera.ai
Apache License 2.0
4.11k stars 772 forks source link

Moving from storage object to presigned URLs #2154

Closed bhargav191098 closed 1 month ago

bhargav191098 commented 1 month ago

For security purposes, the backend no longer sends the api tokens to the python side. These changes directly impact the fedml storage module that relies upon the boto3 client to upload, download and delete dataset directories. These changes implement the upload and download of data using presigned urls generated from the backend. These sidestep getting the api key data directly on the python side.

Scope of Changes: