search

FedML-AI / FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
https://TensorOpera.ai
Apache License 2.0
4.19k stars 787 forks source link

Upload Byte data #2196

Open bhargav191098 opened 5 months ago

bhargav191098 commented 5 months ago

This PR will enable uploading byte data through fedml storage upload command.

alaydshah commented 5 months ago

To summarize, we should try to make following modifications:

  1. Figure out a way to eliminate the byte data type flag and just recognize the type internally.
  2. Restructure code a bit to avoid long functions with many if-else conditions. We can make it more readable by having more manageable sub functions for different data type and centralize the common logic in main function and call respective sub-function based on the detected data type
alaydshah commented 5 months ago

Please remember to always propogate the newly added variables all the way up to CLIs and APIs. This is the entrypoint, and we don't have the byte_data property wired in here. Always good to test it from end user perspective so we can catch such obvious slip-ups.

image