search

FedML-AI / FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
https://TensorOpera.ai
Apache License 2.0
4.2k stars 787 forks source link

The Reason of Changing Total Number of Clients #95

Closed hiyuchang closed 3 years ago

hiyuchang commented 3 years ago

Thank you for your nice codes.

I have a question about main_fedavg.py#L115:

args.client_num_in_total = client_num

While loading data, why should we change args.client_num_in_total = client_numinto 1000? What is the intuition behind this?

chaoyanghe commented 3 years ago

for cross-device FL, the client number is normally much larger than the worker number (the processes that execute the local SGD), so as the original FedAvg paper suggested, we need to do a client sampling to tackle the scalability issue. So here we set the client number equal to the number in the dataset and then do client sampling: https://github.com/FedML-AI/FedML/blob/79825b44f3860e6b4aa077326a964c585c8f1e9b/fedml_api/distributed/fedavg/FedAvgServerManager.py#L66

Let's think about a concrete example in practice. We may have 1 million users in our cloud service, but you cannot ask all of them to join a round of federated training because of straggler, device failure, time zone gap, out of battery, user rejection, etc. So we sample those users who have a reliable connection and high intention to help the training. As for research, we can use random sampling to mimic this practice.

hiyuchang commented 3 years ago

Thank you for your clear explanation.