Averaging manner of Aggregators: over all or only sampled clients?

[yxdyc]

Hi, thank you for your interesting work!

It's nice to see the Aggregators support client sampling, which is natural and useful in cross-device FL scenarios. However, I found that only the global averaging of FFLAggregator is conducted over sampled clients (L717), while all the other aggregators perform the global model averaging over all the T clients, e.g., in CentralizedAggregator. As a result, for your implementation, the averaging of model $w_{t+1}$ emphasizes more the global model $w_t$ from the last communication round as the non-sampled clients have not trained $w_t$ on their local data.

Why did you choose this setting rather than averaging only over the sampled clients? Do the conclusions hold if the experiments are conducted by averaging over the sampled clients? Thanks

[omarfoq]

Hello,

This part of the code corresponds to the following aggregation scheme:

$w_{t+1} = \sum_{k \in \S_{k}} p_{k} w_{t+1}^{k} + \sum_{k \notin \S_{k}} p_{k} w_{t}$,

where $\S{k}$ is the set of sampled clients at iteration $k$, $p{k}$ is the weight associated with client $k$ and $w_{t+1}^{k}$ is the local model at client $k$ obtained after few gradient descent steps. This is the aggregation scheme used in (McMahan et al., 2016). Note how in (McMahan et al., 2016, Algorithm~1) the sum is over all the clients.

Other aggregation schemes could also be used, see for example (Li et al., 2019, Table~1). The convergence bounds will vary depending on the used one, but we also did part of our experiments (not published) with the aggregation scheme from (Li et al., 2019) with no significant difference on the performance of the final model. In practice both schemes will lead to the same result if tuned properly.

I hope this answers your question. Please let me know if I am missing something.