search

LabeliaLabs / distributed-learning-contributivity

Simulate collaborative ML scenarios, experiment multi-partner learning approaches and measure respective contributions of different datasets to model performance.
https://www.labelia.org
Apache License 2.0
56 stars 12 forks source link

re-build the model reset the optimizer #307

Closed arthurPignet closed 3 years ago

arthurPignet commented 3 years ago

When performing mpl, the model is erased and rebuilt every time we need it. Thus the model is recompiled, and the optimizer is reset, so the learning rate is not decreasing.

arthurPignet commented 3 years ago

On the same subject: Adam or RMS opts use previous gradients in the descente direction. How does it work in fedavg ? For now on the optimizer is reset every time we need it, so this issue is hidden (we use this kind of gradient memory in the .fit() only, with gradient-pass-per-update batch). In fedavg, the weights are averaged, so how the optimizer reacts to this ?