server aggregation about BN layers

[wintercfsad]

Hello, this is a great repo, I have a question about BN layers. Should the server aggregate 'bn1.running_mean', 'bn1.running_var', and 'bn1.num_batches_tracked'? Now it seems the first two are aggregated. And about 'bn1.num_batches_tracked', should it be reset to zero after aggregation for everyone?

[KarhouTam]

Should the server aggregate 'bn1.running_mean', 'bn1.running_var', and 'bn1.num_batches_tracked'? Now it seems the first two are aggregated.

Actually FL-bench doesn't aggregate these buffer tensors. These buffers are stored in self.clients_personal_model_params (server side variable) for each client.

[wintercfsad]

Thanks! But in this case, how does the running mean or var of the global model evolve? I suppose when evaluating the loss or accuracy over the whole dataset, the global model has to use a running mean or var.

[KarhouTam]

FL-bench doesn't set testset or valset globally. Each client holds their own test and val data. When evaluating global model, first, the global model would be sent to all FL clients and then client uses their local test/val data to evaluate the model and returns the statistics to the server. Server will aggregate those stats with weights and produce the final and single result (show on the log file and your terminal).

So back to your question. Server will send each client's personal BN buffers to them and each client will set it before model evaluation (in set_parameters()).

BTW, I'm developing a feature about these buffers. This feature will let user decide how to deal with these buffers. So if you care about buffers, stay tuned 😉.

[wintercfsad]

I see, I am curious about the influence of the aggregation with/out BN statistics. Thanks for your reply!

[KarhouTam]

I'm developing a feature about these buffers. This feature will let user decide how to deal with these buffers.

This feature is online.

This issue can be reopened if any issues are encountered while using this feature.