search

vacancy / Synchronized-BatchNorm-PyTorch

Synchronized Batch Normalization implementation in PyTorch.
MIT License
1.5k stars 189 forks source link

The mean is not averaged realy. #4

Closed acgtyrant closed 6 years ago

acgtyrant commented 6 years ago

I noticed that you divide the sum by it sum_size, however the sum_size is not multiplied by the device_ids: https://github.com/vacancy/Synchronized-BatchNorm-PyTorch/blob/81d983920c828cb50398bf6e4f6c9044f85fd568/sync_batchnorm/batchnorm.py#L117

vacancy commented 6 years ago

Thank you for your attention! Please see the following lines for your information. We sum up the size of the tensors distributed on different devices at L101.

https://github.com/vacancy/Synchronized-BatchNorm-PyTorch/blob/81d983920c828cb50398bf6e4f6c9044f85fd568/sync_batchnorm/batchnorm.py#L101-L103

Please feel free to reopen this issue if your question remains.