Closed kuz44ma69 closed 3 years ago
Thank you for your attention.
Indeed, as you noted, similar to BN, these statistics are computed at each GPU. They are not shared by all GPUs. Typically, this will not cause problems given a reasonable batch size.
Thank you for your reply. I understand.
I think that means and coVariances are calculated by each GPU in "update_CV()" https://github.com/blackfeather-wang/ISDA-for-Deep-Networks/blob/318c30976d0c412a7dd10250b0164beac6d4fbeb/Image%20classification%20on%20ImageNet/ISDA_imagenet.py#L13 and think that there is no code for integrating these parameters. How are these parameters shared by all gpus?