search

vacancy / Synchronized-BatchNorm-PyTorch

Synchronized Batch Normalization implementation in PyTorch.
MIT License
1.5k stars 189 forks source link

Can one past the test ? #17

Closed chenchr closed 5 years ago

chenchr commented 6 years ago

Hello. Thanks for the awesome code. I read this code base and the adaptation from zhanghang1989 . His code customs two operators for saving gpu memory. I've just combined the cuda extension for PyTorch 0.4.1. Although I have past the gradcheck of the operator bn and the opeartor sum_square, and for each operator I've compared the operator ouput with the ouput from imperative implementation using PyTorch, I can not past the test case provided here... May you have any suggestion about the numeric stability ?

vacancy commented 5 years ago

We do find that our implementation has some numeric issue. It seems to be less numerically stable than the pytorch built-in BN. But in our experiments, empirically, it does not hurt the performance.

Closing for now. Feel free to reopen if you have further questions.