search

octree-nn / ocnn-pytorch

Octree-based Sparse Convolutional Neural Networks
MIT License
150 stars 16 forks source link

About the choice of momentum and eps for “batch normalization”. #26

Closed qixuema closed 1 year ago

qixuema commented 1 year ago

Hello, Professor Wang, I noticed that in the code, the comments suggest using 0.1 and 1e-5 for certain parameters in PyTorch. However, it seems that in the actual usage, values like 0.01 and 0.001 are used. I was wondering about the reasoning behind this difference?

image

https://github.com/octree-nn/ocnn-pytorch/blob/36072d8e8dc1ed803c77c34a3806a5c5dfb4af7a/ocnn/modules/modules.py#L17-L18

wang-ps commented 1 year ago

Originally, ocnn was implemented with TensorFlow and Caffe. When I tried to implement ocnn on Pytorch, I used the default values of batch norm according to my initial TensorFlow code for reference.

According to my experience, the network is generally robust to these values. You can safely choose either setting as you desire. However, for the released trained weights, I used the parameter setting in Line 17.

qixuema commented 1 year ago

Cool! If the model proves to be robust with respect to these parameters, it would be excellent for the training process. Thank you very much for your assistance.