The SyncBatchNorm function will stop, when one of multi-GPU run into "no valid points". How to solve it?

zju3dv / NeuralRecon

Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral

https://zju3dv.github.io/neuralrecon/

Apache License 2.0

2.03k stars 294 forks source link

Open shihfuchen opened 1 year ago

shihfuchen commented 1 year ago

The SyncBatchNorm function will stop when one of my GPUs runs into "no valid points". Does anyone have any idea? How to solve the problem?

I replaced the original BatchNorm function with SyncBatchNorm function. When I used the multi GPUs to train the model.