search

openseg-group / OCNet.pytorch

Please choose the openseg.pytorch project for the updated code that achieve SOTA on 6 benchmarks!
MIT License
812 stars 128 forks source link

When I run the Base_OC_Module at a certain iteration, it will stop. #16

Open yu-changqian opened 6 years ago

yu-changqian commented 6 years ago

Pytorch version: 0.4.1 GPU benchmark: 8 x 1080 Ti CUDA version: 9.0

Experiment config: batch size: 16 num workers: 16 input size: 480x480

When I use the Base_OC_Module on the ADE20K dataset, my experiment will stop at a certain iteration without other notion output. And the utilization rate of GPU will drop to 0. Did you have the similar experience?

PkuRainBow commented 6 years ago

Sorry, I think it can be a bug of inplace-abn. Please refer to https://github.com/mapillary/inplace_abn/issues/58.

We use 4 x P100 for all the experiments.

yu-changqian commented 6 years ago

Yes. When I use 4 x 1080Ti, all of it becomes normal. This should be a bug of inplace-abn. Thank you.

KeyKy commented 6 years ago

I get the same problem and it only work for 4 gpus.