MarrNet2 training stops at Eval 3/1000

xiumingzhang / GenRe-ShapeHD

Code and Data Release for GenRe (NeurIPS 2018) and ShapeHD (ECCV 2018)

http://genre.csail.mit.edu/

294 stars 62 forks source link

MarrNet2 training stops at Eval 3/1000 #18

Closed rozentill closed 5 years ago

rozentill commented 5 years ago

Hi,

I am training MarrNet2 using default setting on shapenet chair dataset. It seems it always stops at Eval 3/1000 as shown below: Capture

Any idea what's going on?

Thanks!

ztzhang commented 5 years ago

hi, This might be an issue with Pytorch dataloader dead locking. Please see https://github.com/pytorch/pytorch/issues/1355. One possible fix is to turn off the multiprocessing/multithreading for numpy and opencv. This works for most of our cases.

rozentill commented 5 years ago

Works now, thanks!