search

juglab / n2v

This is the implementation of Noise2Void training.
Other
385 stars 107 forks source link

making patch size smaller in the given 3D example notebook lead to nan loss #114

Closed tonyPan123 closed 2 years ago

tonyPan123 commented 2 years ago

I have runned the 3D jupyter notebook again on my PC and it is ok with the default 32x64x64(ZXY) patch size. However, if I changed it into 8x32x16 or even smaller. The nan loss will appear on the first or zero batch of first epoch and then the training will be stopped. What causes this? The second question is that when I tried to save my 32x64x32 patch-size model, it fails the assertion which said that the dim(x) and dim(y) must equal. I am wondering how to fix this.

tibuch commented 2 years ago

Hi @tonyPan123,

The U-Net applies a stack of pooling and convolution layers which requires the input to have a certain size. If you go below this minimal size the operations run out of pixels to work on which could lead to the NaN loss.

Setting the patch size to 32x64x32 (ZYX) results in non-square patches in XY which we can not handle at the time. You could use 32x32x32 (ZYX) patches.

If you are running out of space on your GPU I would also recommend to reduce the batch_size.

tonyPan123 commented 2 years ago

Thanks a lot!