search

milesial / Pytorch-UNet

PyTorch implementation of the U-Net for image semantic segmentation with high quality images
GNU General Public License v3.0
9.32k stars 2.52k forks source link

Predicted output is black. #506

Open akshay9396 opened 3 months ago

akshay9396 commented 3 months ago

I trained the model on Carvana dataset. Below is the screen shot of training. There i you can clearly see that validation Dice score is constant for 5 epochs and loss is "nan". I used one checkpoint.pth for predication and i got black output. could you please help me to resolved this issue. image Capture

Feng2100 commented 2 months ago

Have you solved it? I'm having the exact same problem

chenshans commented 2 months ago

I'm having the exact same problem，too

berda-ak commented 1 month ago

I'm having the exact same problem，too

CZG0712 commented 1 month ago

I'm having the exact same problem，too

taipain commented 2 weeks ago

I trained the model on Carvana dataset. Below is the screen shot of training. There i you can clearly see that validation Dice score is constant for 5 epochs and loss is "nan". I used one checkpoint.pth for predication and i got black output. could you please help me to resolved this issue. image Capture

Probably the reason is that torch.float16 is used in autocast() when using AMP. So, change torch.float16 to torch.bfloat16. But, if you wanna use torch.float16, you add grad_scaler in init_scale=4096． I solved this issue. I referred to the following sites． This is my first time commenting on github, so sorry if there is something wrong and my poor English． https://qiita.com/takeuchiseijin/items/909c48b57127a37fbd12 https://qiita.com/bowdbeg/items/71c62cf8ef891d164ecd