When training an AttentionUnet, the model collapsed to zero outputting nan for Validation Loss, Training Loss, and Soft Dice:
2024-04-16 07:20:41.119 | INFO | __main__:run_training:251 - Epoch 995 Validation Loss: nan Hard Dice: 0.00
2024-04-16 07:20:42.117 | INFO | __main__:run_training:284 - Checkpoint saved at epoch 995
2024-04-16 07:21:20.168 | INFO | __main__:run_training:245 - Epoch 996/1000 Training Loss: nan Soft Dice: nan
2024-04-16 07:21:21.264 | INFO | __main__:run_training:284 - Checkpoint saved at epoch 996
2024-04-16 07:22:05.010 | INFO | __main__:run_training:245 - Epoch 997/1000 Training Loss: nan Soft Dice: nan
nan is problematic because it cannot be plotted, resulting in a figure without the epochs with nan:
We could replace nans in such cases with 0. @naga-karthik, what do you think?
When training an AttentionUnet, the model collapsed to zero outputting
nan
for Validation Loss, Training Loss, and Soft Dice:nan
is problematic because it cannot be plotted, resulting in a figure without the epochs withnan
:We could replace
nans
in such cases with0
. @naga-karthik, what do you think?