search

ivadomed / model-seg-dcm

Segmentation of lesions on MRI scans in patients with Degenerative Cervical Myelopathy (DCM)
MIT License
3 stars 0 forks source link

Replace `nan` with `0` to support loss plotting #13

Open valosekj opened 4 months ago

valosekj commented 4 months ago

When training an AttentionUnet, the model collapsed to zero outputting nan for Validation Loss, Training Loss, and Soft Dice:

2024-04-16 07:20:41.119 | INFO     | __main__:run_training:251 - Epoch 995   Validation Loss: nan    Hard Dice: 0.00
2024-04-16 07:20:42.117 | INFO     | __main__:run_training:284 - Checkpoint saved at epoch 995
2024-04-16 07:21:20.168 | INFO     | __main__:run_training:245 - Epoch 996/1000      Training Loss: nan      Soft Dice: nan
2024-04-16 07:21:21.264 | INFO     | __main__:run_training:284 - Checkpoint saved at epoch 996
2024-04-16 07:22:05.010 | INFO     | __main__:run_training:245 - Epoch 997/1000      Training Loss: nan      Soft Dice: nan

nan is problematic because it cannot be plotted, resulting in a figure without the epochs with nan:

loss_plots

We could replace nans in such cases with 0. @naga-karthik, what do you think?