WARNING:root:NaN or Inf found in input tensor when training UNETR network.

[chuxiang93]

Describe the bug Nan popped up while I was training the UNETR network. WARNING:root:NaN or Inf found in input tensor.

To Reproduce Steps to reproduce the behavior:

1. Go to 'UNETR/BTCV'
2. Run commands: CUDA_VISIBLE_DEVICES=1,2,3 python main.py --distributed --feature_size=32 --batch_size=4 --logdir=unetr_test --optim_lr=1e-3 --lrschedule=warmup_cosine --infer_overlap=0.5 --save_checkpoint --data_dir=/data/BTCV/Abdomen/RawData/Training/ --workers=12

Screenshots

Environment (please complete the following information):

• OS
• Python version 3.8
• MONAI version [e.g. git commit hash]
• CUDA/cuDNN version CUDA11.1
• GPU models and configuration. V100

[ahatamiz]

Hi @mjq93

I am sorry you are facing this difficulty. We never ran into a NaN while training the UNETR. It is most likely caused by DiceCELoss as the values of denominator become very small.

One quick fix it to set squared_pred = False in DiceCELoss. This prevents the value of predictions, which are between 0 and 1, to further become smaller -- although we observed slightly better performance by squaring the predictions.

Another fix would be to pass a larger value such as smooth_dr = 1e-5 to prevent division by zero. I believe first fix will work out of the box and is more preferred.

Thanks

[liuyanice]

@mjq93 I have met the same issue that the loss became Nan when training. Both the two fixs are invalid, how do you solve them in the end?

[liuyanice]

I have met the same issue that the loss became Nan when training. Both the two fixs are invalid, how do you solve them in the end?

[liuyanice]

[liuyanice]

Thank you, I have solved it