Training with clipped_grad_norm value of NaN

NVlabs / ODISE

Official PyTorch implementation of ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models [CVPR 2023 Highlight]

https://arxiv.org/abs/2303.04803

Other

845 stars 45 forks source link

Training with clipped_grad_norm value of NaN #38

Open jim464190755 opened 9 months ago

jim464190755 commented 9 months ago

When I train on coco dataset, I find that the clipped_grad_norm value is NaN and total_loss is difficult to decrease, what is wrong or what might be the reason for this？