Encountered with Nan loss when training classification task

OpenGVLab / DCNv4

[CVPR 2024] Deformable Convolution v4

https://arxiv.org/pdf/2401.06197.pdf

MIT License

516 stars 27 forks source link

Open jin-yc10 opened 8 months ago

jin-yc10 commented 8 months ago

Encountered with Nan loss when training classification task on imagenet.

Cuda: 11.6 Torch: 1.12.1+cu116 Timm: 0.6.11

Any help is appreciated! Thanks

Ckkxx commented 7 months ago

Yes, nan values appear when using DCNV4 instead of the normal volume downsampling operation, which is the problem I encountered

fuyannnnn commented 6 months ago

How did you solve this problem?

87831743Sakura commented 5 months ago

I also encountered this problem, is there a solution?