Loss and grad_norm are 'nan' when training

wzzheng / TPVFormer

[CVPR 2023] An academic alternative to Tesla's occupancy network for autonomous driving.

https://wzzheng.net/TPVFormer/

Apache License 2.0

1.19k stars 107 forks source link

Closed Thecats-Jfm closed 1 year ago

Thecats-Jfm commented 1 year ago

Hi! I want to train the semantic occupancy prediciton task, but after I run the following command:

python train.py --py-config config/tpv04_occupancy.py --work-dir out/tpv_my_occupancy

I meet a nan and some strange grad_norm as follow，could you give me some suggestions?

Thecats-Jfm commented 1 year ago

I have found out this problem. There is something wrong with one of my gpus. >_<