grad_norm: nan - Githubissues

HXMap / MapQR

[ECCV 2024] This is the official implementation of MapQR, an end-to-end method with an emphasis on enhancing query capabilities for constructing online vectorized maps.

https://arxiv.org/abs/2402.17430

MIT License

164 stars 8 forks source link

grad_norm: nan #10

Closed missTL closed 3 months ago

missTL commented 3 months ago

During training, grad_norm suddenly becomes nan

fishmarch commented 3 months ago

We also found this issue recently, but have not figured out where it comes from. It would happen one or two times during the whole training, thus we didn't notice before. We did some experiments and found it would appear when combing the preposed GKT-h and SGQ, although they both work well seperately.

fishmarch commented 1 month ago

Delete the following may solve this problem. https://github.com/HXMap/MapQR/blob/7dc705740fe026b7908b10e60fcd2846f9bb83d9/projects/mmdet3d_plugin/bevformer/modules/encoder.py#L149-L150