Open LikeStarting opened 4 months ago
It is suggested to use gradient monitoring and logging during training to identify the layer(s) or operation(s) causing the problem.
Thank you a lot. I will try~
I'm having the same problem, have you solved it?
Hi, When it comes to training step, the problem arises with 'grad_norm' of NaN. I used diffuSeq-v2 and used FP16 for GPU acceleration. Where is the problem and how can it be fixed.Thank you!
![image](https://github.com/Shark-NLP/DiffuSeq/assets/37132357/abc3fcd5-8ec3-48db-b83a-45ed0cb29be5)