Open author31 opened 1 year ago
Hi! Thanks for your attention.
Could you please specify where NaN is produced in your code?
Furthermore, can you briefly describe your plan to change the lr scheduler?
In my cases, the NaN problem is because learning rates to large even is 3e-6. So, I added the lr scheduler to do the warmcos scheduling style which like this
Hi! I really like the your work and I tried that on my RTX 3060 12G, but it kept going NaN so I added lr scheduler and refactor the whole codebase. Here is the link https://github.com/author31/scm_nbdev. Once again, thanks for your work, keep doing.