Loss turned `NaN` when custom training

lucidrains / gigagan-pytorch

Implementation of GigaGAN, new SOTA GAN out of Adobe. Culmination of nearly a decade of research into GANs

MIT License

1.81k stars 104 forks source link

Loss turned `NaN` when custom training #56

Open yasohasakii opened 3 months ago

yasohasakii commented 3 months ago

Hi , I met theloss nan issue too, and I added torch.autograd.set_detect_anomaly(True) then got RuntimeError: Function 'MulBackward0' returned nan values in its 1th output.. The gigagan-python of mine is 0.2.20, how can I fix this issue, please!

yasohasakii commented 3 months ago

I have set vision_aided_divergence_loss_weight=0. and still got wrong like below, please help me if you are convenient, thanks!