search

sp-uhh / sgmse

Score-based Generative Models (Diffusion Models) for Speech Enhancement and Dereverberation
MIT License
454 stars 69 forks source link

Why i got Nan when i train the ncsnpp? #30

Closed manmushanhe closed 2 months ago

manmushanhe commented 1 year ago

I change the nfft to 510, and backbone to ncsnpp, but i got nan in the training. Could you help me ?

zoupengpeng128 commented 10 months ago

I change the nfft to 510, and backbone to ncsnpp, but i got nan in the training. Could you help me ?

I have the same problem. Have you solved it?

manmushanhe commented 10 months ago

I change the nfft to 510, and backbone to ncsnpp, but i got nan in the training. Could you help me ?

I have the same problem. Have you solved it?

No, I didn't train successfully

julius-richter commented 7 months ago

We have never encountered a problem like this before. Maybe it has something to do with the internal precision of matrix multiplications in PyTorch.

In any case, we have just updated the code to PyTorch 2.X and Lightning 2.X to ensure support for newer GPUs. Try running the code again after you have created a virtual environment with the new requirements.