loss nan in VAE training

[shwj114514]

Thank you for your excellent work and the well-designed open-source code.

When I use your training code to train from scratch, I frequently encounter a situation where the loss becomes NaN after a certain number of training steps. Is this behavior expected?

This issue occurs when training both 44100 mono and stereo audio files. I have to repeat the training multiple times to ensure the loss remains stable.

I am using the stable audio 2.0 config.

[apply74]

I also encountered this problem. When I increased the model parameters, the training was unstable. Is it going to be solved?

[shwj114514]

I also encountered this problem. When I increased the model parameters, the training was unstable. Is it going to be solved?

I solved this problem by reducing the learning rates of both the generator and discriminator to 1/10 of their original values, and the training became stable.

[apply74]

I also tried reducing the learning rate. Although the training is stable, the reconstruction result will be very poor.

[fletcherist]

the same thing

[apply74]

the same thing I have solved the problem by increating the batch_size from 1 to 5.

[fletcherist]

the same thing

I have solved the problem by increating the batch_size from 1 to 5. @apply74 oh rly? let me try it but i think this batch size doesn't fit to gpu)) i'll message here after a try. thanks for your help very appreciate it

[fletcherist]

reducing the learning rates of both the generator and discriminator to 1/10 of their original values

this works

[nateraw]

You have to tune the learning rates. Higher batch size helps keep things stable.

Another tip is if you can't get large enough batch size, you can reduce the sample size which should free up enough memory to bump back up the batch size.

Hope this helps ❤️