search

modelscope / FunCodec

FunCodec is a research-oriented toolkit for audio quantization and downstream applications, such as text-to-speech synthesis, music generation et.al.
https://funcodec.github.io/
MIT License
370 stars 30 forks source link

Model sometimes stops training due to nan inputs #47

Closed hanshounsu closed 3 months ago

hanshounsu commented 4 months ago

The model sometimes prints out "got nan" on the log file, and gets exited during training.

Seems there has been nan values occurring rarely, in the Conv1d layers.

Can you provide any insights on how you resolved the issue..?

Thanks for the good work!

ZhihaoDU commented 4 months ago

please provide the following information:

  1. which model
  2. what kind of data
  3. snapshot of errors
  4. system information
hanshounsu commented 3 months ago

I assume this is due to my mistake. Thanks!