search

wac81 / vits_chinese

MIT License
48 stars 17 forks source link

Loss中有nan和inf #2

Closed Vimos closed 2 years ago

Vimos commented 2 years ago

目前训练时，losses = [loss_disc, loss_gen, loss_fm, loss_mel, loss_dur, loss_kl] 出现了大量的nan和inf，请问这个属于正常的吗？

INFO:ljs_base:====> Epoch: 36
INFO:ljs_base:Train Epoch: 37 [22%]
INFO:ljs_base:[nan, nan, nan, 83.04302978515625, 0.9958622455596924, inf, 28400, 0.0001990770782180657]
INFO:ljs_base:Train Epoch: 37 [48%]
INFO:ljs_base:[nan, nan, nan, 84.82212829589844, 0.9872239828109741, inf, 28600, 0.0001990770782180657]
INFO:ljs_base:Train Epoch: 37 [73%]
INFO:ljs_base:[nan, nan, nan, 88.02903747558594, 1.0118815898895264, inf, 28800, 0.0001990770782180657]
INFO:ljs_base:Train Epoch: 37 [99%]
INFO:ljs_base:[nan, nan, nan, 91.33515167236328, 0.9768699407577515, inf, 29000, 0.0001990770782180657]
Vimos commented 2 years ago

关闭原因：我自己使用的https://github.com/resemble-ai/monotonic_align和自带的计算结果不一致，改回去就不会inf了。