search

starrytong / SCNet

MIT License
59 stars 5 forks source link

Large config AMP training: loss=nan, grad=nan, nsdr=nan #14

Closed ari-ruokamo closed 3 months ago

ari-ruokamo commented 3 months ago

What might cause the training to go into this state pretty quickly after few epochs?

2024-08-25 14:12:43,465 - INFO - Learning rate adjusted to 0.0005
2024-08-25 14:12:43,466 - INFO - ----------------------------------------------------------------------
2024-08-25 14:12:43,466 - INFO - Training Epoch 4 ...
2024-08-25 14:48:04,998 - INFO - Train Summary | Epoch 4 | Loss=nan | Grad=nan
2024-08-25 14:48:04,999 - INFO - ----------------------------------------------------------------------
2024-08-25 14:48:04,999 - INFO - Cross validation...
2024-08-25 14:58:37,505 - INFO - Valid Summary | Epoch 4 | Loss=0.1838 | Nsdr=9.165 | Nsdr_vocals=6.553 | Nsdr_other=11.777
2024-08-25 14:58:37,506 - INFO - New best valid nsdr 9.1648
2024-08-25 14:58:39,879 - INFO - Learning rate adjusted to 0.0005
2024-08-25 14:58:39,880 - INFO - ----------------------------------------------------------------------
2024-08-25 14:58:39,880 - INFO - Training Epoch 5 ...
2024-08-25 15:33:53,573 - INFO - Train Summary | Epoch 5 | Loss=nan | Grad=nan
2024-08-25 15:33:53,574 - INFO - ----------------------------------------------------------------------
2024-08-25 15:33:53,574 - INFO - Cross validation...
2024-08-25 15:44:56,974 - INFO - Valid Summary | Epoch 5 | Loss=nan | Nsdr=nan | Nsdr_vocals=nan | Nsdr_other=nan
2024-08-25 15:44:59,222 - INFO - Learning rate adjusted to 0.0005
2024-08-25 15:44:59,223 - INFO - ----------------------------------------------------------------------
2024-08-25 19:44:10,499 - INFO - Training Epoch 6 ...
2024-08-25 20:17:51,896 - INFO - Train Summary | Epoch 6 | Loss=nan | Grad=nan
2024-08-25 20:17:51,896 - INFO - ----------------------------------------------------------------------
2024-08-25 20:17:51,896 - INFO - Cross validation...
2024-08-25 20:28:47,714 - INFO - Valid Summary | Epoch 6 | Loss=nan | Nsdr=nan | Nsdr_vocals=nan | Nsdr_other=nan
2024-08-25 20:28:50,124 - INFO - Learning rate adjusted to 0.0005
starrytong commented 3 months ago

Perhaps the learning rate could be reduced to 0.0003 or lower.

ari-ruokamo commented 3 months ago

Thanks.

Yes indeed lowering LR helps, might need to go even lower as convergence looks yet a bit too fast, maybe. Grad persists at inf but I guess it is normal as stated here https://github.com/starrytong/SCNet/issues/11.