search

starrytong / SCNet

MIT License
60 stars 7 forks source link

Grad=inf and Loss=nan with mixed precision #11

Closed lyndonlauder closed 4 months ago

lyndonlauder commented 4 months ago

Hello, I am training the large SCNet model with mixed precision. The grad is very high and sometimes inf, I also received nan loss at epoch 21. Do you have any advice for tracking/improving the stability of the model with mixed precision? Please can you share your training log for large model?

Here is my training log

2024-07-26 16:21:42,549 - INFO - Training Epoch 1 ... 2024-07-26 17:06:38,010 - INFO - Train Summary | Epoch 1 | Loss=0.2242 | Grad=9164.2812 2024-07-26 17:06:38,023 - INFO - ---------------------------------------------------------------------- 2024-07-26 17:06:38,024 - INFO - Cross validation... 2024-07-26 17:06:42,187 - INFO - Valid Summary | Epoch 1 | Loss=0.2084 | Nsdr=6.212 2024-07-26 17:06:42,187 - INFO - New best valid nsdr 6.2124 2024-07-26 17:06:43,442 - INFO - ---------------------------------------------------------------------- 2024-07-26 17:06:43,442 - INFO - Training Epoch 2 ... 2024-07-26 17:51:41,888 - INFO - Train Summary | Epoch 2 | Loss=0.1857 | Grad=16170.3185 2024-07-26 17:51:41,896 - INFO - ---------------------------------------------------------------------- 2024-07-26 17:51:41,897 - INFO - Cross validation... 2024-07-26 17:51:45,884 - INFO - Valid Summary | Epoch 2 | Loss=0.1941 | Nsdr=6.779 2024-07-26 17:51:45,884 - INFO - New best valid nsdr 6.7788 2024-07-26 17:51:47,187 - INFO - ---------------------------------------------------------------------- 2024-07-26 17:51:47,187 - INFO - Training Epoch 3 ... 2024-07-26 18:36:47,348 - INFO - Train Summary | Epoch 3 | Loss=0.1673 | Grad=inf 2024-07-26 18:36:47,361 - INFO - ---------------------------------------------------------------------- 2024-07-26 18:36:47,361 - INFO - Cross validation... 2024-07-26 18:36:51,241 - INFO - Valid Summary | Epoch 3 | Loss=0.1839 | Nsdr=7.248 2024-07-26 18:36:51,242 - INFO - New best valid nsdr 7.2479 2024-07-26 18:36:52,554 - INFO - ---------------------------------------------------------------------- 2024-07-26 18:36:52,555 - INFO - Training Epoch 4 ... 2024-07-26 19:21:42,165 - INFO - Train Summary | Epoch 4 | Loss=0.1549 | Grad=12953.1609 2024-07-26 19:21:42,176 - INFO - ---------------------------------------------------------------------- 2024-07-26 19:21:42,176 - INFO - Cross validation... 2024-07-26 19:21:45,848 - INFO - Valid Summary | Epoch 4 | Loss=0.1763 | Nsdr=7.596 2024-07-26 19:21:45,848 - INFO - New best valid nsdr 7.5961 2024-07-26 19:21:47,106 - INFO - ---------------------------------------------------------------------- 2024-07-26 19:21:47,106 - INFO - Training Epoch 5 ... 2024-07-26 20:06:40,924 - INFO - Train Summary | Epoch 5 | Loss=0.1476 | Grad=26650.9245 2024-07-26 20:06:40,937 - INFO - ---------------------------------------------------------------------- 2024-07-26 20:06:40,937 - INFO - Cross validation... 2024-07-26 20:06:44,503 - INFO - Valid Summary | Epoch 5 | Loss=0.1715 | Nsdr=7.834 2024-07-26 20:06:44,504 - INFO - New best valid nsdr 7.8343 2024-07-26 20:06:45,751 - INFO - ---------------------------------------------------------------------- 2024-07-26 20:06:45,751 - INFO - Training Epoch 6 ... 2024-07-26 20:51:41,458 - INFO - Train Summary | Epoch 6 | Loss=0.1425 | Grad=inf 2024-07-26 20:51:41,477 - INFO - ---------------------------------------------------------------------- 2024-07-26 20:51:41,477 - INFO - Cross validation... 2024-07-26 20:51:45,154 - INFO - Valid Summary | Epoch 6 | Loss=0.1678 | Nsdr=8.023 2024-07-26 20:51:45,154 - INFO - New best valid nsdr 8.0229 2024-07-26 20:51:46,401 - INFO - ---------------------------------------------------------------------- 2024-07-26 20:51:46,402 - INFO - Training Epoch 7 ... 2024-07-26 21:36:43,013 - INFO - Train Summary | Epoch 7 | Loss=0.1387 | Grad=29771.7333 2024-07-26 21:36:43,029 - INFO - ---------------------------------------------------------------------- 2024-07-26 21:36:43,029 - INFO - Cross validation... 2024-07-26 21:36:46,834 - INFO - Valid Summary | Epoch 7 | Loss=0.1655 | Nsdr=8.144 2024-07-26 21:36:46,835 - INFO - New best valid nsdr 8.1442 2024-07-26 21:36:48,284 - INFO - ---------------------------------------------------------------------- 2024-07-26 22:03:26,019 - INFO - Training Epoch 8 ... 2024-07-27 01:02:26,106 - INFO - Train Summary | Epoch 8 | Loss=0.1307 | Grad=inf 2024-07-27 01:02:26,118 - INFO - ---------------------------------------------------------------------- 2024-07-27 01:02:26,118 - INFO - Cross validation... 2024-07-27 01:02:30,236 - INFO - Valid Summary | Epoch 8 | Loss=0.1621 | Nsdr=8.327 2024-07-27 01:02:30,237 - INFO - New best valid nsdr 8.3269 2024-07-27 01:02:31,651 - INFO - ---------------------------------------------------------------------- 2024-07-27 01:02:31,651 - INFO - Training Epoch 9 ... 2024-07-27 04:00:54,322 - INFO - Train Summary | Epoch 9 | Loss=0.1242 | Grad=inf 2024-07-27 04:00:54,338 - INFO - ---------------------------------------------------------------------- 2024-07-27 04:00:54,338 - INFO - Cross validation... 2024-07-27 04:00:58,343 - INFO - Valid Summary | Epoch 9 | Loss=0.1583 | Nsdr=8.527 2024-07-27 04:00:58,344 - INFO - New best valid nsdr 8.5270 2024-07-27 04:00:59,685 - INFO - ---------------------------------------------------------------------- 2024-07-27 06:04:22,210 - INFO - Training Epoch 10 ... 2024-07-27 09:03:22,470 - INFO - Train Summary | Epoch 10 | Loss=0.1197 | Grad=17715.8152 2024-07-27 09:03:22,483 - INFO - ---------------------------------------------------------------------- 2024-07-27 09:03:22,483 - INFO - Cross validation... 2024-07-27 09:03:26,656 - INFO - Valid Summary | Epoch 10 | Loss=0.1577 | Nsdr=8.567 2024-07-27 09:03:26,657 - INFO - New best valid nsdr 8.5673 2024-07-27 09:03:28,050 - INFO - ---------------------------------------------------------------------- 2024-07-27 09:03:28,050 - INFO - Training Epoch 11 ... 2024-07-27 12:02:17,179 - INFO - Train Summary | Epoch 11 | Loss=0.1182 | Grad=inf 2024-07-27 12:02:17,197 - INFO - ---------------------------------------------------------------------- 2024-07-27 12:02:17,198 - INFO - Cross validation... 2024-07-27 12:02:21,135 - INFO - Valid Summary | Epoch 11 | Loss=0.1578 | Nsdr=8.562 2024-07-27 12:02:22,517 - INFO - ---------------------------------------------------------------------- 2024-07-27 13:13:30,290 - INFO - Training Epoch 12 ... 2024-07-27 15:03:27,313 - INFO - Train Summary | Epoch 12 | Loss=0.1152 | Grad=inf 2024-07-27 15:03:27,322 - INFO - ---------------------------------------------------------------------- 2024-07-27 15:03:27,322 - INFO - Cross validation... 2024-07-27 15:03:31,182 - INFO - Valid Summary | Epoch 12 | Loss=0.1573 | Nsdr=8.583 2024-07-27 15:03:31,183 - INFO - New best valid nsdr 8.5831 2024-07-27 15:03:32,594 - INFO - ---------------------------------------------------------------------- 2024-07-27 15:03:32,594 - INFO - Training Epoch 13 ... 2024-07-27 16:53:30,252 - INFO - Train Summary | Epoch 13 | Loss=0.1122 | Grad=inf 2024-07-27 16:53:30,259 - INFO - ---------------------------------------------------------------------- 2024-07-27 16:53:30,259 - INFO - Cross validation... 2024-07-27 16:53:33,852 - INFO - Valid Summary | Epoch 13 | Loss=0.1550 | Nsdr=8.715 2024-07-27 16:53:33,852 - INFO - New best valid nsdr 8.7150 2024-07-27 16:53:35,272 - INFO - ---------------------------------------------------------------------- 2024-07-27 17:11:35,025 - INFO - Training Epoch 14 ... 2024-07-27 19:03:05,821 - INFO - Train Summary | Epoch 14 | Loss=0.1109 | Grad=inf 2024-07-27 19:03:05,838 - INFO - ---------------------------------------------------------------------- 2024-07-27 19:03:05,838 - INFO - Cross validation... 2024-07-27 19:03:09,887 - INFO - Valid Summary | Epoch 14 | Loss=0.1544 | Nsdr=8.750 2024-07-27 19:03:09,887 - INFO - New best valid nsdr 8.7502 2024-07-27 19:03:11,406 - INFO - ---------------------------------------------------------------------- 2024-07-27 19:03:11,406 - INFO - Training Epoch 15 ... 2024-07-27 20:54:43,432 - INFO - Train Summary | Epoch 15 | Loss=0.1105 | Grad=inf 2024-07-27 20:54:43,444 - INFO - ---------------------------------------------------------------------- 2024-07-27 20:54:43,444 - INFO - Cross validation... 2024-07-27 20:54:47,335 - INFO - Valid Summary | Epoch 15 | Loss=0.1552 | Nsdr=8.708 2024-07-27 20:54:48,683 - INFO - ---------------------------------------------------------------------- 2024-07-27 20:54:48,683 - INFO - Training Epoch 16 ... 2024-07-27 22:46:22,225 - INFO - Train Summary | Epoch 16 | Loss=0.1073 | Grad=inf 2024-07-27 22:46:22,246 - INFO - ---------------------------------------------------------------------- 2024-07-27 22:46:22,246 - INFO - Cross validation... 2024-07-27 22:46:25,964 - INFO - Valid Summary | Epoch 16 | Loss=0.1521 | Nsdr=8.880 2024-07-27 22:46:25,964 - INFO - New best valid nsdr 8.8801 2024-07-27 22:46:27,411 - INFO - ---------------------------------------------------------------------- 2024-07-27 22:57:00,857 - INFO - Training Epoch 17 ... 2024-07-28 00:47:55,631 - INFO - Train Summary | Epoch 17 | Loss=0.1046 | Grad=7684.2045 2024-07-28 00:47:55,648 - INFO - ---------------------------------------------------------------------- 2024-07-28 00:47:55,648 - INFO - Cross validation... 2024-07-28 00:47:59,791 - INFO - Valid Summary | Epoch 17 | Loss=0.1519 | Nsdr=8.886 2024-07-28 00:47:59,792 - INFO - New best valid nsdr 8.8855 2024-07-28 00:48:01,284 - INFO - ---------------------------------------------------------------------- 2024-07-28 00:48:01,284 - INFO - Training Epoch 18 ... 2024-07-28 02:39:08,556 - INFO - Train Summary | Epoch 18 | Loss=0.1067 | Grad=inf 2024-07-28 02:39:08,567 - INFO - ---------------------------------------------------------------------- 2024-07-28 02:39:08,568 - INFO - Cross validation... 2024-07-28 02:39:12,635 - INFO - Valid Summary | Epoch 18 | Loss=0.1531 | Nsdr=8.814 2024-07-28 02:39:14,014 - INFO - ---------------------------------------------------------------------- 2024-07-28 02:39:14,014 - INFO - Training Epoch 19 ... 2024-07-28 04:30:12,443 - INFO - Train Summary | Epoch 19 | Loss=0.1058 | Grad=inf 2024-07-28 04:30:12,455 - INFO - ---------------------------------------------------------------------- 2024-07-28 04:30:12,455 - INFO - Cross validation... 2024-07-28 04:30:16,408 - INFO - Valid Summary | Epoch 19 | Loss=0.1527 | Nsdr=8.843 2024-07-28 04:30:17,780 - INFO - ---------------------------------------------------------------------- 2024-07-28 09:09:54,119 - INFO - Training Epoch 20 ... 2024-07-28 12:06:53,195 - INFO - Train Summary | Epoch 20 | Loss=0.1058 | Grad=7435.8403 2024-07-28 12:06:53,212 - INFO - ---------------------------------------------------------------------- 2024-07-28 12:06:53,213 - INFO - Cross validation... 2024-07-28 12:06:57,651 - INFO - Valid Summary | Epoch 20 | Loss=0.1524 | Nsdr=8.858 2024-07-28 12:06:58,989 - INFO - ---------------------------------------------------------------------- 2024-07-28 12:06:58,989 - INFO - Training Epoch 21 ... 2024-07-28 15:03:59,552 - INFO - Train Summary | Epoch 21 | Loss=nan | Grad=nan 2024-07-28 15:03:59,618 - INFO - ---------------------------------------------------------------------- 2024-07-28 15:03:59,618 - INFO - Cross validation... 2024-07-28 15:04:03,674 - INFO - Valid Summary | Epoch 21 | Loss=0.1597 | Nsdr=8.439 2024-07-28 15:04:05,113 - INFO - ----------------------------------------------------------------------

starrytong commented 4 months ago

I have lost my log. if I have enough GPUs, I will retrain it. You might need to lower the initial learning rate to 0.0003.

lyndonlauder commented 4 months ago

I have lost my log. if I have enough GPUs, I will retrain it. You might need to lower the initial learning rate to 0.0003.

Thank you, do I need to do anything different to track the grad with mixed precision? The number sometimes shows inf or big numbers like 29771, this doesnt happen with fp32.

starrytong commented 4 months ago

It is normal for the gradient values to be large. I have encountered inf values before, but as long as the loss is normal, there is no issue.