search

open-mmlab / mmskeleton

A OpenMMLAB toolbox for human pose estimation, skeleton-based action recognition, and action synthesis.
Apache License 2.0
2.92k stars 1.03k forks source link

when training st-gcn the loss will turn into NAN #468

Open ppppppi1 opened 7 months ago

ppppppi1 commented 7 months ago

[02.06.24|21:04:38] Training epoch: 0 [02.06.24|21:13:26] Iter 0 Done. | loss: 1.4058 | lr: 0.010000 [02.06.24|21:13:31] mean_loss: nan [02.06.24|21:13:31] Time consumption: [02.06.24|21:13:31] Done. [02.06.24|21:13:31] Training epoch: 1 [02.06.24|21:13:40] mean_loss: nan [02.06.24|21:13:40] Time consumption: [02.06.24|21:13:40] Done. [02.06.24|21:13:40] The model has been saved as D:/dataset/2024.2.6_result2/epoch2_model.pt. [02.06.24|21:13:40] Eval epoch: 1 [02.06.24|21:13:47] mean_loss: nan [02.06.24|21:13:47] Top1: 25.00% [02.06.24|21:13:47] Top5: 100.00% [02.06.24|21:13:47] Done. [02.06.24|21:13:47] Training epoch: 2 [02.06.24|21:13:56] mean_loss: nan [02.06.24|21:13:56] Time consumption: [02.06.24|21:13:56] Done. [02.06.24|21:13:56] Training epoch: 3 [02.06.24|21:14:05] mean_loss: nan [02.06.24|21:14:05] Time consumption: [02.06.24|21:14:05] Done. [02.06.24|21:14:05] The model has been saved as D:/dataset/2024.2.6_result2/epoch4_model.pt. [02.06.24|21:14:05] Eval epoch: 3 [02.06.24|21:14:11] mean_loss: nan [02.06.24|21:14:11] Top1: 25.00% [02.06.24|21:14:11] Top5: 100.00% [02.06.24|21:14:11] Done. [02.06.24|21:14:11] Training epoch: 4

ppppppi1 commented 7 months ago

why and how