search

modelscope / kws-training-suite

MIT License
73 stars 15 forks source link

你好，训练过程中，突然出现这种情况，就停止了训练是什么原因，没报任何错误 #10

Open 1101354051 opened 2 months ago

1101354051 commented 2 months ago

2024-05-09 23:59:29,749 - modelscope - INFO - Epoch: 0001/0005, batch: 0024/0300, loss: 0.6884 2024-05-09 23:59:37,052 - modelscope - INFO - Epoch: 0001/0005, batch: 0025/0300, loss: 0.6956 2024-05-09 23:59:42,912 - modelscope - INFO - Epoch: 0001/0005, batch: 0026/0300, loss: 0.6516 2024-05-09 23:59:49,405 - modelscope - INFO - Epoch: 0001/0005, batch: 0027/0300, loss: 0.6415 2024-05-09 23:59:56,251 - modelscope - INFO - Epoch: 0001/0005, batch: 0028/0300, loss: 0.5618 2024-05-10 00:00:05,053 - modelscope - INFO - Epoch: 0001/0005, batch: 0029/0300, loss: 0.5162 2024-05-10 00:00:10,203 - modelscope - INFO - Epoch: 0001/0005, batch: 0030/0300, loss: 0.4696 Killed