Open gggxxx1234 opened 1 year ago
training: 0%| | 0/200000 [00:00<?, ?it/s][val] loss:2.060e-01 training: 1%| | 1998/200000 [01:59<3:14:20, 16.98it/s, it=2000, loss=0.020] training: 1%| | 1998/200000 [02:18<3:14:20, 16.98it/s, it=2000, loss=0.020]
The progress bar didn't move for three hours!
Maybe you can debug and see what happens. We have not met the issue before. What is your experimental setting?
training: 0%| | 0/200000 [00:00<?, ?it/s][val] loss:2.060e-01 training: 1%| | 1998/200000 [01:59<3:14:20, 16.98it/s, it=2000, loss=0.020] training: 1%| | 1998/200000 [02:18<3:14:20, 16.98it/s, it=2000, loss=0.020]
The progress bar didn't move for three hours!