训练过程遇到问题

[d-serafly]

您好： 我配好环境以后，按照示例训练train_lm.sh过程中，训练到8%，即两个epoch后抛出错误，错误代码如下，期待您的解答，十分感谢您。 ape_epochs: 0%| | 0/25 [00:40<?, ?it/s]Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32.0=11] Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32.0 ape_epochs: 4%|\u2588\u258e | 1/25 [01:36<38:33, 96.39s/it]kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt/it, total_it=31] kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt/it, total_it=13] kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt ape_epochs: 8%|\u2588\u2588\u258c | 2/25 [02:57<33:36, 87.68s/it]kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt/it, total_it=62] kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt/it, total_it=62] kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt ape_epochs: 8%|\u2588\u2588\u258c kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txtit [00:00, ?it/s]
kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt kps_pth in get_kps: datasets/linemod/kps_orb9_fps/ape_8_kps.txt loss_rgbd_seg 0.14627921491861343 loss_kp_of 11.617005914960588 loss_ctr_of 1.4170470442090715 loss_all 13.32661139351981 loss_target 13.32661139351981 acc_rgbd 80.30672805649893 loss_rgbd_seg 0.14268102556467055 loss_kp_of 11.599615522112165 loss_ctr_of 1.413561065537589 loss_all 13.298538600376673 loss_target 13.298538600376673 acc_rgbd 80.59970440183368 val_loss 13.222978721965443 ape_epochs: 8%|\u2588\u2588\u258d | 2/25 [03:58<45:46, 119.42s/it] Traceback (most recent call last): | 0/64 [00:00<?, ?it/s, total_it=64] File "train_lm.py", line 697, in train() File "train_lm.py", line 688, in train clr_div=clr_div File "train_lm.py", line 518, in train val_loss, res = self.eval_epoch(test_loader, it=it) File "train_lm.py", line 420, in eval_epoch writer.add_scalars('val_acc', acc_dict, it) File "/home/extend/gy/miniconda3/envs/FFB/lib/python3.7/site-packages/tensorboardX/writer.py", line 490, in add_scalars fw.add_summary(scalar(main_tag, scalar_value), File "/home/extend/gy/miniconda3/envs/FFB/lib/python3.7/site-packages/tensorboardX/summary.py", line 153, in scalar assert(scalar.squeeze().ndim == 0), 'scalar should be 0D' AssertionError: scalar should be 0D Traceback (most recent call last): File "/home/extend/gy/miniconda3/envs/FFB/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/extend/gy/miniconda3/envs/FFB/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/extend/gy/miniconda3/envs/FFB/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in main() File "/home/extend/gy/miniconda3/envs/FFB/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/extend/gy/miniconda3/envs/FFB/bin/python3', '-u', 'train_lm.py', '--local_rank=1', '--gpus=2', '--cls=ape']' returned non-zero exit status 1.

[guess-who-ami]

兄弟，问题怎么解决的

[d-serafly]

兄弟，问题怎么解决的

注释掉if args.local_rank == 0: writer.add_scalars('val_acc', acc_dict, it)这两行