Closed leehaining closed 2 years ago
您好，我在训练icdar2015数据集和自己数据集时都出现了以下问题 2021-10-28 14:39:23,165 maskrcnn_benchmark.trainer INFO: Start training 2021-10-28 14:39:26,805 maskrcnn_benchmark.trainer INFO: eta: 5:03:07 iter: 0 loss: 12.6127 (12.6127) loss_classifier: 0.4748 (0.4748) loss_box_reg: 0.0052 (0.0052) loss_mask: 5.0979 (5.0979) loss_char_mask: 0.0000 (0.0000) loss_seq: 6.0541 (6.0541) loss_seg: 0.9807 (0.9807) time: 3.6375 (3.6375) data: 1.3635 (1.3635) lr: 0.002036 max mem: 1099 2021-10-28 14:39:44,816 maskrcnn_benchmark.trainer INFO: eta: 1:25:33 iter: 20 loss: nan (nan) loss_classifier: nan (nan) loss_box_reg: nan (nan) loss_mask: 5.1765 (356.7433) loss_char_mask: 0.0000 (0.0000) loss_seq: 3.6069 (4.1097) loss_seg: nan (nan) time: 0.6030 (1.0309) data: 0.0082 (0.0800) lr: 0.002756 max mem: 1906 loss值在训练过程中变为'nan'，暂时还未找到出现这种情况的原因，还请解答，谢谢。 训练命令：python -m torch.distributed.launch --nproc_per_node 1 tools/train_net.py --config-file configs/pretrain/seg_rec_poly_fuse_feature.yaml 这是训练中用的seg_rec_poly_fuse_feature.yaml，仅修改了SOLVER部分
2021-10-28 14:39:23,165 maskrcnn_benchmark.trainer INFO: Start training 2021-10-28 14:39:26,805 maskrcnn_benchmark.trainer INFO: eta: 5:03:07 iter: 0 loss: 12.6127 (12.6127) loss_classifier: 0.4748 (0.4748) loss_box_reg: 0.0052 (0.0052) loss_mask: 5.0979 (5.0979) loss_char_mask: 0.0000 (0.0000) loss_seq: 6.0541 (6.0541) loss_seg: 0.9807 (0.9807) time: 3.6375 (3.6375) data: 1.3635 (1.3635) lr: 0.002036 max mem: 1099 2021-10-28 14:39:44,816 maskrcnn_benchmark.trainer INFO: eta: 1:25:33 iter: 20 loss: nan (nan) loss_classifier: nan (nan) loss_box_reg: nan (nan) loss_mask: 5.1765 (356.7433) loss_char_mask: 0.0000 (0.0000) loss_seq: 3.6069 (4.1097) loss_seg: nan (nan) time: 0.6030 (1.0309) data: 0.0082 (0.0800) lr: 0.002756 max mem: 1906
python -m torch.distributed.launch --nproc_per_node 1 tools/train_net.py --config-file configs/pretrain/seg_rec_poly_fuse_feature.yaml
您好，我在训练icdar2015数据集和自己数据集时都出现了以下问题
2021-10-28 14:39:23,165 maskrcnn_benchmark.trainer INFO: Start training 2021-10-28 14:39:26,805 maskrcnn_benchmark.trainer INFO: eta: 5:03:07 iter: 0 loss: 12.6127 (12.6127) loss_classifier: 0.4748 (0.4748) loss_box_reg: 0.0052 (0.0052) loss_mask: 5.0979 (5.0979) loss_char_mask: 0.0000 (0.0000) loss_seq: 6.0541 (6.0541) loss_seg: 0.9807 (0.9807) time: 3.6375 (3.6375) data: 1.3635 (1.3635) lr: 0.002036 max mem: 1099 2021-10-28 14:39:44,816 maskrcnn_benchmark.trainer INFO: eta: 1:25:33 iter: 20 loss: nan (nan) loss_classifier: nan (nan) loss_box_reg: nan (nan) loss_mask: 5.1765 (356.7433) loss_char_mask: 0.0000 (0.0000) loss_seq: 3.6069 (4.1097) loss_seg: nan (nan) time: 0.6030 (1.0309) data: 0.0082 (0.0800) lr: 0.002756 max mem: 1906loss值在训练过程中变为'nan'，暂时还未找到出现这种情况的原因，还请解答，谢谢。 训练命令：python -m torch.distributed.launch --nproc_per_node 1 tools/train_net.py --config-file configs/pretrain/seg_rec_poly_fuse_feature.yaml这是训练中用的seg_rec_poly_fuse_feature.yaml，仅修改了SOLVER部分