Open DJever23 opened 2 years ago
Hi: I encountered some problems when running the source code： 2022-02-17 14:43:37,319 - mmdet - INFO - Epoch [1][50/2852] lr: 3.987e-04, eta: 2 days, 6:05:49, time: 1.423, data_time: 0.068, memory: 7011, loss_cls: 0.6662, loss_bbox: 0.9846, loss_centerness: 0.6260, loss_oim: 5.6912, loss: 7.9680, grad_norm: 118.5072 2022-02-17 14:44:35,675 - mmdet - INFO - Epoch [1][100/2852] lr: 4.653e-04, eta: 2 days, 1:12:47, time: 1.167, data_time: 0.016, memory: 7363, loss_cls: 0.5805, loss_bbox: 0.8021, loss_centerness: 0.5967, loss_oim: 5.8589, loss: 7.8382, grad_norm: 104.7832 2022-02-17 14:45:52,820 - mmdet - INFO - Epoch [1][150/2852] lr: 5.320e-04, eta: 2 days, 4:19:57, time: 1.543, data_time: 0.017, memory: 7363, loss_cls: 0.6106, loss_bbox: 0.5222, loss_centerness: 0.5782, loss_oim: 6.3371, loss: 8.0482, grad_norm: 108.9552 2022-02-17 14:47:02,130 - mmdet - INFO - Epoch [1][200/2852] lr: 5.987e-04, eta: 2 days, 4:23:37, time: 1.386, data_time: 0.017, memory: 7363, loss_cls: 0.4726, loss_bbox: 0.5253, loss_centerness: 0.6215, loss_oim: 5.8503, loss: 7.4698, grad_norm: 86.1120 2022-02-17 14:48:04,005 - mmdet - INFO - Epoch [1][250/2852] lr: 6.653e-04, eta: 2 days, 3:17:38, time: 1.237, data_time: 0.015, memory: 7363, loss_cls: 0.4858, loss_bbox: 0.4937, loss_centerness: 0.6017, loss_oim: 6.1532, loss: 7.7343, grad_norm: 94.5427 2022-02-17 14:49:06,965 - mmdet - INFO - Epoch [1][300/2852] lr: 7.320e-04, eta: 2 days, 2:41:31, time: 1.259, data_time: 0.017, memory: 7363, loss_cls: 0.5772, loss_bbox: 0.5174, loss_centerness: 0.5998, loss_oim: 6.5115, loss: 8.2059, grad_norm: 92.6229 2022-02-17 14:50:16,175 - mmdet - INFO - Epoch [1][350/2852] lr: 7.987e-04, eta: 2 days, 2:56:04, time: 1.384, data_time: 0.018, memory: 7363, loss_cls: 0.5621, loss_bbox: 0.5079, loss_centerness: 0.6014, loss_oim: 6.6282, loss: 8.2996, grad_norm: 83.7950 2022-02-17 14:51:23,770 - mmdet - INFO - Epoch [1][400/2852] lr: 8.653e-04, eta: 2 days, 2:57:31, time: 1.352, data_time: 0.019, memory: 7363, loss_cls: 0.6787, loss_bbox: 0.5543, loss_centerness: 0.5968, loss_oim: 6.6915, loss: 8.5213, grad_norm: 136.0015 2022-02-17 14:52:28,603 - mmdet - INFO - Epoch [1][450/2852] lr: 9.320e-04, eta: 2 days, 2:44:25, time: 1.297, data_time: 0.016, memory: 7363, loss_cls: 0.6619, loss_bbox: 0.5619, loss_centerness: 0.6255, loss_oim: 8.8492, loss: 10.6984, grad_norm: 40.6709 2022-02-17 14:53:31,481 - mmdet - INFO - Epoch [1][500/2852] lr: 9.987e-04, eta: 2 days, 2:24:51, time: 1.258, data_time: 0.018, memory: 7363, loss_cls: 0.6701, loss_bbox: 0.5613, loss_centerness: 0.6302, loss_oim: 8.8498, loss: 10.7114, grad_norm: 56.6717 2022-02-17 14:54:34,786 - mmdet - INFO - Epoch [1][550/2852] lr: 1.000e-03, eta: 2 days, 2:10:25, time: 1.266, data_time: 0.015, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan 2022-02-17 14:55:39,327 - mmdet - INFO - Epoch [1][600/2852] lr: 1.000e-03, eta: 2 days, 2:02:53, time: 1.291, data_time: 0.018, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan 2022-02-17 14:56:43,064 - mmdet - INFO - Epoch [1][650/2852] lr: 1.000e-03, eta: 2 days, 1:53:32, time: 1.275, data_time: 0.018, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan 2022-02-17 14:57:46,297 - mmdet - INFO - Epoch [1][700/2852] lr: 1.000e-03, eta: 2 days, 1:43:44, time: 1.265, data_time: 0.017, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan All these values become nan，the command I ran was：python tools/train.py configs/fcos/prw_base_focal_labelnorm_sub_ldcn_fg15_wd1-3.py --gpu-ids 6 --no-validate and I did not change the parameters in the configuration file。 Can you help me？
maybe you could adjust lr
Tracy-git
commented
1 year ago Tracy-git
commented
1 year ago
Hi: I encountered some problems when running the source code： 2022-02-17 14:43:37,319 - mmdet - INFO - Epoch [1][50/2852] lr: 3.987e-04, eta: 2 days, 6:05:49, time: 1.423, data_time: 0.068, memory: 7011, loss_cls: 0.6662, loss_bbox: 0.9846, loss_centerness: 0.6260, loss_oim: 5.6912, loss: 7.9680, grad_norm: 118.5072 2022-02-17 14:44:35,675 - mmdet - INFO - Epoch [1][100/2852] lr: 4.653e-04, eta: 2 days, 1:12:47, time: 1.167, data_time: 0.016, memory: 7363, loss_cls: 0.5805, loss_bbox: 0.8021, loss_centerness: 0.5967, loss_oim: 5.8589, loss: 7.8382, grad_norm: 104.7832 2022-02-17 14:45:52,820 - mmdet - INFO - Epoch [1][150/2852] lr: 5.320e-04, eta: 2 days, 4:19:57, time: 1.543, data_time: 0.017, memory: 7363, loss_cls: 0.6106, loss_bbox: 0.5222, loss_centerness: 0.5782, loss_oim: 6.3371, loss: 8.0482, grad_norm: 108.9552 2022-02-17 14:47:02,130 - mmdet - INFO - Epoch [1][200/2852] lr: 5.987e-04, eta: 2 days, 4:23:37, time: 1.386, data_time: 0.017, memory: 7363, loss_cls: 0.4726, loss_bbox: 0.5253, loss_centerness: 0.6215, loss_oim: 5.8503, loss: 7.4698, grad_norm: 86.1120 2022-02-17 14:48:04,005 - mmdet - INFO - Epoch [1][250/2852] lr: 6.653e-04, eta: 2 days, 3:17:38, time: 1.237, data_time: 0.015, memory: 7363, loss_cls: 0.4858, loss_bbox: 0.4937, loss_centerness: 0.6017, loss_oim: 6.1532, loss: 7.7343, grad_norm: 94.5427 2022-02-17 14:49:06,965 - mmdet - INFO - Epoch [1][300/2852] lr: 7.320e-04, eta: 2 days, 2:41:31, time: 1.259, data_time: 0.017, memory: 7363, loss_cls: 0.5772, loss_bbox: 0.5174, loss_centerness: 0.5998, loss_oim: 6.5115, loss: 8.2059, grad_norm: 92.6229 2022-02-17 14:50:16,175 - mmdet - INFO - Epoch [1][350/2852] lr: 7.987e-04, eta: 2 days, 2:56:04, time: 1.384, data_time: 0.018, memory: 7363, loss_cls: 0.5621, loss_bbox: 0.5079, loss_centerness: 0.6014, loss_oim: 6.6282, loss: 8.2996, grad_norm: 83.7950 2022-02-17 14:51:23,770 - mmdet - INFO - Epoch [1][400/2852] lr: 8.653e-04, eta: 2 days, 2:57:31, time: 1.352, data_time: 0.019, memory: 7363, loss_cls: 0.6787, loss_bbox: 0.5543, loss_centerness: 0.5968, loss_oim: 6.6915, loss: 8.5213, grad_norm: 136.0015 2022-02-17 14:52:28,603 - mmdet - INFO - Epoch [1][450/2852] lr: 9.320e-04, eta: 2 days, 2:44:25, time: 1.297, data_time: 0.016, memory: 7363, loss_cls: 0.6619, loss_bbox: 0.5619, loss_centerness: 0.6255, loss_oim: 8.8492, loss: 10.6984, grad_norm: 40.6709 2022-02-17 14:53:31,481 - mmdet - INFO - Epoch [1][500/2852] lr: 9.987e-04, eta: 2 days, 2:24:51, time: 1.258, data_time: 0.018, memory: 7363, loss_cls: 0.6701, loss_bbox: 0.5613, loss_centerness: 0.6302, loss_oim: 8.8498, loss: 10.7114, grad_norm: 56.6717 2022-02-17 14:54:34,786 - mmdet - INFO - Epoch [1][550/2852] lr: 1.000e-03, eta: 2 days, 2:10:25, time: 1.266, data_time: 0.015, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan 2022-02-17 14:55:39,327 - mmdet - INFO - Epoch [1][600/2852] lr: 1.000e-03, eta: 2 days, 2:02:53, time: 1.291, data_time: 0.018, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan 2022-02-17 14:56:43,064 - mmdet - INFO - Epoch [1][650/2852] lr: 1.000e-03, eta: 2 days, 1:53:32, time: 1.275, data_time: 0.018, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan 2022-02-17 14:57:46,297 - mmdet - INFO - Epoch [1][700/2852] lr: 1.000e-03, eta: 2 days, 1:43:44, time: 1.265, data_time: 0.017, memory: 7363, loss_cls: nan, loss_bbox: nan, loss_centerness: nan, loss_oim: nan, loss: nan, grad_norm: nan All these values become nan，the command I ran was：python tools/train.py configs/fcos/prw_base_focal_labelnorm_sub_ldcn_fg15_wd1-3.py --gpu-ids 6 --no-validate and I did not change the parameters in the configuration file。 Can you help me？