raise Exception('Error: Invalid box type: %s' % box)

[WangZZJJ]

2024-07-08T14:00:20.858620949Z Loaded ground truth annotations for 6019 samples. 2024-07-08T14:00:21.030217541Z Filtering tracks 2024-07-08T14:00:21.030556548Z Traceback (most recent call last): 2024-07-08T14:00:21.030567733Z File "./tools/train.py", line 319, in 2024-07-08T14:00:21.030708329Z main() 2024-07-08T14:00:21.030718795Z File "./tools/train.py", line 294, in main 2024-07-08T14:00:21.030751728Z custom_train_model( 2024-07-08T14:00:21.030758649Z File "/ml-engine/code/sparse4d-v3/projects/mmdet3d_plugin/apis/train.py", line 29, in custom_train_model 2024-07-08T14:00:21.030788541Z custom_train_detector( 2024-07-08T14:00:21.030797541Z File "/ml-engine/code/sparse4d-v3/projects/mmdet3d_plugin/apis/mmdet_train.py", line 219, in custom_train_detector 2024-07-08T14:00:21.030843389Z runner.run(data_loaders, cfg.workflow) 2024-07-08T14:00:21.030876659Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 144, in run 2024-07-08T14:00:21.030890559Z iter_runner(iter_loaders[i], **kwargs) 2024-07-08T14:00:21.030898151Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 70, in train 2024-07-08T14:00:21.030907128Z self.call_hook('after_train_iter') 2024-07-08T14:00:21.030914039Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 317, in call_hook 2024-07-08T14:00:21.030963458Z getattr(hook, fn_name)(self) 2024-07-08T14:00:21.030972758Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/mmcv/runner/hooks/evaluation.py", line 266, in after_train_iter 2024-07-08T14:00:21.031000297Z self._do_evaluate(runner) 2024-07-08T14:00:21.031007438Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/mmdet/core/evaluation/eval_hooks.py", line 63, in _do_evaluate 2024-07-08T14:00:21.031033978Z key_score = self.evaluate(runner, results) 2024-07-08T14:00:21.031041173Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/mmcv/runner/hooks/evaluation.py", line 367, in evaluate 2024-07-08T14:00:21.031079651Z eval_res = self.dataloader.dataset.evaluate( 2024-07-08T14:00:21.031084892Z File "/ml-engine/code/sparse4d-v3/projects/mmdet3d_plugin/datasets/nuscenes_3d_det_track_dataset.py", line 589, in evaluate 2024-07-08T14:00:21.031154730Z ret_dict = self._evaluate_single( 2024-07-08T14:00:21.031158960Z File "/ml-engine/code/sparse4d-v3/projects/mmdet3d_plugin/datasets/nuscenes_3d_det_track_dataset.py", line 499, in _evaluate_single 2024-07-08T14:00:21.031203155Z nusc_eval = TrackingEval( 2024-07-08T14:00:21.031206816Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/nuscenes/eval/tracking/evaluate.py", line 97, in init 2024-07-08T14:00:21.031252699Z pred_boxes = filter_eval_boxes(nusc, pred_boxes, self.cfg.class_range, verbose=verbose) 2024-07-08T14:00:21.031256982Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/nuscenes/eval/common/loaders.py", line 219, in filter_eval_boxes 2024-07-08T14:00:21.031295552Z class_field = _get_box_class_field(eval_boxes) 2024-07-08T14:00:21.031299258Z File "/opt/conda/envs/mm_sparse4d/lib/python3.8/site-packages/nuscenes/eval/common/loaders.py", line 283, in _get_box_class_field 2024-07-08T14:00:21.031333192Z raise Exception('Error: Invalid box type: %s' % box) 2024-07-08T14:00:21.031336949Z Exception: Error: Invalid box type: None

[linxuewu]

The model training failed, resulting in nan during inference, maybe.

[WangZZJJ]

请问下，为什么我的loss一开始就这么小，然后减小到3之后就不再下降了，这是正常的吗？ 2024-07-10T03:51:24.718169803Z 2024-07-10 11:51:24,717 - mmdet - INFO - Iter [51/58600] lr: 1.200e-05, eta: 10:04:26, time: 0.619, data_time: 0.056, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0801, loss_cns_0: 0.0174, loss_yns_0: 0.0060, loss_cls_1: 0.0000, loss_box_1: 0.0514, loss_cns_1: 0.0103, loss_yns_1: 0.0027, loss_cls_2: 0.0000, loss_box_2: 0.0042, loss_cns_2: 0.0004, loss_yns_2: 0.0002, loss_cls_3: 0.0000, loss_box_3: 0.0084, loss_cns_3: 0.0012, loss_yns_3: 0.0005, loss_cls_4: 0.0000, loss_box_4: 0.0742, loss_cns_4: 0.0097, loss_yns_4: 0.0050, loss_cls_5: 0.0000, loss_box_5: 0.0334, loss_cns_5: 0.0036, loss_yns_5: 0.0013, loss_cls_dn_0: 0.0000, loss_box_dn_0: 1.0636, loss_cls_dn_1: 0.0000, loss_box_dn_1: 1.7411, loss_cls_dn_2: 0.0000, loss_box_dn_2: 1.7406, loss_cls_dn_3: 0.0000, loss_box_dn_3: 1.7801, loss_cls_dn_4: 0.0000, loss_box_dn_4: 1.8153, loss_cls_dn_5: 0.0000, loss_box_dn_5: 1.8829, loss_dense_depth: 1.3855, loss: 11.7189, grad_norm: 1140.5030 2024-07-10T03:51:53.541868346Z 2024-07-10 11:51:53,541 - mmdet - INFO - Iter [102/58600] lr: 1.404e-05, eta: 9:37:27, time: 0.565, data_time: 0.019, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0740, loss_cns_0: 0.0181, loss_yns_0: 0.0058, loss_cls_1: 0.0000, loss_box_1: 0.0049, loss_cns_1: 0.0012, loss_yns_1: 0.0005, loss_cls_2: 0.0000, loss_box_2: 0.0130, loss_cns_2: 0.0022, loss_yns_2: 0.0009, loss_cls_3: 0.0000, loss_box_3: 0.0221, loss_cns_3: 0.0031, loss_yns_3: 0.0013, loss_cls_4: 0.0000, loss_box_4: 0.0203, loss_cns_4: 0.0034, loss_yns_4: 0.0013, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.9735, loss_cls_dn_1: 0.0000, loss_box_dn_1: 1.2260, loss_cls_dn_2: 0.0000, loss_box_dn_2: 1.1897, loss_cls_dn_3: 0.0000, loss_box_dn_3: 1.1840, loss_cls_dn_4: 0.0000, loss_box_dn_4: 1.1804, loss_cls_dn_5: 0.0000, loss_box_dn_5: 1.1926, loss_dense_depth: 0.9920, loss: 8.1101, grad_norm: 146.5865 2024-07-10T03:52:22.793694828Z 2024-07-10 11:52:22,793 - mmdet - INFO - Iter [153/58600] lr: 1.608e-05, eta: 9:30:51, time: 0.574, data_time: 0.025, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0683, loss_cns_0: 0.0167, loss_yns_0: 0.0058, loss_cls_1: 0.0000, loss_box_1: 0.0399, loss_cns_1: 0.0065, loss_yns_1: 0.0025, loss_cls_2: 0.0000, loss_box_2: 0.0010, loss_cns_2: 0.0003, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0060, loss_cns_3: 0.0008, loss_yns_3: 0.0002, loss_cls_4: 0.0000, loss_box_4: 0.0475, loss_cns_4: 0.0059, loss_yns_4: 0.0030, loss_cls_5: 0.0000, loss_box_5: 0.0007, loss_cns_5: 0.0001, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.9466, loss_cls_dn_1: 0.0000, loss_box_dn_1: 1.0599, loss_cls_dn_2: 0.0000, loss_box_dn_2: 1.0092, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.9997, loss_cls_dn_4: 0.0000, loss_box_dn_4: 1.0038, loss_cls_dn_5: 0.0000, loss_box_dn_5: 1.0126, loss_dense_depth: 0.9270, loss: 7.1642, grad_norm: 106.4319 2024-07-10T03:52:52.140227687Z 2024-07-10 11:52:52,139 - mmdet - INFO - Iter [204/58600] lr: 1.812e-05, eta: 9:27:46, time: 0.575, data_time: 0.026, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0057, loss_cns_0: 0.0016, loss_yns_0: 0.0005, loss_cls_1: 0.0000, loss_box_1: 0.0748, loss_cns_1: 0.0117, loss_yns_1: 0.0046, loss_cls_2: 0.0000, loss_box_2: 0.0024, loss_cns_2: 0.0004, loss_yns_2: 0.0002, loss_cls_3: 0.0000, loss_box_3: 0.0173, loss_cns_3: 0.0019, loss_yns_3: 0.0013, loss_cls_4: 0.0000, loss_box_4: 0.0319, loss_cns_4: 0.0069, loss_yns_4: 0.0025, loss_cls_5: 0.0000, loss_box_5: 0.0006, loss_cns_5: 0.0001, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.9217, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.9992, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.9547, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.9620, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.9730, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.9894, loss_dense_depth: 0.8982, loss: 6.8624, grad_norm: 67.3712 2024-07-10T03:53:21.172355799Z 2024-07-10 11:53:21,172 - mmdet - INFO - Iter [255/58600] lr: 2.016e-05, eta: 9:24:32, time: 0.569, data_time: 0.022, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0226, loss_cns_1: 0.0040, loss_yns_1: 0.0012, loss_cls_2: 0.0000, loss_box_2: 0.0028, loss_cns_2: 0.0006, loss_yns_2: 0.0002, loss_cls_3: 0.0000, loss_box_3: 0.0223, loss_cns_3: 0.0030, loss_yns_3: 0.0014, loss_cls_4: 0.0000, loss_box_4: 0.0126, loss_cns_4: 0.0028, loss_yns_4: 0.0010, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.8785, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.9294, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.8690, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.8705, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.8778, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.8860, loss_dense_depth: 0.8013, loss: 6.1870, grad_norm: 63.1020 2024-07-10T03:53:50.364096957Z 2024-07-10 11:53:50,363 - mmdet - INFO - Iter [306/58600] lr: 2.220e-05, eta: 9:22:43, time: 0.572, data_time: 0.022, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0469, loss_cns_1: 0.0068, loss_yns_1: 0.0028, loss_cls_2: 0.0000, loss_box_2: 0.0005, loss_cns_2: 0.0002, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0150, loss_cns_3: 0.0011, loss_yns_3: 0.0007, loss_cls_4: 0.0000, loss_box_4: 0.0038, loss_cns_4: 0.0008, loss_yns_4: 0.0002, loss_cls_5: 0.0000, loss_box_5: 0.0004, loss_cns_5: 0.0002, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.8427, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.8799, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.8324, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.8339, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.8386, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.8488, loss_dense_depth: 0.8236, loss: 5.9794, grad_norm: 61.6182 2024-07-10T03:54:19.698328546Z 2024-07-10 11:54:19,698 - mmdet - INFO - Iter [357/58600] lr: 2.424e-05, eta: 9:21:40, time: 0.575, data_time: 0.022, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0260, loss_cns_1: 0.0042, loss_yns_1: 0.0017, loss_cls_2: 0.0000, loss_box_2: 0.0003, loss_cns_2: 0.0001, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0029, loss_cns_3: 0.0002, loss_yns_3: 0.0002, loss_cls_4: 0.0000, loss_box_4: 0.0076, loss_cns_4: 0.0012, loss_yns_4: 0.0004, loss_cls_5: 0.0000, loss_box_5: 0.0010, loss_cns_5: 0.0002, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.8191, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.8017, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.7617, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.7621, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.7640, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.7682, loss_dense_depth: 0.9342, loss: 5.6569, grad_norm: 57.4429 2024-07-10T03:54:48.773835364Z 2024-07-10 11:54:48,773 - mmdet - INFO - Iter [408/58600] lr: 2.628e-05, eta: 9:20:08, time: 0.570, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0268, loss_cns_1: 0.0041, loss_yns_1: 0.0019, loss_cls_2: 0.0000, loss_box_2: 0.0005, loss_cns_2: 0.0002, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0030, loss_cns_3: 0.0005, loss_yns_3: 0.0001, loss_cls_4: 0.0000, loss_box_4: 0.0047, loss_cns_4: 0.0010, loss_yns_4: 0.0003, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.8114, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.8268, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.7869, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.7902, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.7951, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.8023, loss_dense_depth: 0.8991, loss: 5.7550, grad_norm: 48.6020 2024-07-10T03:55:17.758672916Z 2024-07-10 11:55:17,758 - mmdet - INFO - Iter [459/58600] lr: 2.832e-05, eta: 9:18:39, time: 0.568, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0433, loss_cns_1: 0.0091, loss_yns_1: 0.0032, loss_cls_2: 0.0000, loss_box_2: 0.0011, loss_cns_2: 0.0003, loss_yns_2: 0.0002, loss_cls_3: 0.0000, loss_box_3: 0.0173, loss_cns_3: 0.0016, loss_yns_3: 0.0011, loss_cls_4: 0.0000, loss_box_4: 0.0041, loss_cns_4: 0.0009, loss_yns_4: 0.0002, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.8110, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.8212, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.7887, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.7919, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.7977, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.8067, loss_dense_depth: 0.8688, loss: 5.7686, grad_norm: 43.3542 2024-07-10T03:55:46.559700812Z 2024-07-10 11:55:46,559 - mmdet - INFO - Iter [510/58600] lr: 2.999e-05, eta: 9:17:01, time: 0.565, data_time: 0.024, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0268, loss_cns_1: 0.0053, loss_yns_1: 0.0019, loss_cls_2: 0.0000, loss_box_2: 0.0008, loss_cns_2: 0.0004, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0140, loss_cns_3: 0.0017, loss_yns_3: 0.0009, loss_cls_4: 0.0000, loss_box_4: 0.0074, loss_cns_4: 0.0009, loss_yns_4: 0.0004, loss_cls_5: 0.0000, loss_box_5: 0.0008, loss_cns_5: 0.0001, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.8045, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.8125, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.7938, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.7995, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.8092, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.8207, loss_dense_depth: 0.7913, loss: 5.6929, grad_norm: 39.9963 2024-07-10T03:56:15.621762978Z 2024-07-10 11:56:15,621 - mmdet - INFO - Iter [561/58600] lr: 2.999e-05, eta: 9:16:03, time: 0.570, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0203, loss_cns_1: 0.0040, loss_yns_1: 0.0012, loss_cls_2: 0.0000, loss_box_2: 0.0051, loss_cns_2: 0.0016, loss_yns_2: 0.0001, loss_cls_3: 0.0000, loss_box_3: 0.0107, loss_cns_3: 0.0015, loss_yns_3: 0.0006, loss_cls_4: 0.0000, loss_box_4: 0.0089, loss_cns_4: 0.0024, loss_yns_4: 0.0007, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7800, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.7455, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.7203, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.7210, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.7241, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.7315, loss_dense_depth: 0.7743, loss: 5.2538, grad_norm: 32.9235 2024-07-10T03:56:44.532849999Z 2024-07-10 11:56:44,532 - mmdet - INFO - Iter [612/58600] lr: 2.999e-05, eta: 9:14:55, time: 0.567, data_time: 0.022, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0090, loss_cns_1: 0.0018, loss_yns_1: 0.0008, loss_cls_2: 0.0000, loss_box_2: 0.0037, loss_cns_2: 0.0004, loss_yns_2: 0.0002, loss_cls_3: 0.0000, loss_box_3: 0.0147, loss_cns_3: 0.0019, loss_yns_3: 0.0010, loss_cls_4: 0.0000, loss_box_4: 0.0026, loss_cns_4: 0.0009, loss_yns_4: 0.0004, loss_cls_5: 0.0000, loss_box_5: 0.0006, loss_cns_5: 0.0002, loss_yns_5: 0.0001, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7819, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.7566, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.7439, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.7484, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.7553, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.7615, loss_dense_depth: 0.7870, loss: 5.3729, grad_norm: 34.4658 2024-07-10T03:57:13.605863136Z 2024-07-10 11:57:13,605 - mmdet - INFO - Iter [663/58600] lr: 2.999e-05, eta: 9:14:07, time: 0.570, data_time: 0.021, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0117, loss_cns_1: 0.0024, loss_yns_1: 0.0007, loss_cls_2: 0.0000, loss_box_2: 0.0012, loss_cns_2: 0.0003, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0162, loss_cns_3: 0.0032, loss_yns_3: 0.0013, loss_cls_4: 0.0000, loss_box_4: 0.0000, loss_cns_4: 0.0000, loss_yns_4: 0.0000, loss_cls_5: 0.0000, loss_box_5: 0.0004, loss_cns_5: 0.0001, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7675, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6985, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6858, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6882, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6908, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6967, loss_dense_depth: 0.7123, loss: 4.9773, grad_norm: 27.5327 2024-07-10T03:57:42.637711871Z 2024-07-10 11:57:42,637 - mmdet - INFO - Iter [714/58600] lr: 2.999e-05, eta: 9:13:19, time: 0.569, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0034, loss_cns_1: 0.0008, loss_yns_1: 0.0003, loss_cls_2: 0.0000, loss_box_2: 0.0019, loss_cns_2: 0.0002, loss_yns_2: 0.0003, loss_cls_3: 0.0000, loss_box_3: 0.0121, loss_cns_3: 0.0021, loss_yns_3: 0.0010, loss_cls_4: 0.0000, loss_box_4: 0.0000, loss_cns_4: 0.0000, loss_yns_4: 0.0000, loss_cls_5: 0.0000, loss_box_5: 0.0006, loss_cns_5: 0.0002, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7752, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.7089, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6937, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6947, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6968, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.7010, loss_dense_depth: 0.6921, loss: 4.9852, grad_norm: 26.1546 2024-07-10T03:58:11.867425752Z 2024-07-10 11:58:11,867 - mmdet - INFO - Iter [765/58600] lr: 2.999e-05, eta: 9:12:48, time: 0.573, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0067, loss_cns_1: 0.0021, loss_yns_1: 0.0007, loss_cls_2: 0.0000, loss_box_2: 0.0009, loss_cns_2: 0.0001, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0479, loss_cns_3: 0.0057, loss_yns_3: 0.0025, loss_cls_4: 0.0000, loss_box_4: 0.0000, loss_cns_4: 0.0000, loss_yns_4: 0.0000, loss_cls_5: 0.0000, loss_box_5: 0.0008, loss_cns_5: 0.0002, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7488, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.7009, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6855, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6847, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6859, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6887, loss_dense_depth: 0.7423, loss: 5.0045, grad_norm: 23.1586 2024-07-10T03:58:41.066027902Z 2024-07-10 11:58:41,065 - mmdet - INFO - Iter [816/58600] lr: 2.999e-05, eta: 9:12:15, time: 0.573, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0023, loss_cns_1: 0.0006, loss_yns_1: 0.0002, loss_cls_2: 0.0000, loss_box_2: 0.0012, loss_cns_2: 0.0000, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0343, loss_cns_3: 0.0053, loss_yns_3: 0.0021, loss_cls_4: 0.0000, loss_box_4: 0.0000, loss_cns_4: 0.0000, loss_yns_4: 0.0000, loss_cls_5: 0.0000, loss_box_5: 0.0002, loss_cns_5: 0.0002, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7490, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6892, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6769, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6770, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6789, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6829, loss_dense_depth: 0.8063, loss: 5.0067, grad_norm: 24.8598 2024-07-10T03:59:11.349374833Z 2024-07-10 11:59:11,349 - mmdet - INFO - Iter [867/58600] lr: 2.998e-05, eta: 9:12:55, time: 0.594, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0039, loss_cns_1: 0.0009, loss_yns_1: 0.0002, loss_cls_2: 0.0000, loss_box_2: 0.0000, loss_cns_2: 0.0000, loss_yns_2: 0.0000, loss_cls_3: 0.0000, loss_box_3: 0.0107, loss_cns_3: 0.0018, loss_yns_3: 0.0006, loss_cls_4: 0.0000, loss_box_4: 0.0009, loss_cns_4: 0.0000, loss_yns_4: 0.0001, loss_cls_5: 0.0000, loss_box_5: 0.0013, loss_cns_5: 0.0006, loss_yns_5: 0.0011, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7505, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6801, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6683, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6705, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6755, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6814, loss_dense_depth: 0.7253, loss: 4.8735, grad_norm: 22.6808 2024-07-10T03:59:40.526659993Z 2024-07-10 11:59:40,526 - mmdet - INFO - Iter [918/58600] lr: 2.998e-05, eta: 9:12:17, time: 0.572, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0011, loss_cns_1: 0.0004, loss_yns_1: 0.0001, loss_cls_2: 0.0000, loss_box_2: 0.0032, loss_cns_2: 0.0007, loss_yns_2: 0.0003, loss_cls_3: 0.0000, loss_box_3: 0.0095, loss_cns_3: 0.0018, loss_yns_3: 0.0006, loss_cls_4: 0.0000, loss_box_4: 0.0010, loss_cns_4: 0.0002, loss_yns_4: 0.0001, loss_cls_5: 0.0000, loss_box_5: 0.0005, loss_cns_5: 0.0002, loss_yns_5: 0.0001, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7493, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6824, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6719, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6718, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6735, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6763, loss_dense_depth: 0.6632, loss: 4.8083, grad_norm: 18.7320 2024-07-10T04:00:09.694823414Z 2024-07-10 12:00:09,694 - mmdet - INFO - Iter [969/58600] lr: 2.998e-05, eta: 9:11:40, time: 0.572, data_time: 0.021, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0015, loss_cns_1: 0.0006, loss_yns_1: 0.0001, loss_cls_2: 0.0000, loss_box_2: 0.0007, loss_cns_2: 0.0001, loss_yns_2: 0.0004, loss_cls_3: 0.0000, loss_box_3: 0.0331, loss_cns_3: 0.0047, loss_yns_3: 0.0020, loss_cls_4: 0.0000, loss_box_4: 0.0000, loss_cns_4: 0.0000, loss_yns_4: 0.0000, loss_cls_5: 0.0000, loss_box_5: 0.0002, loss_cns_5: 0.0001, loss_yns_5: 0.0001, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7483, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6943, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6895, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6927, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6970, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.7031, loss_dense_depth: 0.7237, loss: 4.9921, grad_norm: 21.4205 2024-07-10T04:00:38.708817849Z 2024-07-10 12:00:38,708 - mmdet - INFO - Iter [1020/58600] lr: 2.998e-05, eta: 9:10:55, time: 0.569, data_time: 0.021, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0047, loss_cns_1: 0.0013, loss_yns_1: 0.0007, loss_cls_2: 0.0000, loss_box_2: 0.0009, loss_cns_2: 0.0001, loss_yns_2: 0.0001, loss_cls_3: 0.0000, loss_box_3: 0.0518, loss_cns_3: 0.0064, loss_yns_3: 0.0028, loss_cls_4: 0.0000, loss_box_4: 0.0012, loss_cns_4: 0.0003, loss_yns_4: 0.0001, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7416, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6468, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6379, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6368, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6380, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6402, loss_dense_depth: 0.6968, loss: 4.7086, grad_norm: 21.9242 2024-07-10T04:01:07.561628293Z 2024-07-10 12:01:07,561 - mmdet - INFO - Iter [1071/58600] lr: 2.998e-05, eta: 9:10:03, time: 0.566, data_time: 0.023, memory: 14356, loss_cls_0: 0.0000, loss_box_0: 0.0000, loss_cns_0: 0.0000, loss_yns_0: 0.0000, loss_cls_1: 0.0000, loss_box_1: 0.0131, loss_cns_1: 0.0036, loss_yns_1: 0.0014, loss_cls_2: 0.0000, loss_box_2: 0.0010, loss_cns_2: 0.0002, loss_yns_2: 0.0001, loss_cls_3: 0.0000, loss_box_3: 0.0059, loss_cns_3: 0.0003, loss_yns_3: 0.0003, loss_cls_4: 0.0000, loss_box_4: 0.0003, loss_cns_4: 0.0001, loss_yns_4: 0.0000, loss_cls_5: 0.0000, loss_box_5: 0.0000, loss_cns_5: 0.0000, loss_yns_5: 0.0000, loss_cls_dn_0: 0.0000, loss_box_dn_0: 0.7543, loss_cls_dn_1: 0.0000, loss_box_dn_1: 0.6806, loss_cls_dn_2: 0.0000, loss_box_dn_2: 0.6727, loss_cls_dn_3: 0.0000, loss_box_dn_3: 0.6729, loss_cls_dn_4: 0.0000, loss_box_dn_4: 0.6756, loss_cls_dn_5: 0.0000, loss_box_dn_5: 0.6794, loss_dense_depth: 0.7308, loss: 4.8924, grad_norm: 19.8908

[linxuewu]

你什么配置训的，batch size多少？

[WangZZJJ]

我是用projects/configs/sparse4dv3_temporal_r50_1x8_bs6_256x704.py训的，参数都没有改过。 total_batch_size = 48 num_gpus = 8

[WangZZJJ]

问题已解决，是因为我编译deformable_aggregation的环境和训练的环境没有保持一致，感谢回复~