训练异常,grad_norm异常

这是部分log 2024-09-16 03:48:01,681 - mmdet - INFO - Iter [15750/87900] lr: 1.846e-04, eta: 18:45:47, time: 0.905, data_time: 0.037, memory: 18408, loss_cls3d_0: 0.5535, loss_box3d_0: 1.5157, loss_cns3d_0: 0.6291, loss_yns3d_0: 0.1046, loss_cls3d_1: 0.5385, loss_box3d_1: 0.9064, loss_cns3d_1: 0.6552, loss_yns3d_1: 0.0683, loss_cls3d_2: 0.5157, loss_box3d_2: 0.8664, loss_cns3d_2: 0.6489, loss_yns3d_2: 0.0657, loss_cls3d_3: 0.5120, loss_box3d_3: 0.8412, loss_cns3d_3: 0.6435, loss_yns3d_3: 0.0660, loss_cls3d_4: 0.4929, loss_box3d_4: 0.8177, loss_cns3d_4: 0.6447, loss_yns3d_4: 0.0639, loss_cls3d_5: 0.4899, loss_box3d_5: 0.8087, loss_cns3d_5: 0.6434, loss_yns3d_5: 0.0631, loss_cls2d_0: 0.5529, loss_iou2d_0: 0.6409, loss_box2d_0: 0.2765, loss_alpha2d_0: 0.3558, loss_cls2d_1: 0.4199, loss_iou2d_1: 0.5695, loss_box2d_1: 0.2277, loss_alpha2d_1: 0.2796, loss_cls2d_2: 0.3770, loss_iou2d_2: 0.5535, loss_box2d_2: 0.2198, loss_alpha2d_2: 0.2665, loss_cls3d_dn_0: 0.0548, loss_box3d_dn_0: 0.6735, loss_cls3d_dn_1: 0.0565, loss_box3d_dn_1: 0.4596, loss_cls3d_dn_2: 0.0423, loss_box3d_dn_2: 0.4419, loss_cls3d_dn_3: 0.0428, loss_box3d_dn_3: 0.4121, loss_cls3d_dn_4: 0.0410, loss_box3d_dn_4: 0.4063, loss_cls3d_dn_5: 0.0401, loss_box3d_dn_5: 0.3993, loss_cls2d_dn_0: 0.0443, loss_iou2d_dn_0: 0.6463, loss_box2d_dn_0: 0.3947, loss_alpha2d_dn_0: 0.1954, loss_cls2d_dn_1: 0.0050, loss_iou2d_dn_1: 0.4881, loss_box2d_dn_1: 0.1966, loss_alpha2d_dn_1: 0.0740, loss_cls2d_dn_2: 0.0048, loss_iou2d_dn_2: 0.4710, loss_box2d_dn_2: 0.1830, loss_alpha2d_dn_2: 0.0666, loss_dense_depth: 0.5964, loss: 24.3311, grad_norm: 56.0316 2024-09-16 03:48:47,327 - mmdet - INFO - Iter [15800/87900] lr: 1.845e-04, eta: 18:44:55, time: 0.913, data_time: 0.035, memory: 18408, loss_cls3d_0: 0.5737, loss_box3d_0: 1.4508, loss_cns3d_0: 0.6294, loss_yns3d_0: 0.1016, loss_cls3d_1: 0.5361, loss_box3d_1: 0.8905, loss_cns3d_1: 0.6565, loss_yns3d_1: 0.0637, loss_cls3d_2: 0.5083, loss_box3d_2: 0.8552, loss_cns3d_2: 0.6524, loss_yns3d_2: 0.0600, loss_cls3d_3: 0.5027, loss_box3d_3: 0.8307, loss_cns3d_3: 0.6470, loss_yns3d_3: 0.0600, loss_cls3d_4: 0.4890, loss_box3d_4: 0.8096, loss_cns3d_4: 0.6482, loss_yns3d_4: 0.0586, loss_cls3d_5: 0.4854, loss_box3d_5: 0.8008, loss_cns3d_5: 0.6469, loss_yns3d_5: 0.0581, loss_cls2d_0: 0.5626, loss_iou2d_0: 0.6087, loss_box2d_0: 0.2840, loss_alpha2d_0: 0.3236, loss_cls2d_1: 0.4141, loss_iou2d_1: 0.5436, loss_box2d_1: 0.2390, loss_alpha2d_1: 0.2500, loss_cls2d_2: 0.3770, loss_iou2d_2: 0.5463, loss_box2d_2: 0.2424, loss_alpha2d_2: 0.2375, loss_cls3d_dn_0: 0.0628, loss_box3d_dn_0: 0.6754, loss_cls3d_dn_1: 0.0597, loss_box3d_dn_1: 0.4413, loss_cls3d_dn_2: 0.0451, loss_box3d_dn_2: 0.4247, loss_cls3d_dn_3: 0.0454, loss_box3d_dn_3: 0.3939, loss_cls3d_dn_4: 0.0440, loss_box3d_dn_4: 0.3889, loss_cls3d_dn_5: 0.0425, loss_box3d_dn_5: 0.3827, loss_cls2d_dn_0: 0.0445, loss_iou2d_dn_0: 0.6112, loss_box2d_dn_0: 0.4032, loss_alpha2d_dn_0: 0.1861, loss_cls2d_dn_1: 0.0055, loss_iou2d_dn_1: 0.4610, loss_box2d_dn_1: 0.1958, loss_alpha2d_dn_1: 0.0715, loss_cls2d_dn_2: 0.0052, loss_iou2d_dn_2: 0.4494, loss_box2d_dn_2: 0.1854, loss_alpha2d_dn_2: 0.0628, loss_dense_depth: 0.5677, loss: 23.9003, grad_norm: 48.5352 2024-09-16 03:49:08,761 - mmdet - INFO - Saving checkpoint at 15822 iterations 2024-09-16 03:49:35,384 - mmdet - INFO - Iter [15850/87900] lr: 1.844e-04, eta: 18:44:13, time: 0.961, data_time: 0.035, memory: 18408, loss_cls3d_0: 0.6729, loss_box3d_0: 1.6361, loss_cns3d_0: 0.6246, loss_yns3d_0: 0.1218, loss_cls3d_1: 0.6770, loss_box3d_1: 1.1532, loss_cns3d_1: 0.6691, loss_yns3d_1: 0.0866, loss_cls3d_2: 0.6952, loss_box3d_2: 1.0809, loss_cns3d_2: 0.6644, loss_yns3d_2: 0.0869, loss_cls3d_3: 0.6982, loss_box3d_3: 1.0408, loss_cns3d_3: 0.6611, loss_yns3d_3: 0.0833, loss_cls3d_4: 0.6690, loss_box3d_4: 1.0045, loss_cns3d_4: 0.6598, loss_yns3d_4: 0.0810, loss_cls3d_5: 0.6539, loss_box3d_5: 0.9824, loss_cns3d_5: 0.6565, loss_yns3d_5: 0.0791, loss_cls2d_0: 0.6046, loss_iou2d_0: 0.6795, loss_box2d_0: 0.3138, loss_alpha2d_0: 0.3807, loss_cls2d_1: 0.5932, loss_iou2d_1: 0.6519, loss_box2d_1: 0.2935, loss_alpha2d_1: 0.3348, loss_cls2d_2: 0.5926, loss_iou2d_2: 0.7010, loss_box2d_2: 0.3343, loss_alpha2d_2: 0.3307, loss_cls3d_dn_0: 0.0686, loss_box3d_dn_0: 0.7060, loss_cls3d_dn_1: 0.0762, loss_box3d_dn_1: 0.5247, loss_cls3d_dn_2: 0.0713, loss_box3d_dn_2: 0.4960, loss_cls3d_dn_3: 0.0765, loss_box3d_dn_3: 0.4722, loss_cls3d_dn_4: 0.0681, loss_box3d_dn_4: 0.4654, loss_cls3d_dn_5: 0.0646, loss_box3d_dn_5: 0.4543, loss_cls2d_dn_0: 0.0448, loss_iou2d_dn_0: 0.6739, loss_box2d_dn_0: 0.4439, loss_alpha2d_dn_0: 0.2129, loss_cls2d_dn_1: 0.0105, loss_iou2d_dn_1: 0.5411, loss_box2d_dn_1: 0.2669, loss_alpha2d_dn_1: 0.0989, loss_cls2d_dn_2: 0.0126, loss_iou2d_dn_2: 0.5321, loss_box2d_dn_2: 0.2929, loss_alpha2d_dn_2: 0.0975, loss_dense_depth: 0.6139, loss: 28.6348, grad_norm: 70.7608 2024-09-16 03:50:21,868 - mmdet - INFO - Iter [15900/87900] lr: 1.843e-04, eta: 18:43:25, time: 0.930, data_time: 0.036, memory: 18408, loss_cls3d_0: 0.8628, loss_box3d_0: 1.8840, loss_cns3d_0: 0.6081, loss_yns3d_0: 0.1496, loss_cls3d_1: 0.8565, loss_box3d_1: 1.5533, loss_cns3d_1: 0.6629, loss_yns3d_1: 0.1193, loss_cls3d_2: 0.9352, loss_box3d_2: 1.4634, loss_cns3d_2: 0.6738, loss_yns3d_2: 0.1299, loss_cls3d_3: 0.8849, loss_box3d_3: 1.3899, loss_cns3d_3: 0.6645, loss_yns3d_3: 0.1178, loss_cls3d_4: 0.8439, loss_box3d_4: 1.3184, loss_cns3d_4: 0.6599, loss_yns3d_4: 0.1014, loss_cls3d_5: 0.8164, loss_box3d_5: 1.2761, loss_cns3d_5: 0.6543, loss_yns3d_5: 0.0953, loss_cls2d_0: 0.6963, loss_iou2d_0: 0.7512, loss_box2d_0: 0.3774, loss_alpha2d_0: 0.3935, loss_cls2d_1: 0.8196, loss_iou2d_1: 0.7373, loss_box2d_1: 0.3701, loss_alpha2d_1: 0.3904, loss_cls2d_2: 0.7889, loss_iou2d_2: 0.6802, loss_box2d_2: 0.3254, loss_alpha2d_2: 0.3590, loss_cls3d_dn_0: 0.1038, loss_box3d_dn_0: 0.7653, loss_cls3d_dn_1: 0.1364, loss_box3d_dn_1: 0.6149, loss_cls3d_dn_2: 0.1387, loss_box3d_dn_2: 0.5910, loss_cls3d_dn_3: 0.1427, loss_box3d_dn_3: 0.5581, loss_cls3d_dn_4: 0.1160, loss_box3d_dn_4: 0.5443, loss_cls3d_dn_5: 0.1097, loss_box3d_dn_5: 0.5276, loss_cls2d_dn_0: 0.0489, loss_iou2d_dn_0: 0.6969, loss_box2d_dn_0: 0.5142, loss_alpha2d_dn_0: 0.2219, loss_cls2d_dn_1: 0.0221, loss_iou2d_dn_1: 0.6080, loss_box2d_dn_1: 0.3687, loss_alpha2d_dn_1: 0.1534, loss_cls2d_dn_2: 0.0215, loss_iou2d_dn_2: 0.5753, loss_box2d_dn_2: 0.3572, loss_alpha2d_dn_2: 0.1379, loss_dense_depth: 0.6733, loss: 34.1590, grad_norm: 103.1606 2024-09-16 03:51:09,585 - mmdet - INFO - Iter [15950/87900] lr: 1.842e-04, eta: 18:42:42, time: 0.954, data_time: 0.035, memory: 18408, loss_cls3d_0: 0.8301, loss_box3d_0: 1.8211, loss_cns3d_0: 0.6187, loss_yns3d_0: 0.1423, loss_cls3d_1: 0.8341, loss_box3d_1: 1.4643, loss_cns3d_1: 0.6689, loss_yns3d_1: 0.1032, loss_cls3d_2: 0.8564, loss_box3d_2: 1.3426, loss_cns3d_2: 0.6706, loss_yns3d_2: 0.1019, loss_cls3d_3: 0.8246, loss_box3d_3: 1.2688, loss_cns3d_3: 0.6553, loss_yns3d_3: 0.0958, loss_cls3d_4: 0.7827, loss_box3d_4: 1.2084, loss_cns3d_4: 0.6572, loss_yns3d_4: 0.0880, loss_cls3d_5: 0.7687, loss_box3d_5: 1.1795, loss_cns3d_5: 0.6548, loss_yns3d_5: 0.0862, loss_cls2d_0: 0.6812, loss_iou2d_0: 0.7671, loss_box2d_0: 0.3664, loss_alpha2d_0: 0.4424, loss_cls2d_1: 0.7280, loss_iou2d_1: 0.6953, loss_box2d_1: 0.3160, loss_alpha2d_1: 0.3993, loss_cls2d_2: 0.6677, loss_iou2d_2: 0.6560, loss_box2d_2: 0.2895, loss_alpha2d_2: 0.3706, loss_cls3d_dn_0: 0.1123, loss_box3d_dn_0: 0.7599, loss_cls3d_dn_1: 0.1520, loss_box3d_dn_1: 0.5884, loss_cls3d_dn_2: 0.1184, loss_box3d_dn_2: 0.5615, loss_cls3d_dn_3: 0.1161, loss_box3d_dn_3: 0.5286, loss_cls3d_dn_4: 0.0978, loss_box3d_dn_4: 0.5182, loss_cls3d_dn_5: 0.0939, loss_box3d_dn_5: 0.5027, loss_cls2d_dn_0: 0.0487, loss_iou2d_dn_0: 0.7193, loss_box2d_dn_0: 0.5160, loss_alpha2d_dn_0: 0.2381, loss_cls2d_dn_1: 0.0179, loss_iou2d_dn_1: 0.5926, loss_box2d_dn_1: 0.3203, loss_alpha2d_dn_1: 0.1341, loss_cls2d_dn_2: 0.0139, loss_iou2d_dn_2: 0.5542, loss_box2d_dn_2: 0.2907, loss_alpha2d_dn_2: 0.1051, loss_dense_depth: 0.6703, loss: 32.4746, grad_norm: 62.4061 2024-09-16 03:52:00,181 - mmdet - INFO - Exp name: simpb_nus_r50_uimg_704x256.py 2024-09-16 03:52:00,183 - mmdet - INFO - Iter [16000/87900] lr: 1.841e-04, eta: 18:42:13, time: 1.012, data_time: 0.038, memory: 18408, loss_cls3d_0: 0.8729, loss_box3d_0: 1.7809, loss_cns3d_0: 0.6162, loss_yns3d_0: 0.1576, loss_cls3d_1: 0.8208, loss_box3d_1: 1.4031, loss_cns3d_1: 0.6673, loss_yns3d_1: 0.1058, loss_cls3d_2: 0.8217, loss_box3d_2: 1.2757, loss_cns3d_2: 0.6646, loss_yns3d_2: 0.1022, loss_cls3d_3: 0.7842, loss_box3d_3: 1.2226, loss_cns3d_3: 0.6555, loss_yns3d_3: 0.0993, loss_cls3d_4: 0.7569, loss_box3d_4: 1.1656, loss_cns3d_4: 0.6523, loss_yns3d_4: 0.0918, loss_cls3d_5: 0.7505, loss_box3d_5: 1.1366, loss_cns3d_5: 0.6505, loss_yns3d_5: 0.0898, loss_cls2d_0: 0.7196, loss_iou2d_0: 0.7717, loss_box2d_0: 0.4051, loss_alpha2d_0: 0.4197, loss_cls2d_1: 0.6833, loss_iou2d_1: 0.6720, loss_box2d_1: 0.3311, loss_alpha2d_1: 0.3616, loss_cls2d_2: 0.6255, loss_iou2d_2: 0.6387, loss_box2d_2: 0.3100, loss_alpha2d_2: 0.3379, loss_cls3d_dn_0: 0.1139, loss_box3d_dn_0: 0.7583, loss_cls3d_dn_1: 0.1522, loss_box3d_dn_1: 0.5825, loss_cls3d_dn_2: 0.1143, loss_box3d_dn_2: 0.5463, loss_cls3d_dn_3: 0.1103, loss_box3d_dn_3: 0.5124, loss_cls3d_dn_4: 0.0972, loss_box3d_dn_4: 0.5036, loss_cls3d_dn_5: 0.0928, loss_box3d_dn_5: 0.4927, loss_cls2d_dn_0: 0.0483, loss_iou2d_dn_0: 0.7233, loss_box2d_dn_0: 0.5541, loss_alpha2d_dn_0: 0.2328, loss_cls2d_dn_1: 0.0163, loss_iou2d_dn_1: 0.5726, loss_box2d_dn_1: 0.3279, loss_alpha2d_dn_1: 0.1161, loss_cls2d_dn_2: 0.0133, loss_iou2d_dn_2: 0.5339, loss_box2d_dn_2: 0.2917, loss_alpha2d_dn_2: 0.0967, loss_dense_depth: 0.6909, loss: 31.9149, grad_norm: 65.6015 2024-09-16 03:52:45,360 - mmdet - INFO - Iter [16050/87900] lr: 1.840e-04, eta: 18:41:18, time: 0.903, data_time: 0.037, memory: 18408, loss_cls3d_0: 0.9670, loss_box3d_0: 1.8707, loss_cns3d_0: 0.6110, loss_yns3d_0: 0.1610, loss_cls3d_1: 0.8836, loss_box3d_1: 1.6082, loss_cns3d_1: 0.6564, loss_yns3d_1: 0.1149, loss_cls3d_2: 0.8779, loss_box3d_2: 1.4920, loss_cns3d_2: 0.6603, loss_yns3d_2: 0.1106, loss_cls3d_3: 0.8536, loss_box3d_3: 1.4343, loss_cns3d_3: 0.6529, loss_yns3d_3: 0.1062, loss_cls3d_4: 0.8226, loss_box3d_4: 1.3708, loss_cns3d_4: 0.6468, loss_yns3d_4: 0.1002, loss_cls3d_5: 0.8092, loss_box3d_5: 1.3447, loss_cns3d_5: 0.6455, loss_yns3d_5: 0.0975, loss_cls2d_0: 0.8631, loss_iou2d_0: 0.8790, loss_box2d_0: 0.4605, loss_alpha2d_0: 0.4488, loss_cls2d_1: 0.7664, loss_iou2d_1: 0.7946, loss_box2d_1: 0.4028, loss_alpha2d_1: 0.3990, loss_cls2d_2: 0.7048, loss_iou2d_2: 0.7564, loss_box2d_2: 0.3725, loss_alpha2d_2: 0.3697, loss_cls3d_dn_0: 0.1300, loss_box3d_dn_0: 0.7868, loss_cls3d_dn_1: 0.1668, loss_box3d_dn_1: 0.6150, loss_cls3d_dn_2: 0.1273, loss_box3d_dn_2: 0.5790, loss_cls3d_dn_3: 0.1276, loss_box3d_dn_3: 0.5467, loss_cls3d_dn_4: 0.1076, loss_box3d_dn_4: 0.5352, loss_cls3d_dn_5: 0.1033, loss_box3d_dn_5: 0.5247, loss_cls2d_dn_0: 0.0508, loss_iou2d_dn_0: 0.7947, loss_box2d_dn_0: 0.6141, loss_alpha2d_dn_0: 0.2475, loss_cls2d_dn_1: 0.0181, loss_iou2d_dn_1: 0.6384, loss_box2d_dn_1: 0.3733, loss_alpha2d_dn_1: 0.1403, loss_cls2d_dn_2: 0.0151, loss_iou2d_dn_2: 0.5951, loss_box2d_dn_2: 0.3230, loss_alpha2d_dn_2: 0.1054, loss_dense_depth: 0.8102, loss: 35.1915, grad_norm: 103.8281 2024-09-16 03:53:30,982 - mmdet - INFO - Iter [16100/87900] lr: 1.839e-04, eta: 18:40:26, time: 0.913, data_time: 0.038, memory: 18408, loss_cls3d_0: 1.0174, loss_box3d_0: 1.8811, loss_cns3d_0: 0.6110, loss_yns3d_0: 0.1615, loss_cls3d_1: 0.9094, loss_box3d_1: 1.6727, loss_cns3d_1: 0.6470, loss_yns3d_1: 0.1208, loss_cls3d_2: 0.8893, loss_box3d_2: 1.5427, loss_cns3d_2: 0.6554, loss_yns3d_2: 0.1178, loss_cls3d_3: 0.8643, loss_box3d_3: 1.4755, loss_cns3d_3: 0.6506, loss_yns3d_3: 0.1124, loss_cls3d_4: 0.8277, loss_box3d_4: 1.4107, loss_cns3d_4: 0.6452, loss_yns3d_4: 0.1051, loss_cls3d_5: 0.8167, loss_box3d_5: 1.3897, loss_cns3d_5: 0.6452, loss_yns3d_5: 0.1046, loss_cls2d_0: 0.8829, loss_iou2d_0: 0.8962, loss_box2d_0: 0.4975, loss_alpha2d_0: 0.4899, loss_cls2d_1: 0.7897, loss_iou2d_1: 0.8174, loss_box2d_1: 0.4185, loss_alpha2d_1: 0.4453, loss_cls2d_2: 0.7320, loss_iou2d_2: 0.7721, loss_box2d_2: 0.3867, loss_alpha2d_2: 0.4178, loss_cls3d_dn_0: 0.1395, loss_box3d_dn_0: 0.7904, loss_cls3d_dn_1: 0.1802, loss_box3d_dn_1: 0.6549, loss_cls3d_dn_2: 0.1310, loss_box3d_dn_2: 0.6113, loss_cls3d_dn_3: 0.1302, loss_box3d_dn_3: 0.5751, loss_cls3d_dn_4: 0.1129, loss_box3d_dn_4: 0.5629, loss_cls3d_dn_5: 0.1099, loss_box3d_dn_5: 0.5524, loss_cls2d_dn_0: 0.0525, loss_iou2d_dn_0: 0.8182, loss_box2d_dn_0: 0.6735, loss_alpha2d_dn_0: 0.2656, loss_cls2d_dn_1: 0.0196, loss_iou2d_dn_1: 0.6595, loss_box2d_dn_1: 0.4094, loss_alpha2d_dn_1: 0.1463, loss_cls2d_dn_2: 0.0152, loss_iou2d_dn_2: 0.6081, loss_box2d_dn_2: 0.3553, loss_alpha2d_dn_2: 0.1125, loss_dense_depth: 0.8543, loss: 36.3608, grad_norm: 121.7326 2024-09-16 03:54:16,229 - mmdet - INFO - Iter [16150/87900] lr: 1.838e-04, eta: 18:39:32, time: 0.905, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.1118, loss_box3d_0: 2.0929, loss_cns3d_0: 0.5718, loss_yns3d_0: 0.1705, loss_cls3d_1: 1.0747, loss_box3d_1: 1.8987, loss_cns3d_1: 0.6325, loss_yns3d_1: 0.1450, loss_cls3d_2: 1.0061, loss_box3d_2: 1.7560, loss_cns3d_2: 0.6473, loss_yns3d_2: 0.1466, loss_cls3d_3: 0.9579, loss_box3d_3: 1.6762, loss_cns3d_3: 0.6461, loss_yns3d_3: 0.1396, loss_cls3d_4: 0.9170, loss_box3d_4: 1.6089, loss_cns3d_4: 0.6440, loss_yns3d_4: 0.1308, loss_cls3d_5: 0.9082, loss_box3d_5: 1.5762, loss_cns3d_5: 0.6424, loss_yns3d_5: 0.1287, loss_cls2d_0: 0.9373, loss_iou2d_0: 1.0217, loss_box2d_0: 0.6493, loss_alpha2d_0: 0.5222, loss_cls2d_1: 0.9237, loss_iou2d_1: 0.9037, loss_box2d_1: 0.5048, loss_alpha2d_1: 0.4949, loss_cls2d_2: 0.8436, loss_iou2d_2: 0.8347, loss_box2d_2: 0.4438, loss_alpha2d_2: 0.4614, loss_cls3d_dn_0: 0.1844, loss_box3d_dn_0: 0.8285, loss_cls3d_dn_1: 0.2378, loss_box3d_dn_1: 0.7169, loss_cls3d_dn_2: 0.1616, loss_box3d_dn_2: 0.6625, loss_cls3d_dn_3: 0.1560, loss_box3d_dn_3: 0.6249, loss_cls3d_dn_4: 0.1262, loss_box3d_dn_4: 0.6067, loss_cls3d_dn_5: 0.1249, loss_box3d_dn_5: 0.5931, loss_cls2d_dn_0: 0.0568, loss_iou2d_dn_0: 0.8352, loss_box2d_dn_0: 0.7133, loss_alpha2d_dn_0: 0.2821, loss_cls2d_dn_1: 0.0261, loss_iou2d_dn_1: 0.7162, loss_box2d_dn_1: 0.5212, loss_alpha2d_dn_1: 0.1953, loss_cls2d_dn_2: 0.0198, loss_iou2d_dn_2: 0.6380, loss_box2d_dn_2: 0.4042, loss_alpha2d_dn_2: 0.1388, loss_dense_depth: 0.9080, loss: 40.2498, grad_norm: 741.6997 2024-09-16 03:55:03,064 - mmdet - INFO - Iter [16200/87900] lr: 1.837e-04, eta: 18:38:46, time: 0.936, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.1586, loss_box3d_0: 2.1439, loss_cns3d_0: 0.5976, loss_yns3d_0: 0.1698, loss_cls3d_1: 1.0520, loss_box3d_1: 1.9622, loss_cns3d_1: 0.6205, loss_yns3d_1: 0.1417, loss_cls3d_2: 1.0025, loss_box3d_2: 1.7660, loss_cns3d_2: 0.6379, loss_yns3d_2: 0.1362, loss_cls3d_3: 0.9576, loss_box3d_3: 1.7011, loss_cns3d_3: 0.6397, loss_yns3d_3: 0.1326, loss_cls3d_4: 0.9244, loss_box3d_4: 1.6340, loss_cns3d_4: 0.6363, loss_yns3d_4: 0.1273, loss_cls3d_5: 0.9145, loss_box3d_5: 1.6042, loss_cns3d_5: 0.6362, loss_yns3d_5: 0.1252, loss_cls2d_0: 1.0312, loss_iou2d_0: 1.0939, loss_box2d_0: 0.6785, loss_alpha2d_0: 0.5832, loss_cls2d_1: 0.9288, loss_iou2d_1: 0.9419, loss_box2d_1: 0.5273, loss_alpha2d_1: 0.5208, loss_cls2d_2: 0.8549, loss_iou2d_2: 0.8768, loss_box2d_2: 0.4730, loss_alpha2d_2: 0.4875, loss_cls3d_dn_0: 0.2224, loss_box3d_dn_0: 0.8922, loss_cls3d_dn_1: 0.2626, loss_box3d_dn_1: 0.7168, loss_cls3d_dn_2: 0.1537, loss_box3d_dn_2: 0.6621, loss_cls3d_dn_3: 0.1518, loss_box3d_dn_3: 0.6302, loss_cls3d_dn_4: 0.1238, loss_box3d_dn_4: 0.6118, loss_cls3d_dn_5: 0.1205, loss_box3d_dn_5: 0.5995, loss_cls2d_dn_0: 0.0529, loss_iou2d_dn_0: 0.8645, loss_box2d_dn_0: 0.7817, loss_alpha2d_dn_0: 0.3023, loss_cls2d_dn_1: 0.0242, loss_iou2d_dn_1: 0.7228, loss_box2d_dn_1: 0.4890, loss_alpha2d_dn_1: 0.1866, loss_cls2d_dn_2: 0.0180, loss_iou2d_dn_2: 0.6647, loss_box2d_dn_2: 0.4118, loss_alpha2d_dn_2: 0.1390, loss_dense_depth: 0.9400, loss: 41.1647, grad_norm: 285.4884 2024-09-16 03:55:47,769 - mmdet - INFO - Iter [16250/87900] lr: 1.836e-04, eta: 18:37:50, time: 0.894, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2150, loss_box3d_0: 2.1375, loss_cns3d_0: 0.6044, loss_yns3d_0: 0.1681, loss_cls3d_1: 1.0703, loss_box3d_1: 1.8710, loss_cns3d_1: 0.6323, loss_yns3d_1: 0.1363, loss_cls3d_2: 0.9877, loss_box3d_2: 1.6986, loss_cns3d_2: 0.6427, loss_yns3d_2: 0.1363, loss_cls3d_3: 0.9514, loss_box3d_3: 1.6315, loss_cns3d_3: 0.6429, loss_yns3d_3: 0.1312, loss_cls3d_4: 0.9151, loss_box3d_4: 1.5723, loss_cns3d_4: 0.6389, loss_yns3d_4: 0.1252, loss_cls3d_5: 0.9059, loss_box3d_5: 1.5365, loss_cns3d_5: 0.6381, loss_yns3d_5: 0.1243, loss_cls2d_0: 1.0418, loss_iou2d_0: 1.1127, loss_box2d_0: 0.6864, loss_alpha2d_0: 0.5792, loss_cls2d_1: 0.9174, loss_iou2d_1: 0.9160, loss_box2d_1: 0.4914, loss_alpha2d_1: 0.5067, loss_cls2d_2: 0.8475, loss_iou2d_2: 0.8485, loss_box2d_2: 0.4415, loss_alpha2d_2: 0.4664, loss_cls3d_dn_0: 0.2084, loss_box3d_dn_0: 0.8612, loss_cls3d_dn_1: 0.2317, loss_box3d_dn_1: 0.7029, loss_cls3d_dn_2: 0.1424, loss_box3d_dn_2: 0.6552, loss_cls3d_dn_3: 0.1404, loss_box3d_dn_3: 0.6222, loss_cls3d_dn_4: 0.1146, loss_box3d_dn_4: 0.6064, loss_cls3d_dn_5: 0.1127, loss_box3d_dn_5: 0.5939, loss_cls2d_dn_0: 0.0547, loss_iou2d_dn_0: 0.8917, loss_box2d_dn_0: 0.8264, loss_alpha2d_dn_0: 0.3007, loss_cls2d_dn_1: 0.0212, loss_iou2d_dn_1: 0.7082, loss_box2d_dn_1: 0.4937, loss_alpha2d_dn_1: 0.1594, loss_cls2d_dn_2: 0.0164, loss_iou2d_dn_2: 0.6462, loss_box2d_dn_2: 0.3957, loss_alpha2d_dn_2: 0.1148, loss_dense_depth: 0.9886, loss: 40.5784, grad_norm: 289.5313 2024-09-16 03:56:33,821 - mmdet - INFO - Iter [16300/87900] lr: 1.835e-04, eta: 18:37:00, time: 0.921, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.1977, loss_box3d_0: 2.1199, loss_cns3d_0: 0.6005, loss_yns3d_0: 0.1718, loss_cls3d_1: 1.0696, loss_box3d_1: 1.9275, loss_cns3d_1: 0.6195, loss_yns3d_1: 0.1513, loss_cls3d_2: 1.0197, loss_box3d_2: 1.7496, loss_cns3d_2: 0.6427, loss_yns3d_2: 0.1493, loss_cls3d_3: 0.9759, loss_box3d_3: 1.6884, loss_cns3d_3: 0.6463, loss_yns3d_3: 0.1443, loss_cls3d_4: 0.9467, loss_box3d_4: 1.6187, loss_cns3d_4: 0.6443, loss_yns3d_4: 0.1359, loss_cls3d_5: 0.9365, loss_box3d_5: 1.5872, loss_cns3d_5: 0.6468, loss_yns3d_5: 0.1342, loss_cls2d_0: 1.0511, loss_iou2d_0: 1.0570, loss_box2d_0: 0.6685, loss_alpha2d_0: 0.5766, loss_cls2d_1: 0.9374, loss_iou2d_1: 0.8797, loss_box2d_1: 0.4879, loss_alpha2d_1: 0.5020, loss_cls2d_2: 0.8671, loss_iou2d_2: 0.8158, loss_box2d_2: 0.4430, loss_alpha2d_2: 0.4685, loss_cls3d_dn_0: 0.1899, loss_box3d_dn_0: 0.9053, loss_cls3d_dn_1: 0.2303, loss_box3d_dn_1: 0.7515, loss_cls3d_dn_2: 0.1464, loss_box3d_dn_2: 0.6841, loss_cls3d_dn_3: 0.1425, loss_box3d_dn_3: 0.6480, loss_cls3d_dn_4: 0.1192, loss_box3d_dn_4: 0.6294, loss_cls3d_dn_5: 0.1183, loss_box3d_dn_5: 0.6155, loss_cls2d_dn_0: 0.0536, loss_iou2d_dn_0: 0.8578, loss_box2d_dn_0: 0.7768, loss_alpha2d_dn_0: 0.2989, loss_cls2d_dn_1: 0.0234, loss_iou2d_dn_1: 0.7228, loss_box2d_dn_1: 0.5215, loss_alpha2d_dn_1: 0.1769, loss_cls2d_dn_2: 0.0168, loss_iou2d_dn_2: 0.6463, loss_box2d_dn_2: 0.4183, loss_alpha2d_dn_2: 0.1310, loss_dense_depth: 0.9247, loss: 41.0284, grad_norm: 421.9854 2024-09-16 03:57:23,728 - mmdet - INFO - Iter [16350/87900] lr: 1.834e-04, eta: 18:36:26, time: 0.998, data_time: 0.038, memory: 18408, loss_cls3d_0: 1.2174, loss_box3d_0: 2.1583, loss_cns3d_0: 0.6034, loss_yns3d_0: 0.1708, loss_cls3d_1: 1.0897, loss_box3d_1: 2.1083, loss_cns3d_1: 0.5985, loss_yns3d_1: 0.1522, loss_cls3d_2: 1.0497, loss_box3d_2: 1.9365, loss_cns3d_2: 0.6254, loss_yns3d_2: 0.1497, loss_cls3d_3: 1.0133, loss_box3d_3: 1.8553, loss_cns3d_3: 0.6311, loss_yns3d_3: 0.1469, loss_cls3d_4: 0.9806, loss_box3d_4: 1.7853, loss_cns3d_4: 0.6300, loss_yns3d_4: 0.1370, loss_cls3d_5: 0.9755, loss_box3d_5: 1.7490, loss_cns3d_5: 0.6319, loss_yns3d_5: 0.1359, loss_cls2d_0: 1.0843, loss_iou2d_0: 1.1548, loss_box2d_0: 0.7107, loss_alpha2d_0: 0.5841, loss_cls2d_1: 0.9766, loss_iou2d_1: 1.0019, loss_box2d_1: 0.5590, loss_alpha2d_1: 0.5408, loss_cls2d_2: 0.9097, loss_iou2d_2: 0.9202, loss_box2d_2: 0.5041, loss_alpha2d_2: 0.5028, loss_cls3d_dn_0: 0.2083, loss_box3d_dn_0: 0.9355, loss_cls3d_dn_1: 0.2425, loss_box3d_dn_1: 0.8000, loss_cls3d_dn_2: 0.1554, loss_box3d_dn_2: 0.7293, loss_cls3d_dn_3: 0.1558, loss_box3d_dn_3: 0.6905, loss_cls3d_dn_4: 0.1266, loss_box3d_dn_4: 0.6688, loss_cls3d_dn_5: 0.1244, loss_box3d_dn_5: 0.6521, loss_cls2d_dn_0: 0.0542, loss_iou2d_dn_0: 0.8992, loss_box2d_dn_0: 0.8002, loss_alpha2d_dn_0: 0.3053, loss_cls2d_dn_1: 0.0213, loss_iou2d_dn_1: 0.7769, loss_box2d_dn_1: 0.5683, loss_alpha2d_dn_1: 0.2121, loss_cls2d_dn_2: 0.0175, loss_iou2d_dn_2: 0.6998, loss_box2d_dn_2: 0.4486, loss_alpha2d_dn_2: 0.1502, loss_dense_depth: 1.0274, loss: 43.4506, grad_norm: 601.1068 2024-09-16 03:58:10,981 - mmdet - INFO - Iter [16400/87900] lr: 1.833e-04, eta: 18:35:41, time: 0.945, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2044, loss_box3d_0: 2.2384, loss_cns3d_0: 0.5940, loss_yns3d_0: 0.1674, loss_cls3d_1: 1.1473, loss_box3d_1: 2.3285, loss_cns3d_1: 0.5740, loss_yns3d_1: 0.1595, loss_cls3d_2: 1.1019, loss_box3d_2: 2.1393, loss_cns3d_2: 0.6056, loss_yns3d_2: 0.1590, loss_cls3d_3: 1.0789, loss_box3d_3: 2.0844, loss_cns3d_3: 0.6114, loss_yns3d_3: 0.1559, loss_cls3d_4: 1.0414, loss_box3d_4: 1.9949, loss_cns3d_4: 0.6134, loss_yns3d_4: 0.1480, loss_cls3d_5: 1.0361, loss_box3d_5: 1.9662, loss_cns3d_5: 0.6150, loss_yns3d_5: 0.1471, loss_cls2d_0: 1.1416, loss_iou2d_0: 1.2288, loss_box2d_0: 0.7920, loss_alpha2d_0: 0.6098, loss_cls2d_1: 1.0433, loss_iou2d_1: 1.1260, loss_box2d_1: 0.6705, loss_alpha2d_1: 0.5876, loss_cls2d_2: 0.9856, loss_iou2d_2: 1.0432, loss_box2d_2: 0.6037, loss_alpha2d_2: 0.5544, loss_cls3d_dn_0: 0.2242, loss_box3d_dn_0: 0.9908, loss_cls3d_dn_1: 0.2755, loss_box3d_dn_1: 0.8471, loss_cls3d_dn_2: 0.1762, loss_box3d_dn_2: 0.7675, loss_cls3d_dn_3: 0.1781, loss_box3d_dn_3: 0.7360, loss_cls3d_dn_4: 0.1447, loss_box3d_dn_4: 0.7032, loss_cls3d_dn_5: 0.1409, loss_box3d_dn_5: 0.6886, loss_cls2d_dn_0: 0.0539, loss_iou2d_dn_0: 0.9193, loss_box2d_dn_0: 0.8506, loss_alpha2d_dn_0: 0.3121, loss_cls2d_dn_1: 0.0267, loss_iou2d_dn_1: 0.8165, loss_box2d_dn_1: 0.6483, loss_alpha2d_dn_1: 0.2355, loss_cls2d_dn_2: 0.0193, loss_iou2d_dn_2: 0.7480, loss_box2d_dn_2: 0.5146, loss_alpha2d_dn_2: 0.1590, loss_dense_depth: 1.1215, loss: 46.5966, grad_norm: nan 2024-09-16 03:58:55,321 - mmdet - INFO - Iter [16450/87900] lr: 1.832e-04, eta: 18:34:44, time: 0.887, data_time: 0.035, memory: 18408, loss_cls3d_0: 1.2433, loss_box3d_0: 2.1553, loss_cns3d_0: 0.6018, loss_yns3d_0: 0.1705, loss_cls3d_1: 1.2024, loss_box3d_1: 2.6862, loss_cns3d_1: 0.5091, loss_yns3d_1: 0.1667, loss_cls3d_2: 1.1954, loss_box3d_2: 2.6125, loss_cns3d_2: 0.5198, loss_yns3d_2: 0.1661, loss_cls3d_3: 1.1899, loss_box3d_3: 2.5400, loss_cns3d_3: 0.5379, loss_yns3d_3: 0.1649, loss_cls3d_4: 1.1560, loss_box3d_4: 2.4401, loss_cns3d_4: 0.5484, loss_yns3d_4: 0.1615, loss_cls3d_5: 1.1518, loss_box3d_5: 2.4071, loss_cns3d_5: 0.5528, loss_yns3d_5: 0.1619, loss_cls2d_0: 1.2132, loss_iou2d_0: 1.3211, loss_box2d_0: 0.8435, loss_alpha2d_0: 0.6277, loss_cls2d_1: 1.1753, loss_iou2d_1: 1.3243, loss_box2d_1: 0.8396, loss_alpha2d_1: 0.6281, loss_cls2d_2: 1.1152, loss_iou2d_2: 1.3139, loss_box2d_2: 0.8007, loss_alpha2d_2: 0.6126, loss_cls3d_dn_0: 0.2008, loss_box3d_dn_0: 0.9629, loss_cls3d_dn_1: 0.2799, loss_box3d_dn_1: 0.9521, loss_cls3d_dn_2: 0.1843, loss_box3d_dn_2: 0.8738, loss_cls3d_dn_3: 0.1917, loss_box3d_dn_3: 0.8422, loss_cls3d_dn_4: 0.1489, loss_box3d_dn_4: 0.7995, loss_cls3d_dn_5: 0.1492, loss_box3d_dn_5: 0.7899, loss_cls2d_dn_0: 0.0564, loss_iou2d_dn_0: 0.9516, loss_box2d_dn_0: 0.8825, loss_alpha2d_dn_0: 0.3166, loss_cls2d_dn_1: 0.0280, loss_iou2d_dn_1: 0.9168, loss_box2d_dn_1: 0.7430, loss_alpha2d_dn_1: 0.2575, loss_cls2d_dn_2: 0.0207, loss_iou2d_dn_2: 0.8769, loss_box2d_dn_2: 0.6331, loss_alpha2d_dn_2: 0.1863, loss_dense_depth: 1.2725, loss: 51.5740, grad_norm: nan 2024-09-16 03:59:39,756 - mmdet - INFO - Iter [16500/87900] lr: 1.831e-04, eta: 18:33:47, time: 0.888, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2513, loss_box3d_0: 2.1432, loss_cns3d_0: 0.6048, loss_yns3d_0: 0.1685, loss_cls3d_1: 1.2022, loss_box3d_1: 2.6076, loss_cns3d_1: 0.5442, loss_yns3d_1: 0.1683, loss_cls3d_2: 1.1940, loss_box3d_2: 2.5379, loss_cns3d_2: 0.5606, loss_yns3d_2: 0.1674, loss_cls3d_3: 1.1783, loss_box3d_3: 2.5104, loss_cns3d_3: 0.5653, loss_yns3d_3: 0.1673, loss_cls3d_4: 1.1624, loss_box3d_4: 2.4459, loss_cns3d_4: 0.5732, loss_yns3d_4: 0.1657, loss_cls3d_5: 1.1608, loss_box3d_5: 2.4148, loss_cns3d_5: 0.5751, loss_yns3d_5: 0.1658, loss_cls2d_0: 1.2308, loss_iou2d_0: 1.3195, loss_box2d_0: 0.9046, loss_alpha2d_0: 0.6312, loss_cls2d_1: 1.1657, loss_iou2d_1: 1.3072, loss_box2d_1: 0.8403, loss_alpha2d_1: 0.6328, loss_cls2d_2: 1.1308, loss_iou2d_2: 1.2513, loss_box2d_2: 0.7936, loss_alpha2d_2: 0.6231, loss_cls3d_dn_0: 0.1874, loss_box3d_dn_0: 0.9429, loss_cls3d_dn_1: 0.2677, loss_box3d_dn_1: 0.8950, loss_cls3d_dn_2: 0.1636, loss_box3d_dn_2: 0.8151, loss_cls3d_dn_3: 0.1784, loss_box3d_dn_3: 0.7903, loss_cls3d_dn_4: 0.1388, loss_box3d_dn_4: 0.7537, loss_cls3d_dn_5: 0.1374, loss_box3d_dn_5: 0.7425, loss_cls2d_dn_0: 0.0535, loss_iou2d_dn_0: 0.9504, loss_box2d_dn_0: 0.9430, loss_alpha2d_dn_0: 0.3161, loss_cls2d_dn_1: 0.0276, loss_iou2d_dn_1: 0.9040, loss_box2d_dn_1: 0.7497, loss_alpha2d_dn_1: 0.2449, loss_cls2d_dn_2: 0.0197, loss_iou2d_dn_2: 0.8430, loss_box2d_dn_2: 0.6173, loss_alpha2d_dn_2: 0.1806, loss_dense_depth: 1.3270, loss: 51.2560, grad_norm: inf 2024-09-16 04:00:23,791 - mmdet - INFO - Iter [16550/87900] lr: 1.830e-04, eta: 18:32:48, time: 0.881, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.3847, loss_box3d_0: 2.2006, loss_cns3d_0: 0.5774, loss_yns3d_0: 0.1746, loss_cls3d_1: 1.2946, loss_box3d_1: 2.5672, loss_cns3d_1: 0.5449, loss_yns3d_1: 0.1700, loss_cls3d_2: 1.2445, loss_box3d_2: 2.4694, loss_cns3d_2: 0.5644, loss_yns3d_2: 0.1693, loss_cls3d_3: 1.2201, loss_box3d_3: 2.4432, loss_cns3d_3: 0.5688, loss_yns3d_3: 0.1688, loss_cls3d_4: 1.1961, loss_box3d_4: 2.3880, loss_cns3d_4: 0.5768, loss_yns3d_4: 0.1688, loss_cls3d_5: 1.1993, loss_box3d_5: 2.3753, loss_cns3d_5: 0.5765, loss_yns3d_5: 0.1677, loss_cls2d_0: 1.2583, loss_iou2d_0: 1.3124, loss_box2d_0: 0.8878, loss_alpha2d_0: 0.6340, loss_cls2d_1: 1.2002, loss_iou2d_1: 1.3068, loss_box2d_1: 0.8632, loss_alpha2d_1: 0.6393, loss_cls2d_2: 1.1514, loss_iou2d_2: 1.2327, loss_box2d_2: 0.7938, loss_alpha2d_2: 0.6268, loss_cls3d_dn_0: 0.2396, loss_box3d_dn_0: 0.9568, loss_cls3d_dn_1: 0.2989, loss_box3d_dn_1: 0.8773, loss_cls3d_dn_2: 0.1910, loss_box3d_dn_2: 0.8055, loss_cls3d_dn_3: 0.1978, loss_box3d_dn_3: 0.7801, loss_cls3d_dn_4: 0.1578, loss_box3d_dn_4: 0.7532, loss_cls3d_dn_5: 0.1589, loss_box3d_dn_5: 0.7452, loss_cls2d_dn_0: 0.0533, loss_iou2d_dn_0: 0.9239, loss_box2d_dn_0: 0.9577, loss_alpha2d_dn_0: 0.3169, loss_cls2d_dn_1: 0.0308, loss_iou2d_dn_1: 0.8812, loss_box2d_dn_1: 0.7458, loss_alpha2d_dn_1: 0.2406, loss_cls2d_dn_2: 0.0233, loss_iou2d_dn_2: 0.8286, loss_box2d_dn_2: 0.6258, loss_alpha2d_dn_2: 0.1828, loss_dense_depth: 1.5465, loss: 51.8371, grad_norm: nan 2024-09-16 04:01:09,342 - mmdet - INFO - Iter [16600/87900] lr: 1.829e-04, eta: 18:31:56, time: 0.911, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2711, loss_box3d_0: 2.2703, loss_cns3d_0: 0.5899, loss_yns3d_0: 0.1727, loss_cls3d_1: 1.2405, loss_box3d_1: 2.5776, loss_cns3d_1: 0.5348, loss_yns3d_1: 0.1706, loss_cls3d_2: 1.2136, loss_box3d_2: 2.4508, loss_cns3d_2: 0.5657, loss_yns3d_2: 0.1702, loss_cls3d_3: 1.2013, loss_box3d_3: 2.4314, loss_cns3d_3: 0.5688, loss_yns3d_3: 0.1703, loss_cls3d_4: 1.1711, loss_box3d_4: 2.3679, loss_cns3d_4: 0.5788, loss_yns3d_4: 0.1686, loss_cls3d_5: 1.1707, loss_box3d_5: 2.3568, loss_cns3d_5: 0.5763, loss_yns3d_5: 0.1676, loss_cls2d_0: 1.2469, loss_iou2d_0: 1.3775, loss_box2d_0: 0.9296, loss_alpha2d_0: 0.6322, loss_cls2d_1: 1.1758, loss_iou2d_1: 1.3607, loss_box2d_1: 0.8738, loss_alpha2d_1: 0.6350, loss_cls2d_2: 1.1340, loss_iou2d_2: 1.3006, loss_box2d_2: 0.8011, loss_alpha2d_2: 0.6280, loss_cls3d_dn_0: 0.2688, loss_box3d_dn_0: 0.9811, loss_cls3d_dn_1: 0.3504, loss_box3d_dn_1: 0.8850, loss_cls3d_dn_2: 0.2031, loss_box3d_dn_2: 0.8077, loss_cls3d_dn_3: 0.2102, loss_box3d_dn_3: 0.7861, loss_cls3d_dn_4: 0.1576, loss_box3d_dn_4: 0.7581, loss_cls3d_dn_5: 0.1546, loss_box3d_dn_5: 0.7499, loss_cls2d_dn_0: 0.0557, loss_iou2d_dn_0: 0.9763, loss_box2d_dn_0: 0.9664, loss_alpha2d_dn_0: 0.3158, loss_cls2d_dn_1: 0.0345, loss_iou2d_dn_1: 0.9147, loss_box2d_dn_1: 0.7301, loss_alpha2d_dn_1: 0.2444, loss_cls2d_dn_2: 0.0243, loss_iou2d_dn_2: 0.8680, loss_box2d_dn_2: 0.6240, loss_alpha2d_dn_2: 0.1915, loss_dense_depth: 1.2387, loss: 51.7494, grad_norm: nan 2024-09-16 04:01:53,746 - mmdet - INFO - Iter [16650/87900] lr: 1.828e-04, eta: 18:30:59, time: 0.888, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2733, loss_box3d_0: 2.1434, loss_cns3d_0: 0.6017, loss_yns3d_0: 0.1800, loss_cls3d_1: 1.2040, loss_box3d_1: 2.5418, loss_cns3d_1: 0.5265, loss_yns3d_1: 0.1714, loss_cls3d_2: 1.1938, loss_box3d_2: 2.4179, loss_cns3d_2: 0.5567, loss_yns3d_2: 0.1710, loss_cls3d_3: 1.1844, loss_box3d_3: 2.3665, loss_cns3d_3: 0.5664, loss_yns3d_3: 0.1708, loss_cls3d_4: 1.1639, loss_box3d_4: 2.3184, loss_cns3d_4: 0.5722, loss_yns3d_4: 0.1703, loss_cls3d_5: 1.1652, loss_box3d_5: 2.2956, loss_cns3d_5: 0.5743, loss_yns3d_5: 0.1694, loss_cls2d_0: 1.2300, loss_iou2d_0: 1.3222, loss_box2d_0: 0.8591, loss_alpha2d_0: 0.6326, loss_cls2d_1: 1.1559, loss_iou2d_1: 1.3302, loss_box2d_1: 0.8240, loss_alpha2d_1: 0.6348, loss_cls2d_2: 1.1293, loss_iou2d_2: 1.2634, loss_box2d_2: 0.7675, loss_alpha2d_2: 0.6289, loss_cls3d_dn_0: 0.3465, loss_box3d_dn_0: 1.0276, loss_cls3d_dn_1: 0.3494, loss_box3d_dn_1: 0.9018, loss_cls3d_dn_2: 0.2082, loss_box3d_dn_2: 0.8347, loss_cls3d_dn_3: 0.2248, loss_box3d_dn_3: 0.8038, loss_cls3d_dn_4: 0.1589, loss_box3d_dn_4: 0.7789, loss_cls3d_dn_5: 0.1610, loss_box3d_dn_5: 0.7669, loss_cls2d_dn_0: 0.0528, loss_iou2d_dn_0: 0.9685, loss_box2d_dn_0: 0.8984, loss_alpha2d_dn_0: 0.3160, loss_cls2d_dn_1: 0.0324, loss_iou2d_dn_1: 0.9141, loss_box2d_dn_1: 0.7482, loss_alpha2d_dn_1: 0.2663, loss_cls2d_dn_2: 0.0223, loss_iou2d_dn_2: 0.8556, loss_box2d_dn_2: 0.6147, loss_alpha2d_dn_2: 0.2037, loss_dense_depth: 1.3162, loss: 51.2487, grad_norm: inf 2024-09-16 04:02:38,285 - mmdet - INFO - Iter [16700/87900] lr: 1.827e-04, eta: 18:30:03, time: 0.891, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2399, loss_box3d_0: 2.4455, loss_cns3d_0: 0.5392, loss_yns3d_0: 0.1704, loss_cls3d_1: 1.2433, loss_box3d_1: 2.5917, loss_cns3d_1: 0.5408, loss_yns3d_1: 0.1700, loss_cls3d_2: 1.1985, loss_box3d_2: 2.5021, loss_cns3d_2: 0.5688, loss_yns3d_2: 0.1693, loss_cls3d_3: 1.1902, loss_box3d_3: 2.4568, loss_cns3d_3: 0.5724, loss_yns3d_3: 0.1694, loss_cls3d_4: 1.1666, loss_box3d_4: 2.3918, loss_cns3d_4: 0.5806, loss_yns3d_4: 0.1696, loss_cls3d_5: 1.1654, loss_box3d_5: 2.3691, loss_cns3d_5: 0.5824, loss_yns3d_5: 0.1682, loss_cls2d_0: 1.2522, loss_iou2d_0: 1.3334, loss_box2d_0: 0.8482, loss_alpha2d_0: 0.6330, loss_cls2d_1: 1.1681, loss_iou2d_1: 1.3012, loss_box2d_1: 0.8165, loss_alpha2d_1: 0.6324, loss_cls2d_2: 1.1120, loss_iou2d_2: 1.2125, loss_box2d_2: 0.7205, loss_alpha2d_2: 0.6275, loss_cls3d_dn_0: 0.2970, loss_box3d_dn_0: 1.1094, loss_cls3d_dn_1: 0.3639, loss_box3d_dn_1: 0.9042, loss_cls3d_dn_2: 0.1861, loss_box3d_dn_2: 0.8304, loss_cls3d_dn_3: 0.1979, loss_box3d_dn_3: 0.8040, loss_cls3d_dn_4: 0.1416, loss_box3d_dn_4: 0.7754, loss_cls3d_dn_5: 0.1396, loss_box3d_dn_5: 0.7664, loss_cls2d_dn_0: 0.0538, loss_iou2d_dn_0: 0.9610, loss_box2d_dn_0: 0.8948, loss_alpha2d_dn_0: 0.3160, loss_cls2d_dn_1: 0.0348, loss_iou2d_dn_1: 0.8942, loss_box2d_dn_1: 0.7051, loss_alpha2d_dn_1: 0.2751, loss_cls2d_dn_2: 0.0210, loss_iou2d_dn_2: 0.8359, loss_box2d_dn_2: 0.5536, loss_alpha2d_dn_2: 0.1918, loss_dense_depth: 1.3167, loss: 51.5888, grad_norm: 633.4407 2024-09-16 04:02:39,163 - mmdet - INFO - Saving checkpoint at 16701 iterations 2024-09-16 04:03:28,708 - mmdet - INFO - Iter [16750/87900] lr: 1.826e-04, eta: 18:29:32, time: 1.009, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2441, loss_box3d_0: 2.4481, loss_cns3d_0: 0.5397, loss_yns3d_0: 0.1727, loss_cls3d_1: 1.2184, loss_box3d_1: 2.5958, loss_cns3d_1: 0.5297, loss_yns3d_1: 0.1692, loss_cls3d_2: 1.1996, loss_box3d_2: 2.4362, loss_cns3d_2: 0.5674, loss_yns3d_2: 0.1677, loss_cls3d_3: 1.1944, loss_box3d_3: 2.3872, loss_cns3d_3: 0.5740, loss_yns3d_3: 0.1664, loss_cls3d_4: 1.1667, loss_box3d_4: 2.3320, loss_cns3d_4: 0.5791, loss_yns3d_4: 0.1653, loss_cls3d_5: 1.1682, loss_box3d_5: 2.3113, loss_cns3d_5: 0.5791, loss_yns3d_5: 0.1643, loss_cls2d_0: 1.2500, loss_iou2d_0: 1.3130, loss_box2d_0: 0.8552, loss_alpha2d_0: 0.6307, loss_cls2d_1: 1.1600, loss_iou2d_1: 1.2887, loss_box2d_1: 0.8146, loss_alpha2d_1: 0.6267, loss_cls2d_2: 1.1184, loss_iou2d_2: 1.2136, loss_box2d_2: 0.7317, loss_alpha2d_2: 0.6218, loss_cls3d_dn_0: 0.2892, loss_box3d_dn_0: 1.0342, loss_cls3d_dn_1: 0.3549, loss_box3d_dn_1: 0.8746, loss_cls3d_dn_2: 0.1785, loss_box3d_dn_2: 0.8098, loss_cls3d_dn_3: 0.1873, loss_box3d_dn_3: 0.7776, loss_cls3d_dn_4: 0.1411, loss_box3d_dn_4: 0.7488, loss_cls3d_dn_5: 0.1378, loss_box3d_dn_5: 0.7370, loss_cls2d_dn_0: 0.0525, loss_iou2d_dn_0: 0.9563, loss_box2d_dn_0: 0.9364, loss_alpha2d_dn_0: 0.3148, loss_cls2d_dn_1: 0.0354, loss_iou2d_dn_1: 0.8702, loss_box2d_dn_1: 0.6892, loss_alpha2d_dn_1: 0.2695, loss_cls2d_dn_2: 0.0210, loss_iou2d_dn_2: 0.8025, loss_box2d_dn_2: 0.5374, loss_alpha2d_dn_2: 0.1809, loss_dense_depth: 1.3352, loss: 50.9729, grad_norm: inf 2024-09-16 04:04:16,803 - mmdet - INFO - Iter [16800/87900] lr: 1.825e-04, eta: 18:28:50, time: 0.962, data_time: 0.038, memory: 18408, loss_cls3d_0: 1.2444, loss_box3d_0: 2.2391, loss_cns3d_0: 0.5939, loss_yns3d_0: 0.1721, loss_cls3d_1: 1.2246, loss_box3d_1: 2.5141, loss_cns3d_1: 0.5438, loss_yns3d_1: 0.1699, loss_cls3d_2: 1.1945, loss_box3d_2: 2.4228, loss_cns3d_2: 0.5622, loss_yns3d_2: 0.1692, loss_cls3d_3: 1.1906, loss_box3d_3: 2.3656, loss_cns3d_3: 0.5722, loss_yns3d_3: 0.1684, loss_cls3d_4: 1.1617, loss_box3d_4: 2.3081, loss_cns3d_4: 0.5773, loss_yns3d_4: 0.1657, loss_cls3d_5: 1.1580, loss_box3d_5: 2.2835, loss_cns3d_5: 0.5780, loss_yns3d_5: 0.1644, loss_cls2d_0: 1.2610, loss_iou2d_0: 1.3330, loss_box2d_0: 0.8662, loss_alpha2d_0: 0.6347, loss_cls2d_1: 1.1710, loss_iou2d_1: 1.2973, loss_box2d_1: 0.7913, loss_alpha2d_1: 0.6334, loss_cls2d_2: 1.1168, loss_iou2d_2: 1.1960, loss_box2d_2: 0.7136, loss_alpha2d_2: 0.6264, loss_cls3d_dn_0: 0.3037, loss_box3d_dn_0: 0.9996, loss_cls3d_dn_1: 0.3582, loss_box3d_dn_1: 0.8379, loss_cls3d_dn_2: 0.1810, loss_box3d_dn_2: 0.7978, loss_cls3d_dn_3: 0.1877, loss_box3d_dn_3: 0.7719, loss_cls3d_dn_4: 0.1413, loss_box3d_dn_4: 0.7460, loss_cls3d_dn_5: 0.1402, loss_box3d_dn_5: 0.7367, loss_cls2d_dn_0: 0.0540, loss_iou2d_dn_0: 0.9479, loss_box2d_dn_0: 0.9352, loss_alpha2d_dn_0: 0.3162, loss_cls2d_dn_1: 0.0325, loss_iou2d_dn_1: 0.8745, loss_box2d_dn_1: 0.7102, loss_alpha2d_dn_1: 0.2725, loss_cls2d_dn_2: 0.0204, loss_iou2d_dn_2: 0.8090, loss_box2d_dn_2: 0.5476, loss_alpha2d_dn_2: 0.1848, loss_dense_depth: 1.2800, loss: 50.5714, grad_norm: nan 2024-09-16 04:05:01,647 - mmdet - INFO - Iter [16850/87900] lr: 1.824e-04, eta: 18:27:55, time: 0.897, data_time: 0.038, memory: 18408, loss_cls3d_0: 1.2441, loss_box3d_0: 2.2326, loss_cns3d_0: 0.6062, loss_yns3d_0: 0.1722, loss_cls3d_1: 1.2234, loss_box3d_1: 2.7394, loss_cns3d_1: 0.5381, loss_yns3d_1: 0.1684, loss_cls3d_2: 1.2070, loss_box3d_2: 2.7073, loss_cns3d_2: 0.5435, loss_yns3d_2: 0.1646, loss_cls3d_3: 1.2010, loss_box3d_3: 2.5949, loss_cns3d_3: 0.5649, loss_yns3d_3: 0.1646, loss_cls3d_4: 1.1840, loss_box3d_4: 2.5000, loss_cns3d_4: 0.5735, loss_yns3d_4: 0.1614, loss_cls3d_5: 1.1794, loss_box3d_5: 2.4525, loss_cns3d_5: 0.5769, loss_yns3d_5: 0.1608, loss_cls2d_0: 1.2727, loss_iou2d_0: 1.3310, loss_box2d_0: 0.8705, loss_alpha2d_0: 0.6269, loss_cls2d_1: 1.1989, loss_iou2d_1: 1.3289, loss_box2d_1: 0.8202, loss_alpha2d_1: 0.6260, loss_cls2d_2: 1.1241, loss_iou2d_2: 1.2206, loss_box2d_2: 0.7264, loss_alpha2d_2: 0.6183, loss_cls3d_dn_0: 0.2908, loss_box3d_dn_0: 1.0047, loss_cls3d_dn_1: 0.3669, loss_box3d_dn_1: 0.8960, loss_cls3d_dn_2: 0.1886, loss_box3d_dn_2: 0.8395, loss_cls3d_dn_3: 0.1925, loss_box3d_dn_3: 0.7974, loss_cls3d_dn_4: 0.1439, loss_box3d_dn_4: 0.7686, loss_cls3d_dn_5: 0.1435, loss_box3d_dn_5: 0.7545, loss_cls2d_dn_0: 0.0541, loss_iou2d_dn_0: 0.9322, loss_box2d_dn_0: 0.8950, loss_alpha2d_dn_0: 0.3133, loss_cls2d_dn_1: 0.0323, loss_iou2d_dn_1: 0.9025, loss_box2d_dn_1: 0.7621, loss_alpha2d_dn_1: 0.2829, loss_cls2d_dn_2: 0.0201, loss_iou2d_dn_2: 0.8150, loss_box2d_dn_2: 0.5625, loss_alpha2d_dn_2: 0.1784, loss_dense_depth: 1.3649, loss: 52.1276, grad_norm: inf 2024-09-16 04:05:46,092 - mmdet - INFO - Iter [16900/87900] lr: 1.823e-04, eta: 18:26:59, time: 0.889, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2241, loss_box3d_0: 2.1160, loss_cns3d_0: 0.6071, loss_yns3d_0: 0.1712, loss_cls3d_1: 1.1861, loss_box3d_1: 2.4646, loss_cns3d_1: 0.5504, loss_yns3d_1: 0.1695, loss_cls3d_2: 1.1867, loss_box3d_2: 2.4212, loss_cns3d_2: 0.5687, loss_yns3d_2: 0.1698, loss_cls3d_3: 1.1741, loss_box3d_3: 2.3439, loss_cns3d_3: 0.5749, loss_yns3d_3: 0.1689, loss_cls3d_4: 1.1560, loss_box3d_4: 2.2623, loss_cns3d_4: 0.5892, loss_yns3d_4: 0.1673, loss_cls3d_5: 1.1473, loss_box3d_5: 2.2341, loss_cns3d_5: 0.5881, loss_yns3d_5: 0.1657, loss_cls2d_0: 1.2231, loss_iou2d_0: 1.3168, loss_box2d_0: 0.8616, loss_alpha2d_0: 0.6288, loss_cls2d_1: 1.1454, loss_iou2d_1: 1.2859, loss_box2d_1: 0.7859, loss_alpha2d_1: 0.6291, loss_cls2d_2: 1.0967, loss_iou2d_2: 1.1876, loss_box2d_2: 0.7074, loss_alpha2d_2: 0.6202, loss_cls3d_dn_0: 0.2858, loss_box3d_dn_0: 1.0008, loss_cls3d_dn_1: 0.3472, loss_box3d_dn_1: 0.8638, loss_cls3d_dn_2: 0.1851, loss_box3d_dn_2: 0.7901, loss_cls3d_dn_3: 0.2029, loss_box3d_dn_3: 0.7607, loss_cls3d_dn_4: 0.1317, loss_box3d_dn_4: 0.7349, loss_cls3d_dn_5: 0.1254, loss_box3d_dn_5: 0.7222, loss_cls2d_dn_0: 0.0515, loss_iou2d_dn_0: 0.9441, loss_box2d_dn_0: 0.8869, loss_alpha2d_dn_0: 0.3139, loss_cls2d_dn_1: 0.0302, loss_iou2d_dn_1: 0.8812, loss_box2d_dn_1: 0.7029, loss_alpha2d_dn_1: 0.2791, loss_cls2d_dn_2: 0.0206, loss_iou2d_dn_2: 0.8107, loss_box2d_dn_2: 0.5462, loss_alpha2d_dn_2: 0.1843, loss_dense_depth: 1.2832, loss: 49.9813, grad_norm: nan 2024-09-16 04:06:32,753 - mmdet - INFO - Iter [16950/87900] lr: 1.822e-04, eta: 18:26:11, time: 0.933, data_time: 0.034, memory: 18408, loss_cls3d_0: 1.2239, loss_box3d_0: 2.1503, loss_cns3d_0: 0.6020, loss_yns3d_0: 0.1702, loss_cls3d_1: 1.1974, loss_box3d_1: 2.5008, loss_cns3d_1: 0.5463, loss_yns3d_1: 0.1694, loss_cls3d_2: 1.1816, loss_box3d_2: 2.4162, loss_cns3d_2: 0.5679, loss_yns3d_2: 0.1694, loss_cls3d_3: 1.1691, loss_box3d_3: 2.3338, loss_cns3d_3: 0.5821, loss_yns3d_3: 0.1691, loss_cls3d_4: 1.1395, loss_box3d_4: 2.2609, loss_cns3d_4: 0.5878, loss_yns3d_4: 0.1661, loss_cls3d_5: 1.1325, loss_box3d_5: 2.2320, loss_cns3d_5: 0.5882, loss_yns3d_5: 0.1650, loss_cls2d_0: 1.2381, loss_iou2d_0: 1.2985, loss_box2d_0: 0.8220, loss_alpha2d_0: 0.6285, loss_cls2d_1: 1.1441, loss_iou2d_1: 1.2567, loss_box2d_1: 0.7457, loss_alpha2d_1: 0.6292, loss_cls2d_2: 1.0903, loss_iou2d_2: 1.1558, loss_box2d_2: 0.6655, loss_alpha2d_2: 0.6198, loss_cls3d_dn_0: 0.2883, loss_box3d_dn_0: 0.9947, loss_cls3d_dn_1: 0.3486, loss_box3d_dn_1: 0.8512, loss_cls3d_dn_2: 0.1740, loss_box3d_dn_2: 0.7794, loss_cls3d_dn_3: 0.1795, loss_box3d_dn_3: 0.7502, loss_cls3d_dn_4: 0.1359, loss_box3d_dn_4: 0.7281, loss_cls3d_dn_5: 0.1307, loss_box3d_dn_5: 0.7160, loss_cls2d_dn_0: 0.0526, loss_iou2d_dn_0: 0.9363, loss_box2d_dn_0: 0.8529, loss_alpha2d_dn_0: 0.3146, loss_cls2d_dn_1: 0.0337, loss_iou2d_dn_1: 0.8636, loss_box2d_dn_1: 0.6705, loss_alpha2d_dn_1: 0.2834, loss_cls2d_dn_2: 0.0186, loss_iou2d_dn_2: 0.7876, loss_box2d_dn_2: 0.4912, loss_alpha2d_dn_2: 0.1681, loss_dense_depth: 1.2615, loss: 49.5268, grad_norm: 961.8456 2024-09-16 04:07:17,720 - mmdet - INFO - Exp name: simpb_nus_r50_uimg_704x256.py 2024-09-16 04:07:17,720 - mmdet - INFO - Iter [17000/87900] lr: 1.821e-04, eta: 18:25:17, time: 0.900, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2132, loss_box3d_0: 2.0938, loss_cns3d_0: 0.6089, loss_yns3d_0: 0.1713, loss_cls3d_1: 1.1928, loss_box3d_1: 2.4051, loss_cns3d_1: 0.5527, loss_yns3d_1: 0.1699, loss_cls3d_2: 1.1901, loss_box3d_2: 2.3381, loss_cns3d_2: 0.5674, loss_yns3d_2: 0.1693, loss_cls3d_3: 1.1719, loss_box3d_3: 2.2637, loss_cns3d_3: 0.5791, loss_yns3d_3: 0.1689, loss_cls3d_4: 1.1333, loss_box3d_4: 2.1819, loss_cns3d_4: 0.5877, loss_yns3d_4: 0.1660, loss_cls3d_5: 1.1270, loss_box3d_5: 2.1548, loss_cns3d_5: 0.5891, loss_yns3d_5: 0.1650, loss_cls2d_0: 1.2315, loss_iou2d_0: 1.2906, loss_box2d_0: 0.8277, loss_alpha2d_0: 0.6309, loss_cls2d_1: 1.1474, loss_iou2d_1: 1.2308, loss_box2d_1: 0.7558, loss_alpha2d_1: 0.6325, loss_cls2d_2: 1.0816, loss_iou2d_2: 1.1395, loss_box2d_2: 0.6669, loss_alpha2d_2: 0.6203, loss_cls3d_dn_0: 0.2841, loss_box3d_dn_0: 1.0014, loss_cls3d_dn_1: 0.3486, loss_box3d_dn_1: 0.8959, loss_cls3d_dn_2: 0.1919, loss_box3d_dn_2: 0.8328, loss_cls3d_dn_3: 0.2020, loss_box3d_dn_3: 0.7812, loss_cls3d_dn_4: 0.1382, loss_box3d_dn_4: 0.7416, loss_cls3d_dn_5: 0.1365, loss_box3d_dn_5: 0.7259, loss_cls2d_dn_0: 0.0525, loss_iou2d_dn_0: 0.9268, loss_box2d_dn_0: 0.8592, loss_alpha2d_dn_0: 0.3147, loss_cls2d_dn_1: 0.0374, loss_iou2d_dn_1: 0.8716, loss_box2d_dn_1: 0.7456, loss_alpha2d_dn_1: 0.2904, loss_cls2d_dn_2: 0.0197, loss_iou2d_dn_2: 0.7917, loss_box2d_dn_2: 0.5334, loss_alpha2d_dn_2: 0.1760, loss_dense_depth: 1.2097, loss: 49.3255, grad_norm: nan 2024-09-16 04:08:02,099 - mmdet - INFO - Iter [17050/87900] lr: 1.820e-04, eta: 18:24:20, time: 0.888, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2444, loss_box3d_0: 2.1264, loss_cns3d_0: 0.6106, loss_yns3d_0: 0.1684, loss_cls3d_1: 1.2243, loss_box3d_1: 2.4172, loss_cns3d_1: 0.5578, loss_yns3d_1: 0.1653, loss_cls3d_2: 1.2079, loss_box3d_2: 2.3387, loss_cns3d_2: 0.5791, loss_yns3d_2: 0.1656, loss_cls3d_3: 1.1882, loss_box3d_3: 2.2603, loss_cns3d_3: 0.5879, loss_yns3d_3: 0.1655, loss_cls3d_4: 1.1447, loss_box3d_4: 2.1921, loss_cns3d_4: 0.5920, loss_yns3d_4: 0.1634, loss_cls3d_5: 1.1437, loss_box3d_5: 2.1593, loss_cns3d_5: 0.5947, loss_yns3d_5: 0.1627, loss_cls2d_0: 1.2542, loss_iou2d_0: 1.3098, loss_box2d_0: 0.8520, loss_alpha2d_0: 0.6306, loss_cls2d_1: 1.1758, loss_iou2d_1: 1.2690, loss_box2d_1: 0.7999, loss_alpha2d_1: 0.6307, loss_cls2d_2: 1.1085, loss_iou2d_2: 1.1727, loss_box2d_2: 0.6949, loss_alpha2d_2: 0.6194, loss_cls3d_dn_0: 0.2893, loss_box3d_dn_0: 1.0135, loss_cls3d_dn_1: 0.3562, loss_box3d_dn_1: 0.8891, loss_cls3d_dn_2: 0.1834, loss_box3d_dn_2: 0.8134, loss_cls3d_dn_3: 0.1983, loss_box3d_dn_3: 0.7712, loss_cls3d_dn_4: 0.1434, loss_box3d_dn_4: 0.7391, loss_cls3d_dn_5: 0.1415, loss_box3d_dn_5: 0.7247, loss_cls2d_dn_0: 0.0533, loss_iou2d_dn_0: 0.9414, loss_box2d_dn_0: 0.9084, loss_alpha2d_dn_0: 0.3156, loss_cls2d_dn_1: 0.0360, loss_iou2d_dn_1: 0.8888, loss_box2d_dn_1: 0.7731, loss_alpha2d_dn_1: 0.2957, loss_cls2d_dn_2: 0.0218, loss_iou2d_dn_2: 0.8089, loss_box2d_dn_2: 0.5671, loss_alpha2d_dn_2: 0.1723, loss_dense_depth: 1.2511, loss: 49.9743, grad_norm: nan 2024-09-16 04:08:48,364 - mmdet - INFO - Iter [17100/87900] lr: 1.819e-04, eta: 18:23:31, time: 0.925, data_time: 0.037, memory: 18408, loss_cls3d_0: 1.2345, loss_box3d_0: 2.1286, loss_cns3d_0: 0.5879, loss_yns3d_0: 0.1738, loss_cls3d_1: 1.2118, loss_box3d_1: 2.2996, loss_cns3d_1: 0.5705, loss_yns3d_1: 0.1701, loss_cls3d_2: 1.1899, loss_box3d_2: 2.2424, loss_cns3d_2: 0.5861, loss_yns3d_2: 0.1707, loss_cls3d_3: 1.1628, loss_box3d_3: 2.1820, loss_cns3d_3: 0.5940, loss_yns3d_3: 0.1699, loss_cls3d_4: 1.1324, loss_box3d_4: 2.1171, loss_cns3d_4: 0.5956, loss_yns3d_4: 0.1660, loss_cls3d_5: 1.1284, loss_box3d_5: 2.0951, loss_cns3d_5: 0.5942, loss_yns3d_5: 0.1656, loss_cls2d_0: 1.2285, loss_iou2d_0: 1.2546, loss_box2d_0: 0.8252, loss_alpha2d_0: 0.6298, loss_cls2d_1: 1.1468, loss_iou2d_1: 1.1892, loss_box2d_1: 0.7559, loss_alpha2d_1: 0.6314, loss_cls2d_2: 1.0870, loss_iou2d_2: 1.1013, loss_box2d_2: 0.6649, loss_alpha2d_2: 0.6168, loss_cls3d_dn_0: 0.2725, loss_box3d_dn_0: 1.0139, loss_cls3d_dn_1: 0.3510, loss_box3d_dn_1: 0.8491, loss_cls3d_dn_2: 0.1812, loss_box3d_dn_2: 0.8051, loss_cls3d_dn_3: 0.1853, loss_box3d_dn_3: 0.7720, loss_cls3d_dn_4: 0.1333, loss_box3d_dn_4: 0.7374, loss_cls3d_dn_5: 0.1328, loss_box3d_dn_5: 0.7223, loss_cls2d_dn_0: 0.0519, loss_iou2d_dn_0: 0.9011, loss_box2d_dn_0: 0.8761, loss_alpha2d_dn_0: 0.3151, loss_cls2d_dn_1: 0.0373, loss_iou2d_dn_1: 0.8504, loss_box2d_dn_1: 0.7228, loss_alpha2d_dn_1: 0.2863, loss_cls2d_dn_2: 0.0191, loss_iou2d_dn_2: 0.7757, loss_box2d_dn_2: 0.5281, loss_alpha2d_dn_2: 0.1652, loss_dense_depth: 1.2272, loss: 48.7122, grad_norm: 405.0991 2024-09-16 04:09:36,893 - mmdet - INFO - Iter [17150/87900] lr: 1.818e-04, eta: 18:22:52, time: 0.971, data_time: 0.035, memory: 18408, loss_cls3d_0: 1.2419, loss_box3d_0: 2.1284, loss_cns3d_0: 0.6110, loss_yns3d_0: 0.1722, loss_cls3d_1: 1.2490, loss_box3d_1: 2.4348, loss_cns3d_1: 0.5696, loss_yns3d_1: 0.1730, loss_cls3d_2: 1.2006, loss_box3d_2: 2.5202, loss_cns3d_2: 0.5385, loss_yns3d_2: 0.1725, loss_cls3d_3: 1.1898, loss_box3d_3: 2.3125, loss_cns3d_3: 0.5896, loss_yns3d_3: 0.1699, loss_cls3d_4: 1.1602, loss_box3d_4: 2.2436, loss_cns3d_4: 0.5942, loss_yns3d_4: 0.1621, loss_cls3d_5: 1.1512, loss_box3d_5: 2.2157, loss_cns3d_5: 0.5951, loss_yns3d_5: 0.1624, loss_cls2d_0: 1.2435, loss_iou2d_0: 1.2767, loss_box2d_0: 0.8311, loss_alpha2d_0: 0.6312, loss_cls2d_1: 1.1723, loss_iou2d_1: 1.2350, loss_box2d_1: 0.7680, loss_alpha2d_1: 0.6331, loss_cls2d_2: 1.1083, loss_iou2d_2: 1.1555, loss_box2d_2: 0.6936, loss_alpha2d_2: 0.6155, loss_cls3d_dn_0: 0.2575, loss_box3d_dn_0: 1.0014, loss_cls3d_dn_1: 0.3730, loss_box3d_dn_1: 0.8665, loss_cls3d_dn_2: 0.2045, loss_box3d_dn_2: 0.8784, loss_cls3d_dn_3: 0.1996, loss_box3d_dn_3: 0.7979, loss_cls3d_dn_4: 0.1404, loss_box3d_dn_4: 0.7560, loss_cls3d_dn_5: 0.1368, loss_box3d_dn_5: 0.7324, loss_cls2d_dn_0: 0.0535, loss_iou2d_dn_0: 0.9237, loss_box2d_dn_0: 0.8812, loss_alpha2d_dn_0: 0.3150, loss_cls2d_dn_1: 0.0456, loss_iou2d_dn_1: 0.8918, loss_box2d_dn_1: 0.7579, loss_alpha2d_dn_1: 0.2887, loss_cls2d_dn_2: 0.0209, loss_iou2d_dn_2: 0.7969, loss_box2d_dn_2: 0.5559, loss_alpha2d_dn_2: 0.1771, loss_dense_depth: 1.2931, loss: 50.2675, grad_norm: inf 2024-09-16 04:10:25,268 - mmdet - INFO - Iter [17200/87900] lr: 1.817e-04, eta: 18:22:12, time: 0.968, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2356, loss_box3d_0: 2.1826, loss_cns3d_0: 0.6064, loss_yns3d_0: 0.1717, loss_cls3d_1: 1.2149, loss_box3d_1: 2.4082, loss_cns3d_1: 0.5558, loss_yns3d_1: 0.1703, loss_cls3d_2: 1.1830, loss_box3d_2: 2.3872, loss_cns3d_2: 0.5583, loss_yns3d_2: 0.1686, loss_cls3d_3: 1.1660, loss_box3d_3: 2.2541, loss_cns3d_3: 0.5845, loss_yns3d_3: 0.1681, loss_cls3d_4: 1.1229, loss_box3d_4: 2.1591, loss_cns3d_4: 0.5945, loss_yns3d_4: 0.1626, loss_cls3d_5: 1.1152, loss_box3d_5: 2.1399, loss_cns3d_5: 0.5943, loss_yns3d_5: 0.1610, loss_cls2d_0: 1.2399, loss_iou2d_0: 1.3029, loss_box2d_0: 0.8235, loss_alpha2d_0: 0.6332, loss_cls2d_1: 1.1481, loss_iou2d_1: 1.2716, loss_box2d_1: 0.7796, loss_alpha2d_1: 0.6335, loss_cls2d_2: 1.0703, loss_iou2d_2: 1.1548, loss_box2d_2: 0.6736, loss_alpha2d_2: 0.6206, loss_cls3d_dn_0: 0.2840, loss_box3d_dn_0: 1.0189, loss_cls3d_dn_1: 0.3695, loss_box3d_dn_1: 0.8726, loss_cls3d_dn_2: 0.1961, loss_box3d_dn_2: 0.8470, loss_cls3d_dn_3: 0.2005, loss_box3d_dn_3: 0.7961, loss_cls3d_dn_4: 0.1389, loss_box3d_dn_4: 0.7453, loss_cls3d_dn_5: 0.1355, loss_box3d_dn_5: 0.7266, loss_cls2d_dn_0: 0.0527, loss_iou2d_dn_0: 0.9347, loss_box2d_dn_0: 0.8592, loss_alpha2d_dn_0: 0.3160, loss_cls2d_dn_1: 0.0452, loss_iou2d_dn_1: 0.9130, loss_box2d_dn_1: 0.8032, loss_alpha2d_dn_1: 0.2964, loss_cls2d_dn_2: 0.0198, loss_iou2d_dn_2: 0.8036, loss_box2d_dn_2: 0.5426, loss_alpha2d_dn_2: 0.1707, loss_dense_depth: 1.3166, loss: 49.8210, grad_norm: nan 2024-09-16 04:11:10,219 - mmdet - INFO - Iter [17250/87900] lr: 1.816e-04, eta: 18:21:18, time: 0.899, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2322, loss_box3d_0: 2.3844, loss_cns3d_0: 0.5477, loss_yns3d_0: 0.1728, loss_cls3d_1: 1.2103, loss_box3d_1: 2.5234, loss_cns3d_1: 0.5311, loss_yns3d_1: 0.1720, loss_cls3d_2: 1.1889, loss_box3d_2: 2.4949, loss_cns3d_2: 0.5405, loss_yns3d_2: 0.1719, loss_cls3d_3: 1.1674, loss_box3d_3: 2.3656, loss_cns3d_3: 0.5623, loss_yns3d_3: 0.1710, loss_cls3d_4: 1.1140, loss_box3d_4: 2.2317, loss_cns3d_4: 0.5783, loss_yns3d_4: 0.1663, loss_cls3d_5: 1.1019, loss_box3d_5: 2.1944, loss_cns3d_5: 0.5819, loss_yns3d_5: 0.1656, loss_cls2d_0: 1.2427, loss_iou2d_0: 1.3193, loss_box2d_0: 0.8570, loss_alpha2d_0: 0.6334, loss_cls2d_1: 1.1409, loss_iou2d_1: 1.2681, loss_box2d_1: 0.7876, loss_alpha2d_1: 0.6320, loss_cls2d_2: 1.0788, loss_iou2d_2: 1.1517, loss_box2d_2: 0.6691, loss_alpha2d_2: 0.6232, loss_cls3d_dn_0: 0.2821, loss_box3d_dn_0: 1.0761, loss_cls3d_dn_1: 0.3724, loss_box3d_dn_1: 0.9803, loss_cls3d_dn_2: 0.2419, loss_box3d_dn_2: 0.9395, loss_cls3d_dn_3: 0.2374, loss_box3d_dn_3: 0.8676, loss_cls3d_dn_4: 0.1540, loss_box3d_dn_4: 0.8033, loss_cls3d_dn_5: 0.1536, loss_box3d_dn_5: 0.7818, loss_cls2d_dn_0: 0.0521, loss_iou2d_dn_0: 0.9433, loss_box2d_dn_0: 0.8950, loss_alpha2d_dn_0: 0.3157, loss_cls2d_dn_1: 0.0460, loss_iou2d_dn_1: 0.9406, loss_box2d_dn_1: 0.9186, loss_alpha2d_dn_1: 0.2964, loss_cls2d_dn_2: 0.0222, loss_iou2d_dn_2: 0.8359, loss_box2d_dn_2: 0.6156, loss_alpha2d_dn_2: 0.1872, loss_dense_depth: 1.2664, loss: 51.1993, grad_norm: 605.4656 2024-09-16 04:11:54,619 - mmdet - INFO - Iter [17300/87900] lr: 1.815e-04, eta: 18:20:21, time: 0.888, data_time: 0.034, memory: 18408, loss_cls3d_0: 1.2467, loss_box3d_0: 2.5773, loss_cns3d_0: 0.5239, loss_yns3d_0: 0.1716, loss_cls3d_1: 1.2500, loss_box3d_1: 2.8479, loss_cns3d_1: 0.4934, loss_yns3d_1: 0.1687, loss_cls3d_2: 1.2396, loss_box3d_2: 2.7518, loss_cns3d_2: 0.5196, loss_yns3d_2: 0.1700, loss_cls3d_3: 1.2118, loss_box3d_3: 2.5731, loss_cns3d_3: 0.5497, loss_yns3d_3: 0.1672, loss_cls3d_4: 1.1661, loss_box3d_4: 2.3859, loss_cns3d_4: 0.5724, loss_yns3d_4: 0.1666, loss_cls3d_5: 1.1570, loss_box3d_5: 2.3446, loss_cns3d_5: 0.5776, loss_yns3d_5: 0.1653, loss_cls2d_0: 1.2773, loss_iou2d_0: 1.3271, loss_box2d_0: 0.8550, loss_alpha2d_0: 0.6289, loss_cls2d_1: 1.2092, loss_iou2d_1: 1.3300, loss_box2d_1: 0.8269, loss_alpha2d_1: 0.6285, loss_cls2d_2: 1.1313, loss_iou2d_2: 1.1950, loss_box2d_2: 0.6860, loss_alpha2d_2: 0.6205, loss_cls3d_dn_0: 0.2766, loss_box3d_dn_0: 1.0559, loss_cls3d_dn_1: 0.3748, loss_box3d_dn_1: 1.0496, loss_cls3d_dn_2: 0.2640, loss_box3d_dn_2: 0.9841, loss_cls3d_dn_3: 0.2482, loss_box3d_dn_3: 0.9014, loss_cls3d_dn_4: 0.1628, loss_box3d_dn_4: 0.8327, loss_cls3d_dn_5: 0.1588, loss_box3d_dn_5: 0.8096, loss_cls2d_dn_0: 0.0541, loss_iou2d_dn_0: 0.9417, loss_box2d_dn_0: 0.8716, loss_alpha2d_dn_0: 0.3143, loss_cls2d_dn_1: 0.0497, loss_iou2d_dn_1: 0.9666, loss_box2d_dn_1: 0.9289, loss_alpha2d_dn_1: 0.2973, loss_cls2d_dn_2: 0.0223, loss_iou2d_dn_2: 0.8445, loss_box2d_dn_2: 0.5829, loss_alpha2d_dn_2: 0.2275, loss_dense_depth: 1.2412, loss: 53.1743, grad_norm: nan 2024-09-16 04:12:40,228 - mmdet - INFO - Iter [17350/87900] lr: 1.814e-04, eta: 18:19:30, time: 0.912, data_time: 0.038, memory: 18408, loss_cls3d_0: 1.2249, loss_box3d_0: 2.3946, loss_cns3d_0: 0.5801, loss_yns3d_0: 0.1730, loss_cls3d_1: 1.2229, loss_box3d_1: 2.9858, loss_cns3d_1: 0.4580, loss_yns3d_1: 0.1711, loss_cls3d_2: 1.2120, loss_box3d_2: 2.8907, loss_cns3d_2: 0.4901, loss_yns3d_2: 0.1717, loss_cls3d_3: 1.1916, loss_box3d_3: 2.7044, loss_cns3d_3: 0.5217, loss_yns3d_3: 0.1702, loss_cls3d_4: 1.1438, loss_box3d_4: 2.5123, loss_cns3d_4: 0.5548, loss_yns3d_4: 0.1686, loss_cls3d_5: 1.1315, loss_box3d_5: 2.4695, loss_cns3d_5: 0.5580, loss_yns3d_5: 0.1682, loss_cls2d_0: 1.2544, loss_iou2d_0: 1.3307, loss_box2d_0: 0.9038, loss_alpha2d_0: 0.6282, loss_cls2d_1: 1.1631, loss_iou2d_1: 1.3774, loss_box2d_1: 0.9343, loss_alpha2d_1: 0.6279, loss_cls2d_2: 1.1104, loss_iou2d_2: 1.2591, loss_box2d_2: 0.8075, loss_alpha2d_2: 0.6231, loss_cls3d_dn_0: 0.2820, loss_box3d_dn_0: 1.0724, loss_cls3d_dn_1: 0.3704, loss_box3d_dn_1: 1.0859, loss_cls3d_dn_2: 0.2727, loss_box3d_dn_2: 1.0313, loss_cls3d_dn_3: 0.2599, loss_box3d_dn_3: 0.9480, loss_cls3d_dn_4: 0.1610, loss_box3d_dn_4: 0.8745, loss_cls3d_dn_5: 0.1572, loss_box3d_dn_5: 0.8535, loss_cls2d_dn_0: 0.0537, loss_iou2d_dn_0: 0.9504, loss_box2d_dn_0: 0.9250, loss_alpha2d_dn_0: 0.3153, loss_cls2d_dn_1: 0.0506, loss_iou2d_dn_1: 0.9839, loss_box2d_dn_1: 1.0537, loss_alpha2d_dn_1: 0.2969, loss_cls2d_dn_2: 0.0224, loss_iou2d_dn_2: 0.8466, loss_box2d_dn_2: 0.6942, loss_alpha2d_dn_2: 0.2101, loss_dense_depth: 1.3433, loss: 54.4043, grad_norm: nan 2024-09-16 04:13:24,521 - mmdet - INFO - Iter [17400/87900] lr: 1.813e-04, eta: 18:18:33, time: 0.885, data_time: 0.034, memory: 18408, loss_cls3d_0: 1.2441, loss_box3d_0: 2.7452, loss_cns3d_0: 0.4698, loss_yns3d_0: 0.1738, loss_cls3d_1: 1.2341, loss_box3d_1: 2.8508, loss_cns3d_1: 0.4922, loss_yns3d_1: 0.1698, loss_cls3d_2: 1.2416, loss_box3d_2: 2.7750, loss_cns3d_2: 0.5173, loss_yns3d_2: 0.1703, loss_cls3d_3: 1.2446, loss_box3d_3: 2.5958, loss_cns3d_3: 0.5492, loss_yns3d_3: 0.1681, loss_cls3d_4: 1.1643, loss_box3d_4: 2.4384, loss_cns3d_4: 0.5691, loss_yns3d_4: 0.1662, loss_cls3d_5: 1.1586, loss_box3d_5: 2.3925, loss_cns3d_5: 0.5737, loss_yns3d_5: 0.1655, loss_cls2d_0: 1.2597, loss_iou2d_0: 1.3282, loss_box2d_0: 0.8562, loss_alpha2d_0: 0.6296, loss_cls2d_1: 1.1829, loss_iou2d_1: 1.3517, loss_box2d_1: 0.8526, loss_alpha2d_1: 0.6294, loss_cls2d_2: 1.1097, loss_iou2d_2: 1.2174, loss_box2d_2: 0.7153, loss_alpha2d_2: 0.6205, loss_cls3d_dn_0: 0.2970, loss_box3d_dn_0: 1.1572, loss_cls3d_dn_1: 0.3888, loss_box3d_dn_1: 1.0784, loss_cls3d_dn_2: 0.3129, loss_box3d_dn_2: 1.0038, loss_cls3d_dn_3: 0.3002, loss_box3d_dn_3: 0.9089, loss_cls3d_dn_4: 0.1663, loss_box3d_dn_4: 0.8420, loss_cls3d_dn_5: 0.1607, loss_box3d_dn_5: 0.8218, loss_cls2d_dn_0: 0.0532, loss_iou2d_dn_0: 0.9546, loss_box2d_dn_0: 0.8785, loss_alpha2d_dn_0: 0.3136, loss_cls2d_dn_1: 0.0527, loss_iou2d_dn_1: 0.9868, loss_box2d_dn_1: 1.0124, loss_alpha2d_dn_1: 0.2958, loss_cls2d_dn_2: 0.0249, loss_iou2d_dn_2: 0.8663, loss_box2d_dn_2: 0.6533, loss_alpha2d_dn_2: 0.2279, loss_dense_depth: 1.2900, loss: 54.0708, grad_norm: nan 2024-09-16 04:14:09,109 - mmdet - INFO - Iter [17450/87900] lr: 1.812e-04, eta: 18:17:38, time: 0.892, data_time: 0.036, memory: 18408, loss_cls3d_0: 1.2196, loss_box3d_0: 2.7287, loss_cns3d_0: 0.4555, loss_yns3d_0: 0.1736, loss_cls3d_1: 1.2552, loss_box3d_1: 2.6298, loss_cns3d_1: 0.5394, loss_yns3d_1: 0.1671, loss_cls3d_2: 1.2450, loss_box3d_2: 2.6554, loss_cns3d_2: 0.5451, loss_yns3d_2: 0.1689, loss_cls3d_3: 1.2115, loss_box3d_3: 2.5192, loss_cns3d_3: 0.5742, loss_yns3d_3: 0.1668, loss_cls3d_4: 1.1433, loss_box3d_4: 2.3777, loss_cns3d_4: 0.5897, loss_yns3d_4: 0.1641, loss_cls3d_5: 1.1303, loss_box3d_5: 2.3324, loss_cns3d_5: 0.5894, loss_yns3d_5: 0.1627, loss_cls2d_0: 1.2310, loss_iou2d_0: 1.3488, loss_box2d_0: 0.8858, loss_alpha2d_0: 0.6299, loss_cls2d_1: 1.1680, loss_iou2d_1: 1.3576, loss_box2d_1: 0.8474, loss_alpha2d_1: 0.6320, loss_cls2d_2: 1.1004, loss_iou2d_2: 1.2131, loss_box2d_2: 0.7095, loss_alpha2d_2: 0.6250, loss_cls3d_dn_0: 0.3078, loss_box3d_dn_0: 1.1869, loss_cls3d_dn_1: 0.3975, loss_box3d_dn_1: 1.0284, loss_cls3d_dn_2: 0.4266, loss_box3d_dn_2: 1.0161, loss_cls3d_dn_3: 0.3930, loss_box3d_dn_3: 0.9271, loss_cls3d_dn_4: 0.1963, loss_box3d_dn_4: 0.8563, loss_cls3d_dn_5: 0.1915, loss_box3d_dn_5: 0.8330, loss_cls2d_dn_0: 0.0522, loss_iou2d_dn_0: 0.9506, loss_box2d_dn_0: 0.8923, loss_alpha2d_dn_0: 0.3149, loss_cls2d_dn_1: 0.0509, loss_iou2d_dn_1: 1.0209, loss_box2d_dn_1: 1.0363, loss_alpha2d_dn_1: 0.2939, loss_cls2d_dn_2: 0.0362, loss_iou2d_dn_2: 0.9019, loss_box2d_dn_2: 0.8172, loss_alpha2d_dn_2: 0.2455, loss_dense_depth: 1.3428, loss: 54.2093, grad_norm: nan

nullmax-vision / SimPB

训练异常,grad_norm异常 #3