Closed slcheng97 closed 1 year ago
rulixiang
commented
1 year ago Thanks for your interest. Unfortunately, I didn't encounter this problem in previous experiments. I'll try to re-evaluate this repo in the next few days when I'm not so busy.
rulixiang
commented
1 year ago Hey, I tested this repo but still didn't encounter the NaN issue. Here are my environment and log.
Since I ended my internship at JD.com, I can't access A100 anymore. Due to the limited memory of 2080Ti, I have to change spg to 1 and nproc_per_node to 4.
GPU: 2080Ti x 4, CUDA 10.1;
Requirements:
imageio==2.9.0
joblib==1.2.0
matplotlib==3.7.0
numpy==1.22.3
omegaconf==2.0.0
Pillow==9.4.0
pydensecrf==1.0rc2
scikit_learn==1.2.1
texttable==1.6.4
timm==0.5.4
torch==1.7.1
torchvision==0.8.2
tqdm==4.64.1Training log:
2023-03-08 02:42:15,529 - dist_train_voc_seg_neg.py - INFO: Pytorch version: 1.7.1
2023-03-08 02:42:15,660 - dist_train_voc_seg_neg.py - INFO: GPU type: NVIDIA GeForce RTX 2080 Ti
2023-03-08 02:42:15,661 - dist_train_voc_seg_neg.py - INFO:
args: Namespace(aux_layer=-3, backbone='deit_base_patch16_224', backend='nccl', betas=(0.9, 0.999), bkg_thre=0.5, cam_scales=(1.0, 0.5, 1.5), ckpt_dir='work_dir_voc/2023-03-08-02-42-15-528714/checkpoints', crop_size=448, data_folder='../VOCdevkit/VOC2012', eval_iters=2000, high_thre=0.7, ignore_index=255, list_folder='datasets/voc', local_crop_size=96, local_rank=0, log_iters=200, low_thre=0.25, lr=6e-05, max_iters=20000, momentum=0.9, num_classes=21, num_workers=10, optimizer='PolyWarmupAdamW', pooling='gmp', power=0.9, pred_dir='work_dir_voc/2023-03-08-02-42-15-528714/predictions', pretrained=True, save_ckpt=False, scales=(0.5, 2), seed=0, spg=1, temp=0.5, train_set='train_aug', val_set='val', w_ctc=0.5, w_ptc=0.2, w_reg=0.05, w_seg=0.1, warmup_iters=1500, warmup_lr=1e-06, work_dir='work_dir_voc/2023-03-08-02-42-15-528714', wt_decay=0.01)
2023-03-08 02:42:19,872 - dist_train_voc_seg_neg.py - INFO: Total gpus: 4, samples per gpu: 1...
2023-03-08 02:42:21,848 - dist_train_voc_seg_neg.py - INFO:
Optimizer:
PolyWarmupAdamW (
Parameter Group 0
amsgrad: False
betas: (0.9, 0.999)
eps: 1e-08
lr: 6e-05
weight_decay: 0.01
Parameter Group 1
amsgrad: False
betas: (0.9, 0.999)
eps: 1e-08
lr: 6e-05
weight_decay: 0.01
Parameter Group 2
amsgrad: False
betas: (0.9, 0.999)
eps: 1e-08
lr: 0.0006000000000000001
weight_decay: 0.01
Parameter Group 3
amsgrad: False
betas: (0.9, 0.999)
eps: 1e-08
lr: 0.0006000000000000001
weight_decay: 0.01
)
2023-03-08 02:46:41,679 - dist_train_voc_seg_neg.py - INFO: Iter: 200; Elasped: 0:04:22; ETA: 7:12:18; LR: 7.960e-06; cls_loss: 0.4185, cls_loss_aux: 0.5568, ptc_loss: 0.4649, ctc_loss: 1.2429, seg_loss: 2.3654...
2023-03-08 02:51:01,213 - dist_train_voc_seg_neg.py - INFO: Iter: 400; Elasped: 0:08:42; ETA: 7:06:18; LR: 1.596e-05; cls_loss: 0.2474, cls_loss_aux: 0.2478, ptc_loss: 0.4548, ctc_loss: 1.0989, seg_loss: 2.5674...
2023-03-08 02:55:18,810 - dist_train_voc_seg_neg.py - INFO: Iter: 600; Elasped: 0:12:59; ETA: 6:59:47; LR: 2.396e-05; cls_loss: 0.2245, cls_loss_aux: 0.2181, ptc_loss: 0.4524, ctc_loss: 0.7777, seg_loss: 2.5648...
2023-03-08 02:59:36,678 - dist_train_voc_seg_neg.py - INFO: Iter: 800; Elasped: 0:17:17; ETA: 6:54:48; LR: 3.196e-05; cls_loss: 0.1654, cls_loss_aux: 0.1640, ptc_loss: 0.4414, ctc_loss: 0.6288, seg_loss: 1.8744...
2023-03-08 03:03:56,609 - dist_train_voc_seg_neg.py - INFO: Iter: 1000; Elasped: 0:21:37; ETA: 6:50:43; LR: 3.996e-05; cls_loss: 0.1347, cls_loss_aux: 0.1719, ptc_loss: 0.4374, ctc_loss: 0.4091, seg_loss: 1.6636...
2023-03-08 03:08:13,912 - dist_train_voc_seg_neg.py - INFO: Iter: 1200; Elasped: 0:25:54; ETA: 6:45:46; LR: 4.796e-05; cls_loss: 0.1154, cls_loss_aux: 0.1569, ptc_loss: 0.4268, ctc_loss: 0.4005, seg_loss: 1.6357...
2023-03-08 03:12:52,073 - dist_train_voc_seg_neg.py - INFO: Iter: 1400; Elasped: 0:30:33; ETA: 6:45:52; LR: 5.596e-05; cls_loss: 0.0998, cls_loss_aux: 0.1385, ptc_loss: 0.3712, ctc_loss: 0.6419, seg_loss: 1.8388...
2023-03-08 03:17:21,425 - dist_train_voc_seg_neg.py - INFO: Iter: 1600; Elasped: 0:35:02; ETA: 6:42:53; LR: 5.566e-05; cls_loss: 0.1007, cls_loss_aux: 0.1429, ptc_loss: 0.3309, ctc_loss: 0.7046, seg_loss: 2.3034...
2023-03-08 03:21:46,945 - dist_train_voc_seg_neg.py - INFO: Iter: 1800; Elasped: 0:39:27; ETA: 6:38:53; LR: 5.512e-05; cls_loss: 0.1072, cls_loss_aux: 0.1507, ptc_loss: 0.2912, ctc_loss: 0.7867, seg_loss: 2.7103...
2023-03-08 03:26:12,917 - dist_train_voc_seg_neg.py - INFO: Iter: 2000; Elasped: 0:43:53; ETA: 6:34:57; LR: 5.457e-05; cls_loss: 0.0894, cls_loss_aux: 0.1229, ptc_loss: 0.2498, ctc_loss: 0.6794, seg_loss: 2.7803...
2023-03-08 03:26:12,917 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 03:46:22,261 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.837739
2023-03-08 03:46:22,262 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 68.501 | 86.462 | 4.146 |
+--------------+--------+---------+----------+
| aeroplane | 49.782 | 73.306 | 0 |
+--------------+--------+---------+----------+
| bicycle | 23.502 | 38.117 | 0 |
+--------------+--------+---------+----------+
| bird | 31.446 | 76.181 | 0 |
+--------------+--------+---------+----------+
| boat | 25.915 | 53.966 | 0.214 |
+--------------+--------+---------+----------+
| bottle | 39.302 | 60.364 | 0 |
+--------------+--------+---------+----------+
| bus | 78.409 | 81.623 | 0.000 |
+--------------+--------+---------+----------+
| car | 37.523 | 66.297 | 0 |
+--------------+--------+---------+----------+
| cat | 59.132 | 59.772 | 0.061 |
+--------------+--------+---------+----------+
| chair | 16.392 | 29.832 | 0.424 |
+--------------+--------+---------+----------+
| cow | 72.795 | 63.320 | 0.000 |
+--------------+--------+---------+----------+
| diningtable | 54.312 | 42.859 | 0.449 |
+--------------+--------+---------+----------+
| dog | 52.336 | 51.335 | 0.978 |
+--------------+--------+---------+----------+
| horse | 58.743 | 75.301 | 0.010 |
+--------------+--------+---------+----------+
| motorbike | 60.869 | 71.835 | 1.111 |
+--------------+--------+---------+----------+
| person | 68.236 | 72.865 | 0.178 |
+--------------+--------+---------+----------+
| pottedplant | 27.003 | 50.040 | 0.013 |
+--------------+--------+---------+----------+
| sheep | 58.226 | 70.417 | 0.222 |
+--------------+--------+---------+----------+
| sofa | 38.700 | 61.695 | 0.077 |
+--------------+--------+---------+----------+
| train | 54.100 | 59.470 | 0 |
+--------------+--------+---------+----------+
| tvmonitor | 28.222 | 48.600 | 0.700 |
+--------------+--------+---------+----------+
| mIoU | 47.783 | 61.603 | 0.409 |
+--------------+--------+---------+----------+
2023-03-08 03:50:38,656 - dist_train_voc_seg_neg.py - INFO: Iter: 2200; Elasped: 1:08:19; ETA: 9:12:44; LR: 5.403e-05; cls_loss: 0.1065, cls_loss_aux: 0.1482, ptc_loss: 0.2882, ctc_loss: 0.7929, seg_loss: 1.3428...
2023-03-08 03:54:54,428 - dist_train_voc_seg_neg.py - INFO: Iter: 2400; Elasped: 1:12:35; ETA: 8:52:16; LR: 5.348e-05; cls_loss: 0.0862, cls_loss_aux: 0.1133, ptc_loss: 0.2733, ctc_loss: 0.7874, seg_loss: 0.6359...
2023-03-08 03:59:10,475 - dist_train_voc_seg_neg.py - INFO: Iter: 2600; Elasped: 1:16:51; ETA: 8:34:18; LR: 5.293e-05; cls_loss: 0.0929, cls_loss_aux: 0.1365, ptc_loss: 0.2698, ctc_loss: 0.8224, seg_loss: 0.6071...
2023-03-08 04:03:28,415 - dist_train_voc_seg_neg.py - INFO: Iter: 2800; Elasped: 1:21:09; ETA: 8:18:29; LR: 5.239e-05; cls_loss: 0.0839, cls_loss_aux: 0.1152, ptc_loss: 0.2537, ctc_loss: 0.7943, seg_loss: 0.4499...
2023-03-08 04:07:44,644 - dist_train_voc_seg_neg.py - INFO: Iter: 3000; Elasped: 1:25:25; ETA: 8:04:01; LR: 5.184e-05; cls_loss: 0.0931, cls_loss_aux: 0.1328, ptc_loss: 0.2534, ctc_loss: 0.7931, seg_loss: 0.4315...
2023-03-08 04:12:01,109 - dist_train_voc_seg_neg.py - INFO: Iter: 3200; Elasped: 1:29:42; ETA: 7:50:55; LR: 5.129e-05; cls_loss: 0.0925, cls_loss_aux: 0.1329, ptc_loss: 0.2508, ctc_loss: 0.7810, seg_loss: 0.4295...
2023-03-08 04:16:17,529 - dist_train_voc_seg_neg.py - INFO: Iter: 3400; Elasped: 1:33:58; ETA: 7:38:46; LR: 5.074e-05; cls_loss: 0.1099, cls_loss_aux: 0.1507, ptc_loss: 0.2639, ctc_loss: 0.7376, seg_loss: 0.4238...
2023-03-08 04:20:33,408 - dist_train_voc_seg_neg.py - INFO: Iter: 3600; Elasped: 1:38:14; ETA: 7:27:30; LR: 5.019e-05; cls_loss: 0.0801, cls_loss_aux: 0.1142, ptc_loss: 0.2544, ctc_loss: 0.7083, seg_loss: 0.3531...
2023-03-08 04:24:49,094 - dist_train_voc_seg_neg.py - INFO: Iter: 3800; Elasped: 1:42:30; ETA: 7:16:58; LR: 4.964e-05; cls_loss: 0.0813, cls_loss_aux: 0.1069, ptc_loss: 0.2522, ctc_loss: 0.6764, seg_loss: 0.3690...
2023-03-08 04:29:21,630 - dist_train_voc_seg_neg.py - INFO: Iter: 4000; Elasped: 1:47:02; ETA: 7:08:08; LR: 4.909e-05; cls_loss: 0.0865, cls_loss_aux: 0.1209, ptc_loss: 0.2806, ctc_loss: 0.7467, seg_loss: 0.4256...
2023-03-08 04:29:21,630 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 04:49:13,669 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.865020
2023-03-08 04:49:13,670 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 86.345 | 86.369 | 86.194 |
+--------------+--------+---------+----------+
| aeroplane | 63.739 | 77.289 | 67.318 |
+--------------+--------+---------+----------+
| bicycle | 37.491 | 42.549 | 34.896 |
+--------------+--------+---------+----------+
| bird | 66.430 | 69.983 | 67.357 |
+--------------+--------+---------+----------+
| boat | 27.150 | 53.783 | 22.617 |
+--------------+--------+---------+----------+
| bottle | 60.492 | 51.921 | 59.932 |
+--------------+--------+---------+----------+
| bus | 83.242 | 70.280 | 75.139 |
+--------------+--------+---------+----------+
| car | 76.300 | 62.948 | 64.881 |
+--------------+--------+---------+----------+
| cat | 87.016 | 50.301 | 84.039 |
+--------------+--------+---------+----------+
| chair | 31.310 | 28.879 | 17.434 |
+--------------+--------+---------+----------+
| cow | 88.825 | 51.505 | 77.595 |
+--------------+--------+---------+----------+
| diningtable | 61.497 | 47.765 | 50.781 |
+--------------+--------+---------+----------+
| dog | 75.162 | 66.520 | 72.989 |
+--------------+--------+---------+----------+
| horse | 84.294 | 67.162 | 75.556 |
+--------------+--------+---------+----------+
| motorbike | 75.526 | 64.002 | 71.127 |
+--------------+--------+---------+----------+
| person | 78.998 | 72.469 | 73.917 |
+--------------+--------+---------+----------+
| pottedplant | 46.218 | 56.123 | 47.656 |
+--------------+--------+---------+----------+
| sheep | 85.366 | 61.249 | 75.210 |
+--------------+--------+---------+----------+
| sofa | 51.364 | 60.159 | 35.074 |
+--------------+--------+---------+----------+
| train | 56.152 | 52.214 | 52.054 |
+--------------+--------+---------+----------+
| tvmonitor | 40.665 | 54.185 | 35.494 |
+--------------+--------+---------+----------+
| mIoU | 64.932 | 59.412 | 59.393 |
+--------------+--------+---------+----------+
2023-03-08 04:53:42,639 - dist_train_voc_seg_neg.py - INFO: Iter: 4200; Elasped: 2:11:23; ETA: 8:14:15; LR: 4.853e-05; cls_loss: 0.0828, cls_loss_aux: 0.1115, ptc_loss: 0.2654, ctc_loss: 0.7083, seg_loss: 0.3159...
2023-03-08 04:58:15,365 - dist_train_voc_seg_neg.py - INFO: Iter: 4400; Elasped: 2:15:56; ETA: 8:01:56; LR: 4.798e-05; cls_loss: 0.0849, cls_loss_aux: 0.1083, ptc_loss: 0.2163, ctc_loss: 0.7525, seg_loss: 0.3662...
2023-03-08 05:02:45,257 - dist_train_voc_seg_neg.py - INFO: Iter: 4600; Elasped: 2:20:26; ETA: 7:50:08; LR: 4.743e-05; cls_loss: 0.0872, cls_loss_aux: 0.1157, ptc_loss: 0.2388, ctc_loss: 0.6819, seg_loss: 0.3157...
2023-03-08 05:07:38,684 - dist_train_voc_seg_neg.py - INFO: Iter: 4800; Elasped: 2:25:19; ETA: 7:40:10; LR: 4.687e-05; cls_loss: 0.0854, cls_loss_aux: 0.1066, ptc_loss: 0.2366, ctc_loss: 0.7154, seg_loss: 0.2954...
2023-03-08 05:13:10,873 - dist_train_voc_seg_neg.py - INFO: Iter: 5000; Elasped: 2:30:51; ETA: 7:32:33; LR: 4.632e-05; cls_loss: 0.0792, cls_loss_aux: 0.1061, ptc_loss: 0.2470, ctc_loss: 0.7548, seg_loss: 0.3050...
2023-03-08 05:18:35,669 - dist_train_voc_seg_neg.py - INFO: Iter: 5200; Elasped: 2:36:16; ETA: 7:24:45; LR: 4.576e-05; cls_loss: 0.0711, cls_loss_aux: 0.0916, ptc_loss: 0.2288, ctc_loss: 0.6108, seg_loss: 0.2429...
2023-03-08 05:24:12,106 - dist_train_voc_seg_neg.py - INFO: Iter: 5400; Elasped: 2:41:53; ETA: 7:17:41; LR: 4.520e-05; cls_loss: 0.0697, cls_loss_aux: 0.0905, ptc_loss: 0.2517, ctc_loss: 0.5877, seg_loss: 0.2363...
2023-03-08 05:29:41,072 - dist_train_voc_seg_neg.py - INFO: Iter: 5600; Elasped: 2:47:22; ETA: 7:10:22; LR: 4.465e-05; cls_loss: 0.0699, cls_loss_aux: 0.0966, ptc_loss: 0.2324, ctc_loss: 0.6957, seg_loss: 0.2114...
2023-03-08 05:35:16,324 - dist_train_voc_seg_neg.py - INFO: Iter: 5800; Elasped: 2:52:57; ETA: 7:03:25; LR: 4.409e-05; cls_loss: 0.0724, cls_loss_aux: 0.0868, ptc_loss: 0.2752, ctc_loss: 0.6107, seg_loss: 0.2404...
2023-03-08 05:40:45,558 - dist_train_voc_seg_neg.py - INFO: Iter: 6000; Elasped: 2:58:26; ETA: 6:56:20; LR: 4.353e-05; cls_loss: 0.0806, cls_loss_aux: 0.0924, ptc_loss: 0.2336, ctc_loss: 0.6122, seg_loss: 0.2248...
2023-03-08 05:40:45,558 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 06:06:02,070 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.877156
2023-03-08 06:06:02,070 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 89.190 | 86.673 | 88.543 |
+--------------+--------+---------+----------+
| aeroplane | 65.069 | 73.123 | 69.947 |
+--------------+--------+---------+----------+
| bicycle | 34.819 | 40.147 | 36.074 |
+--------------+--------+---------+----------+
| bird | 64.290 | 73.726 | 57.918 |
+--------------+--------+---------+----------+
| boat | 59.335 | 56.225 | 58.290 |
+--------------+--------+---------+----------+
| bottle | 67.214 | 56.429 | 64.130 |
+--------------+--------+---------+----------+
| bus | 84.193 | 74.704 | 78.073 |
+--------------+--------+---------+----------+
| car | 79.470 | 71.753 | 73.442 |
+--------------+--------+---------+----------+
| cat | 89.389 | 71.996 | 82.953 |
+--------------+--------+---------+----------+
| chair | 39.908 | 33.634 | 22.504 |
+--------------+--------+---------+----------+
| cow | 87.544 | 41.977 | 72.872 |
+--------------+--------+---------+----------+
| diningtable | 57.042 | 50.187 | 47.480 |
+--------------+--------+---------+----------+
| dog | 82.462 | 51.472 | 78.712 |
+--------------+--------+---------+----------+
| horse | 85.066 | 69.379 | 76.552 |
+--------------+--------+---------+----------+
| motorbike | 76.820 | 69.818 | 73.233 |
+--------------+--------+---------+----------+
| person | 80.920 | 78.005 | 77.538 |
+--------------+--------+---------+----------+
| pottedplant | 55.484 | 57.259 | 49.216 |
+--------------+--------+---------+----------+
| sheep | 87.089 | 67.461 | 75.819 |
+--------------+--------+---------+----------+
| sofa | 56.762 | 62.667 | 39.331 |
+--------------+--------+---------+----------+
| train | 54.961 | 50.893 | 53.503 |
+--------------+--------+---------+----------+
| tvmonitor | 48.725 | 52.960 | 47.745 |
+--------------+--------+---------+----------+
| mIoU | 68.845 | 61.452 | 63.042 |
+--------------+--------+---------+----------+
2023-03-08 06:11:38,161 - dist_train_voc_seg_neg.py - INFO: Iter: 6200; Elasped: 3:29:19; ETA: 7:45:53; LR: 4.297e-05; cls_loss: 0.0779, cls_loss_aux: 0.1050, ptc_loss: 0.2564, ctc_loss: 0.6636, seg_loss: 0.2156...
2023-03-08 06:17:06,784 - dist_train_voc_seg_neg.py - INFO: Iter: 6400; Elasped: 3:34:47; ETA: 7:36:24; LR: 4.241e-05; cls_loss: 0.0755, cls_loss_aux: 0.1054, ptc_loss: 0.2770, ctc_loss: 0.5766, seg_loss: 0.2787...
2023-03-08 06:22:42,829 - dist_train_voc_seg_neg.py - INFO: Iter: 6600; Elasped: 3:40:23; ETA: 7:27:26; LR: 4.185e-05; cls_loss: 0.0695, cls_loss_aux: 0.0811, ptc_loss: 0.2349, ctc_loss: 0.6164, seg_loss: 0.3015...
2023-03-08 06:28:11,628 - dist_train_voc_seg_neg.py - INFO: Iter: 6800; Elasped: 3:45:52; ETA: 7:18:26; LR: 4.128e-05; cls_loss: 0.0733, cls_loss_aux: 0.0949, ptc_loss: 0.2361, ctc_loss: 0.6325, seg_loss: 0.2376...
2023-03-08 06:33:47,463 - dist_train_voc_seg_neg.py - INFO: Iter: 7000; Elasped: 3:51:28; ETA: 7:09:52; LR: 4.072e-05; cls_loss: 0.0699, cls_loss_aux: 0.0898, ptc_loss: 0.2353, ctc_loss: 0.7015, seg_loss: 0.1906...
2023-03-08 06:39:15,631 - dist_train_voc_seg_neg.py - INFO: Iter: 7200; Elasped: 3:56:56; ETA: 7:01:12; LR: 4.016e-05; cls_loss: 0.0637, cls_loss_aux: 0.0797, ptc_loss: 0.2391, ctc_loss: 0.4716, seg_loss: 0.2051...
2023-03-08 06:44:52,024 - dist_train_voc_seg_neg.py - INFO: Iter: 7400; Elasped: 4:02:33; ETA: 6:52:59; LR: 3.959e-05; cls_loss: 0.0676, cls_loss_aux: 0.0855, ptc_loss: 0.2307, ctc_loss: 0.5126, seg_loss: 0.1719...
2023-03-08 06:50:20,226 - dist_train_voc_seg_neg.py - INFO: Iter: 7600; Elasped: 4:08:01; ETA: 6:44:39; LR: 3.902e-05; cls_loss: 0.0673, cls_loss_aux: 0.0805, ptc_loss: 0.2393, ctc_loss: 0.6559, seg_loss: 0.2336...
2023-03-08 06:55:56,300 - dist_train_voc_seg_neg.py - INFO: Iter: 7800; Elasped: 4:13:37; ETA: 6:36:40; LR: 3.846e-05; cls_loss: 0.0789, cls_loss_aux: 0.0920, ptc_loss: 0.2505, ctc_loss: 0.6263, seg_loss: 0.2226...
2023-03-08 07:01:26,033 - dist_train_voc_seg_neg.py - INFO: Iter: 8000; Elasped: 4:19:07; ETA: 6:28:40; LR: 3.789e-05; cls_loss: 0.0692, cls_loss_aux: 0.0838, ptc_loss: 0.2427, ctc_loss: 0.6231, seg_loss: 0.2354...
2023-03-08 07:01:26,034 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 07:26:41,783 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.891833
2023-03-08 07:26:41,784 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 89.713 | 86.351 | 88.415 |
+--------------+--------+---------+----------+
| aeroplane | 73.791 | 76.026 | 74.727 |
+--------------+--------+---------+----------+
| bicycle | 37.177 | 40.142 | 24.607 |
+--------------+--------+---------+----------+
| bird | 58.880 | 71.736 | 59.559 |
+--------------+--------+---------+----------+
| boat | 60.378 | 64.210 | 56.176 |
+--------------+--------+---------+----------+
| bottle | 73.909 | 63.142 | 71.111 |
+--------------+--------+---------+----------+
| bus | 79.141 | 65.288 | 76.546 |
+--------------+--------+---------+----------+
| car | 81.925 | 69.234 | 77.244 |
+--------------+--------+---------+----------+
| cat | 88.494 | 56.811 | 85.015 |
+--------------+--------+---------+----------+
| chair | 37.314 | 32.555 | 22.331 |
+--------------+--------+---------+----------+
| cow | 80.531 | 53.432 | 67.319 |
+--------------+--------+---------+----------+
| diningtable | 46.416 | 46.257 | 41.701 |
+--------------+--------+---------+----------+
| dog | 80.783 | 59.717 | 74.822 |
+--------------+--------+---------+----------+
| horse | 84.247 | 69.773 | 72.462 |
+--------------+--------+---------+----------+
| motorbike | 77.157 | 66.009 | 73.389 |
+--------------+--------+---------+----------+
| person | 76.813 | 69.523 | 73.149 |
+--------------+--------+---------+----------+
| pottedplant | 54.645 | 56.545 | 52.553 |
+--------------+--------+---------+----------+
| sheep | 85.164 | 62.371 | 70.285 |
+--------------+--------+---------+----------+
| sofa | 57.619 | 64.328 | 38.696 |
+--------------+--------+---------+----------+
| train | 56.854 | 51.231 | 53.287 |
+--------------+--------+---------+----------+
| tvmonitor | 62.498 | 64.037 | 63.396 |
+--------------+--------+---------+----------+
| mIoU | 68.736 | 61.368 | 62.704 |
+--------------+--------+---------+----------+
2023-03-08 07:32:12,698 - dist_train_voc_seg_neg.py - INFO: Iter: 8200; Elasped: 4:49:53; ETA: 6:57:08; LR: 3.732e-05; cls_loss: 0.0604, cls_loss_aux: 0.0747, ptc_loss: 0.2181, ctc_loss: 0.5977, seg_loss: 0.1787...
2023-03-08 07:37:44,706 - dist_train_voc_seg_neg.py - INFO: Iter: 8400; Elasped: 4:55:25; ETA: 6:47:57; LR: 3.675e-05; cls_loss: 0.0662, cls_loss_aux: 0.0803, ptc_loss: 0.2444, ctc_loss: 0.6273, seg_loss: 0.2602...
2023-03-08 07:43:15,417 - dist_train_voc_seg_neg.py - INFO: Iter: 8600; Elasped: 5:00:56; ETA: 6:38:54; LR: 3.618e-05; cls_loss: 0.0626, cls_loss_aux: 0.0783, ptc_loss: 0.2413, ctc_loss: 0.6393, seg_loss: 0.2147...
2023-03-08 07:48:46,811 - dist_train_voc_seg_neg.py - INFO: Iter: 8800; Elasped: 5:06:27; ETA: 6:30:01; LR: 3.561e-05; cls_loss: 0.0629, cls_loss_aux: 0.0724, ptc_loss: 0.2373, ctc_loss: 0.6101, seg_loss: 0.1567...
2023-03-08 07:54:17,047 - dist_train_voc_seg_neg.py - INFO: Iter: 9000; Elasped: 5:11:58; ETA: 6:21:17; LR: 3.504e-05; cls_loss: 0.0778, cls_loss_aux: 0.0932, ptc_loss: 0.2480, ctc_loss: 0.5889, seg_loss: 0.2414...
2023-03-08 07:59:50,771 - dist_train_voc_seg_neg.py - INFO: Iter: 9200; Elasped: 5:17:31; ETA: 6:12:44; LR: 3.446e-05; cls_loss: 0.0691, cls_loss_aux: 0.0782, ptc_loss: 0.2729, ctc_loss: 0.5772, seg_loss: 0.1751...
2023-03-08 08:05:20,237 - dist_train_voc_seg_neg.py - INFO: Iter: 9400; Elasped: 5:23:01; ETA: 6:04:15; LR: 3.389e-05; cls_loss: 0.0537, cls_loss_aux: 0.0657, ptc_loss: 0.2596, ctc_loss: 0.6060, seg_loss: 0.1770...
2023-03-08 08:10:52,631 - dist_train_voc_seg_neg.py - INFO: Iter: 9600; Elasped: 5:28:33; ETA: 5:55:55; LR: 3.331e-05; cls_loss: 0.0701, cls_loss_aux: 0.0823, ptc_loss: 0.2456, ctc_loss: 0.7382, seg_loss: 0.2270...
2023-03-08 08:16:22,151 - dist_train_voc_seg_neg.py - INFO: Iter: 9800; Elasped: 5:34:03; ETA: 5:47:41; LR: 3.273e-05; cls_loss: 0.0584, cls_loss_aux: 0.0693, ptc_loss: 0.2132, ctc_loss: 0.6263, seg_loss: 0.2582...
2023-03-08 08:21:55,968 - dist_train_voc_seg_neg.py - INFO: Iter: 10000; Elasped: 5:39:36; ETA: 5:39:36; LR: 3.216e-05; cls_loss: 0.0654, cls_loss_aux: 0.0739, ptc_loss: 0.2210, ctc_loss: 0.6709, seg_loss: 0.1824...
2023-03-08 08:21:55,968 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 08:47:07,225 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.890539
2023-03-08 08:47:07,225 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 90.078 | 86.499 | 89.107 |
+--------------+--------+---------+----------+
| aeroplane | 70.293 | 79.226 | 74.333 |
+--------------+--------+---------+----------+
| bicycle | 38.596 | 40.847 | 37.714 |
+--------------+--------+---------+----------+
| bird | 72.353 | 66.301 | 73.660 |
+--------------+--------+---------+----------+
| boat | 40.382 | 50.510 | 33.878 |
+--------------+--------+---------+----------+
| bottle | 72.892 | 54.527 | 67.801 |
+--------------+--------+---------+----------+
| bus | 86.405 | 69.834 | 81.223 |
+--------------+--------+---------+----------+
| car | 81.807 | 68.610 | 76.357 |
+--------------+--------+---------+----------+
| cat | 90.454 | 66.545 | 87.397 |
+--------------+--------+---------+----------+
| chair | 38.760 | 36.159 | 22.555 |
+--------------+--------+---------+----------+
| cow | 85.796 | 51.145 | 74.941 |
+--------------+--------+---------+----------+
| diningtable | 53.146 | 47.547 | 45.227 |
+--------------+--------+---------+----------+
| dog | 83.112 | 60.220 | 78.544 |
+--------------+--------+---------+----------+
| horse | 85.529 | 68.899 | 75.458 |
+--------------+--------+---------+----------+
| motorbike | 79.513 | 66.393 | 75.028 |
+--------------+--------+---------+----------+
| person | 75.135 | 69.584 | 74.310 |
+--------------+--------+---------+----------+
| pottedplant | 55.803 | 61.632 | 53.561 |
+--------------+--------+---------+----------+
| sheep | 87.157 | 52.524 | 83.868 |
+--------------+--------+---------+----------+
| sofa | 62.001 | 65.720 | 44.287 |
+--------------+--------+---------+----------+
| train | 58.503 | 51.853 | 57.803 |
+--------------+--------+---------+----------+
| tvmonitor | 59.336 | 58.030 | 58.409 |
+--------------+--------+---------+----------+
| mIoU | 69.859 | 60.600 | 65.022 |
+--------------+--------+---------+----------+
2023-03-08 08:52:37,476 - dist_train_voc_seg_neg.py - INFO: Iter: 10200; Elasped: 6:10:18; ETA: 5:55:46; LR: 3.158e-05; cls_loss: 0.0705, cls_loss_aux: 0.0779, ptc_loss: 0.2503, ctc_loss: 0.5706, seg_loss: 0.2097...
2023-03-08 08:58:10,976 - dist_train_voc_seg_neg.py - INFO: Iter: 10400; Elasped: 6:15:51; ETA: 5:46:56; LR: 3.100e-05; cls_loss: 0.0620, cls_loss_aux: 0.0713, ptc_loss: 0.2548, ctc_loss: 0.6164, seg_loss: 0.2073...
2023-03-08 09:03:41,626 - dist_train_voc_seg_neg.py - INFO: Iter: 10600; Elasped: 6:21:22; ETA: 5:38:11; LR: 3.041e-05; cls_loss: 0.0568, cls_loss_aux: 0.0672, ptc_loss: 0.2461, ctc_loss: 0.6308, seg_loss: 0.1421...
2023-03-08 09:09:15,951 - dist_train_voc_seg_neg.py - INFO: Iter: 10800; Elasped: 6:26:56; ETA: 5:29:36; LR: 2.983e-05; cls_loss: 0.0568, cls_loss_aux: 0.0659, ptc_loss: 0.2578, ctc_loss: 0.5976, seg_loss: 0.1710...
2023-03-08 09:14:45,432 - dist_train_voc_seg_neg.py - INFO: Iter: 11000; Elasped: 6:32:26; ETA: 5:21:04; LR: 2.925e-05; cls_loss: 0.0564, cls_loss_aux: 0.0621, ptc_loss: 0.2368, ctc_loss: 0.5872, seg_loss: 0.1535...
2023-03-08 09:20:19,939 - dist_train_voc_seg_neg.py - INFO: Iter: 11200; Elasped: 6:38:00; ETA: 5:12:42; LR: 2.866e-05; cls_loss: 0.0487, cls_loss_aux: 0.0524, ptc_loss: 0.2157, ctc_loss: 0.6136, seg_loss: 0.1554...
2023-03-08 09:25:48,920 - dist_train_voc_seg_neg.py - INFO: Iter: 11400; Elasped: 6:43:29; ETA: 5:04:22; LR: 2.807e-05; cls_loss: 0.0626, cls_loss_aux: 0.0690, ptc_loss: 0.2511, ctc_loss: 0.5473, seg_loss: 0.2279...
2023-03-08 09:31:23,152 - dist_train_voc_seg_neg.py - INFO: Iter: 11600; Elasped: 6:49:04; ETA: 4:56:13; LR: 2.749e-05; cls_loss: 0.0545, cls_loss_aux: 0.0637, ptc_loss: 0.2158, ctc_loss: 0.5644, seg_loss: 0.1559...
2023-03-08 09:36:52,623 - dist_train_voc_seg_neg.py - INFO: Iter: 11800; Elasped: 6:54:33; ETA: 4:48:04; LR: 2.690e-05; cls_loss: 0.0567, cls_loss_aux: 0.0680, ptc_loss: 0.2443, ctc_loss: 0.5047, seg_loss: 0.1598...
2023-03-08 09:42:27,480 - dist_train_voc_seg_neg.py - INFO: Iter: 12000; Elasped: 7:00:08; ETA: 4:40:05; LR: 2.631e-05; cls_loss: 0.0586, cls_loss_aux: 0.0656, ptc_loss: 0.2178, ctc_loss: 0.6490, seg_loss: 0.1231...
2023-03-08 09:42:27,481 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 10:07:39,973 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.896676
2023-03-08 10:07:39,973 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 90.947 | 86.821 | 90.477 |
+--------------+--------+---------+----------+
| aeroplane | 75.396 | 78.069 | 75.610 |
+--------------+--------+---------+----------+
| bicycle | 40.473 | 35.651 | 38.913 |
+--------------+--------+---------+----------+
| bird | 77.496 | 68.549 | 77.749 |
+--------------+--------+---------+----------+
| boat | 66.599 | 58.120 | 65.612 |
+--------------+--------+---------+----------+
| bottle | 73.941 | 58.250 | 73.349 |
+--------------+--------+---------+----------+
| bus | 86.261 | 66.421 | 85.723 |
+--------------+--------+---------+----------+
| car | 83.001 | 69.008 | 76.191 |
+--------------+--------+---------+----------+
| cat | 87.710 | 63.792 | 87.878 |
+--------------+--------+---------+----------+
| chair | 41.279 | 45.012 | 22.932 |
+--------------+--------+---------+----------+
| cow | 80.757 | 51.638 | 76.300 |
+--------------+--------+---------+----------+
| diningtable | 60.061 | 52.471 | 51.952 |
+--------------+--------+---------+----------+
| dog | 84.144 | 69.807 | 78.872 |
+--------------+--------+---------+----------+
| horse | 83.042 | 63.368 | 77.716 |
+--------------+--------+---------+----------+
| motorbike | 74.306 | 61.785 | 62.163 |
+--------------+--------+---------+----------+
| person | 80.666 | 74.378 | 77.504 |
+--------------+--------+---------+----------+
| pottedplant | 56.889 | 58.987 | 56.489 |
+--------------+--------+---------+----------+
| sheep | 86.072 | 57.449 | 84.595 |
+--------------+--------+---------+----------+
| sofa | 54.336 | 56.731 | 35.147 |
+--------------+--------+---------+----------+
| train | 58.589 | 52.359 | 58.590 |
+--------------+--------+---------+----------+
| tvmonitor | 56.741 | 62.396 | 57.787 |
+--------------+--------+---------+----------+
| mIoU | 71.367 | 61.479 | 67.217 |
+--------------+--------+---------+----------+
2023-03-08 10:13:10,501 - dist_train_voc_seg_neg.py - INFO: Iter: 12200; Elasped: 7:30:51; ETA: 4:48:14; LR: 2.571e-05; cls_loss: 0.0543, cls_loss_aux: 0.0609, ptc_loss: 0.2267, ctc_loss: 0.6615, seg_loss: 0.1522...
2023-03-08 10:18:43,721 - dist_train_voc_seg_neg.py - INFO: Iter: 12400; Elasped: 7:36:24; ETA: 4:39:43; LR: 2.512e-05; cls_loss: 0.0475, cls_loss_aux: 0.0532, ptc_loss: 0.2273, ctc_loss: 0.6214, seg_loss: 0.1759...
2023-03-08 10:24:13,969 - dist_train_voc_seg_neg.py - INFO: Iter: 12600; Elasped: 7:41:54; ETA: 4:31:16; LR: 2.452e-05; cls_loss: 0.0593, cls_loss_aux: 0.0679, ptc_loss: 0.2495, ctc_loss: 0.5860, seg_loss: 0.1851...
2023-03-08 10:29:47,494 - dist_train_voc_seg_neg.py - INFO: Iter: 12800; Elasped: 7:47:28; ETA: 4:22:57; LR: 2.393e-05; cls_loss: 0.0561, cls_loss_aux: 0.0633, ptc_loss: 0.2353, ctc_loss: 0.6418, seg_loss: 0.1521...
2023-03-08 10:35:17,212 - dist_train_voc_seg_neg.py - INFO: Iter: 13000; Elasped: 7:52:58; ETA: 4:14:40; LR: 2.333e-05; cls_loss: 0.0479, cls_loss_aux: 0.0536, ptc_loss: 0.2460, ctc_loss: 0.6571, seg_loss: 0.1433...
2023-03-08 10:40:51,007 - dist_train_voc_seg_neg.py - INFO: Iter: 13200; Elasped: 7:58:32; ETA: 4:06:31; LR: 2.273e-05; cls_loss: 0.0520, cls_loss_aux: 0.0604, ptc_loss: 0.2323, ctc_loss: 0.6059, seg_loss: 0.1726...
2023-03-08 10:46:22,942 - dist_train_voc_seg_neg.py - INFO: Iter: 13400; Elasped: 8:04:03; ETA: 3:58:24; LR: 2.212e-05; cls_loss: 0.0518, cls_loss_aux: 0.0575, ptc_loss: 0.2401, ctc_loss: 0.5873, seg_loss: 0.1659...
2023-03-08 10:51:56,157 - dist_train_voc_seg_neg.py - INFO: Iter: 13600; Elasped: 8:09:37; ETA: 3:50:24; LR: 2.152e-05; cls_loss: 0.0538, cls_loss_aux: 0.0525, ptc_loss: 0.2244, ctc_loss: 0.6135, seg_loss: 0.1310...
2023-03-08 10:57:27,068 - dist_train_voc_seg_neg.py - INFO: Iter: 13800; Elasped: 8:15:08; ETA: 3:42:27; LR: 2.091e-05; cls_loss: 0.0505, cls_loss_aux: 0.0578, ptc_loss: 0.2243, ctc_loss: 0.6077, seg_loss: 0.1634...
2023-03-08 11:02:59,933 - dist_train_voc_seg_neg.py - INFO: Iter: 14000; Elasped: 8:20:40; ETA: 3:34:34; LR: 2.031e-05; cls_loss: 0.0578, cls_loss_aux: 0.0570, ptc_loss: 0.2329, ctc_loss: 0.6095, seg_loss: 0.1715...
2023-03-08 11:02:59,933 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 11:25:23,980 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.895572
2023-03-08 11:25:23,981 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 90.734 | 85.843 | 90.198 |
+--------------+--------+---------+----------+
| aeroplane | 77.839 | 79.928 | 78.874 |
+--------------+--------+---------+----------+
| bicycle | 38.393 | 37.421 | 37.744 |
+--------------+--------+---------+----------+
| bird | 77.218 | 69.949 | 72.559 |
+--------------+--------+---------+----------+
| boat | 63.346 | 52.245 | 60.830 |
+--------------+--------+---------+----------+
| bottle | 70.250 | 58.482 | 72.558 |
+--------------+--------+---------+----------+
| bus | 79.709 | 60.618 | 77.307 |
+--------------+--------+---------+----------+
| car | 82.630 | 68.665 | 76.827 |
+--------------+--------+---------+----------+
| cat | 88.590 | 65.167 | 88.305 |
+--------------+--------+---------+----------+
| chair | 41.538 | 41.091 | 21.952 |
+--------------+--------+---------+----------+
| cow | 83.425 | 43.301 | 80.867 |
+--------------+--------+---------+----------+
| diningtable | 55.552 | 44.892 | 44.758 |
+--------------+--------+---------+----------+
| dog | 84.934 | 65.101 | 81.173 |
+--------------+--------+---------+----------+
| horse | 84.159 | 62.560 | 77.281 |
+--------------+--------+---------+----------+
| motorbike | 76.677 | 66.312 | 71.790 |
+--------------+--------+---------+----------+
| person | 80.100 | 73.473 | 78.321 |
+--------------+--------+---------+----------+
| pottedplant | 60.892 | 54.449 | 56.680 |
+--------------+--------+---------+----------+
| sheep | 87.043 | 51.278 | 84.670 |
+--------------+--------+---------+----------+
| sofa | 56.076 | 63.729 | 35.953 |
+--------------+--------+---------+----------+
| train | 56.134 | 49.450 | 56.342 |
+--------------+--------+---------+----------+
| tvmonitor | 60.263 | 59.569 | 62.088 |
+--------------+--------+---------+----------+
| mIoU | 71.214 | 59.692 | 67.004 |
+--------------+--------+---------+----------+
2023-03-08 11:30:00,780 - dist_train_voc_seg_neg.py - INFO: Iter: 14200; Elasped: 8:47:41; ETA: 3:35:31; LR: 1.970e-05; cls_loss: 0.0466, cls_loss_aux: 0.0488, ptc_loss: 0.2368, ctc_loss: 0.5995, seg_loss: 0.1200...
2023-03-08 11:34:50,886 - dist_train_voc_seg_neg.py - INFO: Iter: 14400; Elasped: 8:52:31; ETA: 3:27:05; LR: 1.908e-05; cls_loss: 0.0564, cls_loss_aux: 0.0601, ptc_loss: 0.2531, ctc_loss: 0.5904, seg_loss: 0.1750...
2023-03-08 11:39:33,392 - dist_train_voc_seg_neg.py - INFO: Iter: 14600; Elasped: 8:57:14; ETA: 3:18:42; LR: 1.847e-05; cls_loss: 0.0477, cls_loss_aux: 0.0532, ptc_loss: 0.2475, ctc_loss: 0.5692, seg_loss: 0.1416...
2023-03-08 11:44:21,829 - dist_train_voc_seg_neg.py - INFO: Iter: 14800; Elasped: 9:02:02; ETA: 3:10:26; LR: 1.785e-05; cls_loss: 0.0515, cls_loss_aux: 0.0542, ptc_loss: 0.2256, ctc_loss: 0.6364, seg_loss: 0.1670...
2023-03-08 11:49:06,604 - dist_train_voc_seg_neg.py - INFO: Iter: 15000; Elasped: 9:06:47; ETA: 3:02:15; LR: 1.723e-05; cls_loss: 0.0431, cls_loss_aux: 0.0455, ptc_loss: 0.2090, ctc_loss: 0.6116, seg_loss: 0.1187...
2023-03-08 11:53:57,729 - dist_train_voc_seg_neg.py - INFO: Iter: 15200; Elasped: 9:11:38; ETA: 2:54:12; LR: 1.661e-05; cls_loss: 0.0481, cls_loss_aux: 0.0560, ptc_loss: 0.2060, ctc_loss: 0.5729, seg_loss: 0.1422...
2023-03-08 11:58:48,812 - dist_train_voc_seg_neg.py - INFO: Iter: 15400; Elasped: 9:16:29; ETA: 2:46:13; LR: 1.599e-05; cls_loss: 0.0464, cls_loss_aux: 0.0500, ptc_loss: 0.2469, ctc_loss: 0.6279, seg_loss: 0.1050...
2023-03-08 12:03:40,042 - dist_train_voc_seg_neg.py - INFO: Iter: 15600; Elasped: 9:21:21; ETA: 2:38:19; LR: 1.536e-05; cls_loss: 0.0434, cls_loss_aux: 0.0477, ptc_loss: 0.2283, ctc_loss: 0.6654, seg_loss: 0.1453...
2023-03-08 12:08:32,361 - dist_train_voc_seg_neg.py - INFO: Iter: 15800; Elasped: 9:26:13; ETA: 2:30:30; LR: 1.473e-05; cls_loss: 0.0445, cls_loss_aux: 0.0490, ptc_loss: 0.2388, ctc_loss: 0.6830, seg_loss: 0.1180...
2023-03-08 12:13:18,609 - dist_train_voc_seg_neg.py - INFO: Iter: 16000; Elasped: 9:30:59; ETA: 2:22:44; LR: 1.410e-05; cls_loss: 0.0472, cls_loss_aux: 0.0527, ptc_loss: 0.1922, ctc_loss: 0.6252, seg_loss: 0.1264...
2023-03-08 12:13:18,610 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 12:35:13,500 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.911788
2023-03-08 12:35:13,501 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 90.320 | 85.899 | 89.859 |
+--------------+--------+---------+----------+
| aeroplane | 74.137 | 77.335 | 77.283 |
+--------------+--------+---------+----------+
| bicycle | 37.659 | 36.238 | 37.447 |
+--------------+--------+---------+----------+
| bird | 72.563 | 71.525 | 59.428 |
+--------------+--------+---------+----------+
| boat | 63.456 | 51.358 | 61.088 |
+--------------+--------+---------+----------+
| bottle | 71.458 | 57.509 | 72.881 |
+--------------+--------+---------+----------+
| bus | 80.249 | 58.213 | 79.389 |
+--------------+--------+---------+----------+
| car | 80.719 | 65.597 | 78.026 |
+--------------+--------+---------+----------+
| cat | 87.800 | 64.327 | 87.531 |
+--------------+--------+---------+----------+
| chair | 42.422 | 43.511 | 22.403 |
+--------------+--------+---------+----------+
| cow | 82.466 | 57.091 | 83.671 |
+--------------+--------+---------+----------+
| diningtable | 55.222 | 44.147 | 50.042 |
+--------------+--------+---------+----------+
| dog | 84.913 | 68.378 | 80.400 |
+--------------+--------+---------+----------+
| horse | 83.241 | 60.032 | 79.402 |
+--------------+--------+---------+----------+
| motorbike | 73.975 | 60.751 | 70.996 |
+--------------+--------+---------+----------+
| person | 75.750 | 67.835 | 73.215 |
+--------------+--------+---------+----------+
| pottedplant | 59.784 | 55.121 | 55.464 |
+--------------+--------+---------+----------+
| sheep | 86.190 | 49.566 | 84.391 |
+--------------+--------+---------+----------+
| sofa | 58.530 | 63.482 | 40.135 |
+--------------+--------+---------+----------+
| train | 58.094 | 50.079 | 58.735 |
+--------------+--------+---------+----------+
| tvmonitor | 62.405 | 64.059 | 62.435 |
+--------------+--------+---------+----------+
| mIoU | 70.541 | 59.622 | 66.868 |
+--------------+--------+---------+----------+
2023-03-08 12:40:07,198 - dist_train_voc_seg_neg.py - INFO: Iter: 16200; Elasped: 9:57:48; ETA: 2:20:13; LR: 1.346e-05; cls_loss: 0.0482, cls_loss_aux: 0.0497, ptc_loss: 0.2102, ctc_loss: 0.5726, seg_loss: 0.1321...
2023-03-08 12:44:55,902 - dist_train_voc_seg_neg.py - INFO: Iter: 16400; Elasped: 10:02:36; ETA: 2:12:16; LR: 1.282e-05; cls_loss: 0.0356, cls_loss_aux: 0.0406, ptc_loss: 0.2571, ctc_loss: 0.6087, seg_loss: 0.1207...
2023-03-08 12:49:49,893 - dist_train_voc_seg_neg.py - INFO: Iter: 16600; Elasped: 10:07:30; ETA: 2:04:25; LR: 1.218e-05; cls_loss: 0.0423, cls_loss_aux: 0.0421, ptc_loss: 0.2130, ctc_loss: 0.7242, seg_loss: 0.1259...
2023-03-08 12:54:38,436 - dist_train_voc_seg_neg.py - INFO: Iter: 16800; Elasped: 10:12:19; ETA: 1:56:37; LR: 1.153e-05; cls_loss: 0.0424, cls_loss_aux: 0.0431, ptc_loss: 0.2339, ctc_loss: 0.5757, seg_loss: 0.1233...
2023-03-08 12:59:31,836 - dist_train_voc_seg_neg.py - INFO: Iter: 17000; Elasped: 10:17:12; ETA: 1:48:55; LR: 1.088e-05; cls_loss: 0.0376, cls_loss_aux: 0.0369, ptc_loss: 0.2286, ctc_loss: 0.6467, seg_loss: 0.1025...
2023-03-08 13:04:20,211 - dist_train_voc_seg_neg.py - INFO: Iter: 17200; Elasped: 10:22:01; ETA: 1:41:15; LR: 1.023e-05; cls_loss: 0.0355, cls_loss_aux: 0.0409, ptc_loss: 0.2247, ctc_loss: 0.6520, seg_loss: 0.1283...
2023-03-08 13:09:13,793 - dist_train_voc_seg_neg.py - INFO: Iter: 17400; Elasped: 10:26:54; ETA: 1:33:40; LR: 9.569e-06; cls_loss: 0.0428, cls_loss_aux: 0.0445, ptc_loss: 0.2443, ctc_loss: 0.6620, seg_loss: 0.1267...
2023-03-08 13:14:01,922 - dist_train_voc_seg_neg.py - INFO: Iter: 17600; Elasped: 10:31:42; ETA: 1:26:08; LR: 8.904e-06; cls_loss: 0.0418, cls_loss_aux: 0.0419, ptc_loss: 0.2236, ctc_loss: 0.6845, seg_loss: 0.1161...
2023-03-08 13:18:55,303 - dist_train_voc_seg_neg.py - INFO: Iter: 17800; Elasped: 10:36:36; ETA: 1:18:40; LR: 8.233e-06; cls_loss: 0.0363, cls_loss_aux: 0.0353, ptc_loss: 0.2194, ctc_loss: 0.6314, seg_loss: 0.1416...
2023-03-08 13:23:43,516 - dist_train_voc_seg_neg.py - INFO: Iter: 18000; Elasped: 10:41:24; ETA: 1:11:16; LR: 7.557e-06; cls_loss: 0.0462, cls_loss_aux: 0.0459, ptc_loss: 0.2141, ctc_loss: 0.5349, seg_loss: 0.1229...
2023-03-08 13:23:43,517 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 13:45:46,437 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.909125
2023-03-08 13:45:46,437 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 89.924 | 86.098 | 89.784 |
+--------------+--------+---------+----------+
| aeroplane | 73.171 | 76.993 | 76.486 |
+--------------+--------+---------+----------+
| bicycle | 38.416 | 35.799 | 37.041 |
+--------------+--------+---------+----------+
| bird | 70.942 | 70.741 | 72.534 |
+--------------+--------+---------+----------+
| boat | 57.832 | 57.114 | 57.352 |
+--------------+--------+---------+----------+
| bottle | 71.217 | 61.527 | 72.300 |
+--------------+--------+---------+----------+
| bus | 79.929 | 63.502 | 78.569 |
+--------------+--------+---------+----------+
| car | 79.901 | 65.225 | 77.734 |
+--------------+--------+---------+----------+
| cat | 88.545 | 66.390 | 89.325 |
+--------------+--------+---------+----------+
| chair | 43.736 | 43.033 | 25.305 |
+--------------+--------+---------+----------+
| cow | 81.560 | 49.092 | 83.409 |
+--------------+--------+---------+----------+
| diningtable | 61.131 | 49.689 | 53.275 |
+--------------+--------+---------+----------+
| dog | 85.759 | 68.984 | 82.619 |
+--------------+--------+---------+----------+
| horse | 84.497 | 63.929 | 80.181 |
+--------------+--------+---------+----------+
| motorbike | 74.997 | 64.775 | 72.468 |
+--------------+--------+---------+----------+
| person | 77.915 | 68.992 | 75.292 |
+--------------+--------+---------+----------+
| pottedplant | 60.112 | 57.770 | 55.880 |
+--------------+--------+---------+----------+
| sheep | 85.241 | 46.181 | 84.015 |
+--------------+--------+---------+----------+
| sofa | 59.241 | 62.212 | 42.141 |
+--------------+--------+---------+----------+
| train | 55.709 | 49.981 | 54.865 |
+--------------+--------+---------+----------+
| tvmonitor | 58.306 | 62.197 | 60.689 |
+--------------+--------+---------+----------+
| mIoU | 70.385 | 60.487 | 67.679 |
+--------------+--------+---------+----------+
2023-03-08 13:50:39,570 - dist_train_voc_seg_neg.py - INFO: Iter: 18200; Elasped: 11:08:20; ETA: 1:06:05; LR: 6.874e-06; cls_loss: 0.0397, cls_loss_aux: 0.0403, ptc_loss: 0.2109, ctc_loss: 0.6530, seg_loss: 0.1078...
2023-03-08 13:55:27,820 - dist_train_voc_seg_neg.py - INFO: Iter: 18400; Elasped: 11:13:08; ETA: 0:58:32; LR: 6.183e-06; cls_loss: 0.0348, cls_loss_aux: 0.0342, ptc_loss: 0.2400, ctc_loss: 0.7244, seg_loss: 0.1175...
2023-03-08 14:00:22,558 - dist_train_voc_seg_neg.py - INFO: Iter: 18600; Elasped: 11:18:03; ETA: 0:51:02; LR: 5.483e-06; cls_loss: 0.0396, cls_loss_aux: 0.0394, ptc_loss: 0.2103, ctc_loss: 0.4967, seg_loss: 0.1253...
2023-03-08 14:05:11,226 - dist_train_voc_seg_neg.py - INFO: Iter: 18800; Elasped: 11:22:52; ETA: 0:43:35; LR: 4.773e-06; cls_loss: 0.0374, cls_loss_aux: 0.0335, ptc_loss: 0.2219, ctc_loss: 0.6013, seg_loss: 0.1226...
2023-03-08 14:10:04,050 - dist_train_voc_seg_neg.py - INFO: Iter: 19000; Elasped: 11:27:45; ETA: 0:36:11; LR: 4.051e-06; cls_loss: 0.0368, cls_loss_aux: 0.0357, ptc_loss: 0.2344, ctc_loss: 0.6085, seg_loss: 0.1353...
2023-03-08 14:14:53,317 - dist_train_voc_seg_neg.py - INFO: Iter: 19200; Elasped: 11:32:34; ETA: 0:28:51; LR: 3.315e-06; cls_loss: 0.0487, cls_loss_aux: 0.0466, ptc_loss: 0.2321, ctc_loss: 0.6304, seg_loss: 0.1371...
2023-03-08 14:19:45,277 - dist_train_voc_seg_neg.py - INFO: Iter: 19400; Elasped: 11:37:26; ETA: 0:21:34; LR: 2.560e-06; cls_loss: 0.0279, cls_loss_aux: 0.0289, ptc_loss: 0.2033, ctc_loss: 0.6375, seg_loss: 0.1039...
2023-03-08 14:24:35,817 - dist_train_voc_seg_neg.py - INFO: Iter: 19600; Elasped: 11:42:16; ETA: 0:14:19; LR: 1.779e-06; cls_loss: 0.0377, cls_loss_aux: 0.0369, ptc_loss: 0.2071, ctc_loss: 0.6338, seg_loss: 0.0950...
2023-03-08 14:29:27,278 - dist_train_voc_seg_neg.py - INFO: Iter: 19800; Elasped: 11:47:08; ETA: 0:07:08; LR: 9.552e-07; cls_loss: 0.0349, cls_loss_aux: 0.0331, ptc_loss: 0.2252, ctc_loss: 0.6107, seg_loss: 0.1170...
2023-03-08 14:33:53,342 - dist_train_voc_seg_neg.py - INFO: Iter: 20000; Elasped: 11:51:34; ETA: 0:00:00; LR: 8.077e-09; cls_loss: 0.0382, cls_loss_aux: 0.0362, ptc_loss: 0.2251, ctc_loss: 0.5759, seg_loss: 0.1059...
2023-03-08 14:33:53,343 - dist_train_voc_seg_neg.py - INFO: Validating...
2023-03-08 14:55:58,367 - dist_train_voc_seg_neg.py - INFO: val cls score: 0.912178
2023-03-08 14:55:58,367 - dist_train_voc_seg_neg.py - INFO:
+--------------+--------+---------+----------+
| Class | CAM | aux_CAM | Seg_Pred |
+==============+========+=========+==========+
| _background_ | 89.894 | 86.352 | 89.667 |
+--------------+--------+---------+----------+
| aeroplane | 72.510 | 76.906 | 75.622 |
+--------------+--------+---------+----------+
| bicycle | 37.306 | 35.355 | 36.000 |
+--------------+--------+---------+----------+
| bird | 70.308 | 71.586 | 71.343 |
+--------------+--------+---------+----------+
| boat | 50.022 | 56.372 | 45.869 |
+--------------+--------+---------+----------+
| bottle | 71.573 | 61.372 | 72.736 |
+--------------+--------+---------+----------+
| bus | 81.795 | 64.909 | 81.175 |
+--------------+--------+---------+----------+
| car | 81.118 | 67.901 | 78.731 |
+--------------+--------+---------+----------+
| cat | 88.567 | 67.776 | 88.825 |
+--------------+--------+---------+----------+
| chair | 46.013 | 43.617 | 26.475 |
+--------------+--------+---------+----------+
| cow | 81.735 | 51.334 | 84.198 |
+--------------+--------+---------+----------+
| diningtable | 59.565 | 47.597 | 53.272 |
+--------------+--------+---------+----------+
| dog | 85.387 | 69.029 | 81.542 |
+--------------+--------+---------+----------+
| horse | 84.043 | 60.954 | 81.850 |
+--------------+--------+---------+----------+
| motorbike | 75.741 | 65.340 | 72.918 |
+--------------+--------+---------+----------+
| person | 78.853 | 69.454 | 76.457 |
+--------------+--------+---------+----------+
| pottedplant | 61.645 | 57.872 | 58.641 |
+--------------+--------+---------+----------+
| sheep | 84.683 | 48.005 | 83.606 |
+--------------+--------+---------+----------+
| sofa | 61.054 | 66.279 | 43.143 |
+--------------+--------+---------+----------+
| train | 55.671 | 50.373 | 55.390 |
+--------------+--------+---------+----------+
| tvmonitor | 57.343 | 61.171 | 59.789 |
+--------------+--------+---------+----------+
| mIoU | 70.230 | 60.931 | 67.488 |
+--------------+--------+---------+----------+The best performance is 67.67% mIoU, which is about 0.4% lower than the paper (68.1%), and I believe it's tolerable.
I'm not sure why NaN occurs in your experiment, but maybe you can check the ctc_loss, which involves division operations, or try to re-compile the reg_loss.
liaochuanlin
commented
1 year ago Is a file bilateralfilter_batch missing?
When I followed your tips to train, I found that the model could not converge. The training log is as follows:
The following is my envs:
Do you know how to solve the problem?