search

LiheYoung / UniMatch

[CVPR 2023] Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation
https://arxiv.org/abs/2208.09910
MIT License
483 stars 60 forks source link

Pre-trained model on the Cityscapes dataset #4

Closed Jack-bo1220 closed 2 years ago

Jack-bo1220 commented 2 years ago

Could you provide a pre-trained model on the Cityscapes dataset? (Under different data partitions) I think it will help other researchers quickly reproduce this excellent work, thank you!

LiheYoung commented 2 years ago

Please wait for two days. I will re-run our UniMatch on the Cityscapes under the splits of 1/16, 1/8, 1/4, and 1/2 with ResNet-101. And then I will upload the trained models and logs for better reproduction.

LiheYoung commented 2 years ago

Here are the logs and trained models on the Cityscapes under different data partitions with ResNet-101 and DeepLabV3Plus.

Partitions 1/16 (186) 1/8 (372) 1/4 (744) 1/2 (1488)
Reported 75.7 77.3 78.7 -
Re-run 76.0 77.6 78.6 79.3
Logs log log log log
Checkpoints ckpt ckpt ckpt ckpt

Please let me know if you have any problem reproducing our results. Thank you!

zhibotian commented 2 years ago

Could you please provide the logs and trained models on the VOC under different data partitions with ResNet-101 and DeepLabV3Plus, Thanks!

LiheYoung commented 2 years ago

Hi @zhibotian, you may refer to https://github.com/LiheYoung/UniMatch/issues/5.

zhibotian commented 2 years ago

Hi @zhibotian, you may refer to #5.

Ok, get it. Thanks a lot.

Jack-bo1220 commented 2 years ago

Thanks a lot. It really help me a lot. Also, I would like to ask how the backbone initialization seed is set? Random or fixed? I do not find a description of seed setting in your code. Can you provide this information to me?

LiheYoung commented 2 years ago

Actually, I did not set any seed in my experiments. The backbone is pre-trained from ImageNet, while the head parameters are randomly initialized by default.

Jack-bo1220 commented 2 years ago

Here are the logs and trained models on the Cityscapes under different data partitions with ResNet-101 and DeepLabV3Plus.

Partitions 1/16 (186) 1/8 (372) 1/4 (744) 1/2 (1488) Reported 75.7 77.3 78.7 - Re-run 76.0 77.6 78.6 79.3 Logs log log log log Checkpoints ckpt ckpt ckpt ckpt Please let me know if you have any problem reproducing our results. Thank you!

I'm sorry to disturb you again. I have evaluated the weights you provided offline using a single GPU, but there are always some small gaps in the results. For example, with the Cityscapes 1/4 setting, I ran a result of 78.66 (you provided: 78.61). With the cityscapes 1/2 setting, I ran a result of 79.39 (you provided: 79.29).

Even though the difference is lower for both, I would like to ask if it is the difference between multi-gpu and single-gpu references? Or is it some other difference (other settings I did not modify). Thanks.

LiheYoung commented 2 years ago

I also observe some minor performance differences when I shuffle the data sequence. But with the same sequence defined in our val.txt and the same number of GPUs, the results are always consistent. So you may try to evaluate these checkpoints with 8 GPUs.

Jack-bo1220 commented 2 years ago

Unfortunately, I don't have 8GPU to perform the verification. But I think a minor difference (only 0.05 or 0.1) is acceptable. Since I excluded the data sequence difference, then I guess the difference may come from the difference between single and multiple GPU? Thanks for your help, what is the approximate performance difference from the data sequence you mentioned?

LiheYoung commented 2 years ago

I think so. The differences caused by data sequences are always less than 0.1%. And if you have any idea about other reasons for the minor fluctuation, please tell me. Thank you.

Jack-bo1220 commented 2 years ago

I think so. The differences caused by data sequences are always less than 0.1%. And if you have any idea about other reasons for the minor fluctuation, please tell me. Thank you.

thanks a lot. 0.1% you mentioned means the gap like between 78.1% and 78.2%(mIOU)?

LiheYoung commented 2 years ago

Yes.

JoyHuYY1412 commented 1 year ago

log

Hi Young,

Thanks for sharing the logs! However, I tried reproducing the results in 1/16 data with 4GPUs*2 images per GPU, but the results were not so good. Currently, the highest performance in the already run 70 epochs is 67.59, 2% lower than your log at the same epoch (69.86%). I am not so sure whether it can still catch up in the later training and whether this gap is normal. Perhaps you could help me check with the logs in the beginning 10 epochs? I found that the mask value is much lower than yours. Maybe that is the case? Thank you so much!

JoyHuYY1412 commented 1 year ago

log

Hi Young,

Thanks for sharing the logs! However, I tried reproducing the results in 1/16 data with 4GPUs*2 images per GPU, but the results were not so good. Currently, the highest performance in the already run 70 epochs is 67.59, 2% lower than your log at the same epoch (69.86%). I am not so sure whether it can still catch up in the later training. Perhaps you could help me check with the logs in the beginning 10 epochs? I found that the mask value is much lower than yours. Maybe that is the case? Thank you so much!

/home/anaconda3/envs/unimatch2/lib/python3.6/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects --local_rank argument to be set, please change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions

FutureWarning, WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

[2023-02-12 20:17:40,374][ INFO] {'backbone': 'resnet101', 'batch_size': 2, 'conf_thresh': 0.95, 'criterion': {'kwargs': {'ignore_index': 255, 'min_kept': 100000, 'thresh': 0.7}, 'name': 'OHEM'}, 'crop_size': 769, 'data_root': '/BS/SETR/work/Projects/UniMatch_ICON/data/cityscapes', 'dataset': 'cityscapes', 'dilations': [12, 24, 36], 'epochs': 240, 'lr': 0.005, 'lr_multi': 1.0, 'multi_grid': True, 'nclass': 19, 'replace_stride_with_dilation': [False, True, True]}

[2023-02-12 20:17:53,387][ INFO] Total params: 59.5M

[2023-02-12 20:17:57,276][ INFO] Start resuming from exp/cityscapes/1_16/unimatch... [2023-02-12 20:17:57,277][ INFO] Start resuming from exp/cityscapes/1_16/unimatch... [2023-02-12 20:17:57,297][ INFO] Start resuming from exp/cityscapes/1_16/unimatch... [2023-02-12 20:17:57,376][ INFO] Start resuming from exp/cityscapes/1_16/unimatch... [2023-02-12 20:17:57,697][ INFO] ===========> Epoch: 0, LR: 0.0050, Previous best: 0.00 [2023-02-12 20:18:28,467][ INFO] Iters: 0, Total loss: 1.502, Loss x: 3.005, Loss s: 0.000, Loss w_fp: 0.000, Mask: 0.000 [2023-02-12 20:20:09,990][ INFO] Iters: 43, Total loss: 0.948, Loss x: 1.890, Loss s: 0.005, Loss w_fp: 0.009, Mask: 0.020 [2023-02-12 20:21:51,586][ INFO] Iters: 86, Total loss: 0.854, Loss x: 1.698, Loss s: 0.008, Loss w_fp: 0.012, Mask: 0.054 [2023-02-12 20:23:33,208][ INFO] Iters: 129, Total loss: 0.785, Loss x: 1.560, Loss s: 0.010, Loss w_fp: 0.011, Mask: 0.062 [2023-02-12 20:25:14,839][ INFO] Iters: 172, Total loss: 0.751, Loss x: 1.492, Loss s: 0.010, Loss w_fp: 0.010, Mask: 0.061 [2023-02-12 20:26:56,547][ INFO] Iters: 215, Total loss: 0.721, Loss x: 1.433, Loss s: 0.009, Loss w_fp: 0.009, Mask: 0.057 [2023-02-12 20:28:38,250][ INFO] Iters: 258, Total loss: 0.695, Loss x: 1.381, Loss s: 0.009, Loss w_fp: 0.009, Mask: 0.055 [2023-02-12 20:30:19,985][ INFO] Iters: 301, Total loss: 0.675, Loss x: 1.342, Loss s: 0.009, Loss w_fp: 0.009, Mask: 0.056 [2023-02-12 20:32:01,728][ INFO] Iters: 344, Total loss: 0.662, Loss x: 1.314, Loss s: 0.011, Loss w_fp: 0.009, Mask: 0.069 [2023-02-12 20:32:29,928][ INFO] Evaluation center_crop >>>> meanIOU: 47.56

[2023-02-12 20:32:33,077][ INFO] ===========> Epoch: 1, LR: 0.0050, Previous best: 47.56 [2023-02-12 20:32:36,496][ INFO] Iters: 0, Total loss: 0.545, Loss x: 1.082, Loss s: 0.011, Loss w_fp: 0.005, Mask: 0.099 [2023-02-12 20:34:18,108][ INFO] Iters: 43, Total loss: 0.602, Loss x: 1.188, Loss s: 0.025, Loss w_fp: 0.008, Mask: 0.184 [2023-02-12 20:35:59,770][ INFO] Iters: 86, Total loss: 0.588, Loss x: 1.161, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.171 [2023-02-12 20:37:41,362][ INFO] Iters: 129, Total loss: 0.583, Loss x: 1.152, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.177 [2023-02-12 20:39:22,972][ INFO] Iters: 172, Total loss: 0.577, Loss x: 1.139, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.191 [2023-02-12 20:41:04,621][ INFO] Iters: 215, Total loss: 0.576, Loss x: 1.137, Loss s: 0.023, Loss w_fp: 0.006, Mask: 0.195 [2023-02-12 20:42:46,271][ INFO] Iters: 258, Total loss: 0.572, Loss x: 1.129, Loss s: 0.023, Loss w_fp: 0.006, Mask: 0.193 [2023-02-12 20:44:27,947][ INFO] Iters: 301, Total loss: 0.570, Loss x: 1.125, Loss s: 0.024, Loss w_fp: 0.006, Mask: 0.197 [2023-02-12 20:46:09,630][ INFO] Iters: 344, Total loss: 0.563, Loss x: 1.111, Loss s: 0.024, Loss w_fp: 0.006, Mask: 0.201 [2023-02-12 20:46:35,203][ INFO] Evaluation center_crop >>>> meanIOU: 54.04

[2023-02-12 20:46:38,287][ INFO] ===========> Epoch: 2, LR: 0.0050, Previous best: 54.04 [2023-02-12 20:46:41,870][ INFO] Iters: 0, Total loss: 0.543, Loss x: 1.082, Loss s: 0.004, Loss w_fp: 0.003, Mask: 0.086 [2023-02-12 20:48:23,522][ INFO] Iters: 43, Total loss: 0.538, Loss x: 1.067, Loss s: 0.016, Loss w_fp: 0.004, Mask: 0.215 [2023-02-12 20:50:05,211][ INFO] Iters: 86, Total loss: 0.539, Loss x: 1.066, Loss s: 0.019, Loss w_fp: 0.004, Mask: 0.226 [2023-02-12 20:51:46,828][ INFO] Iters: 129, Total loss: 0.547, Loss x: 1.081, Loss s: 0.020, Loss w_fp: 0.004, Mask: 0.228 [2023-02-12 20:53:28,461][ INFO] Iters: 172, Total loss: 0.531, Loss x: 1.049, Loss s: 0.019, Loss w_fp: 0.004, Mask: 0.226 [2023-02-12 20:55:10,070][ INFO] Iters: 215, Total loss: 0.524, Loss x: 1.036, Loss s: 0.020, Loss w_fp: 0.004, Mask: 0.228 [2023-02-12 20:56:51,823][ INFO] Iters: 258, Total loss: 0.522, Loss x: 1.033, Loss s: 0.020, Loss w_fp: 0.004, Mask: 0.227 [2023-02-12 20:58:33,645][ INFO] Iters: 301, Total loss: 0.516, Loss x: 1.020, Loss s: 0.020, Loss w_fp: 0.005, Mask: 0.233 [2023-02-12 21:00:15,524][ INFO] Iters: 344, Total loss: 0.514, Loss x: 1.016, Loss s: 0.019, Loss w_fp: 0.005, Mask: 0.240 [2023-02-12 21:00:41,295][ INFO] Evaluation center_crop >>>> meanIOU: 58.39

[2023-02-12 21:00:44,620][ INFO] ===========> Epoch: 3, LR: 0.0049, Previous best: 58.39 [2023-02-12 21:00:48,105][ INFO] Iters: 0, Total loss: 0.463, Loss x: 0.907, Loss s: 0.031, Loss w_fp: 0.007, Mask: 0.314 [2023-02-12 21:02:29,794][ INFO] Iters: 43, Total loss: 0.517, Loss x: 1.018, Loss s: 0.026, Loss w_fp: 0.006, Mask: 0.319 [2023-02-12 21:04:11,651][ INFO] Iters: 86, Total loss: 0.491, Loss x: 0.968, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.294 [2023-02-12 21:05:53,638][ INFO] Iters: 129, Total loss: 0.489, Loss x: 0.964, Loss s: 0.020, Loss w_fp: 0.006, Mask: 0.292 [2023-02-12 21:07:35,457][ INFO] Iters: 172, Total loss: 0.480, Loss x: 0.948, Loss s: 0.020, Loss w_fp: 0.006, Mask: 0.307 [2023-02-12 21:09:17,309][ INFO] Iters: 215, Total loss: 0.477, Loss x: 0.941, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.309 [2023-02-12 21:10:59,020][ INFO] Iters: 258, Total loss: 0.482, Loss x: 0.950, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.327 [2023-02-12 21:12:40,740][ INFO] Iters: 301, Total loss: 0.485, Loss x: 0.957, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.342 [2023-02-12 21:14:22,545][ INFO] Iters: 344, Total loss: 0.485, Loss x: 0.957, Loss s: 0.022, Loss w_fp: 0.006, Mask: 0.348 [2023-02-12 21:14:48,305][ INFO] Evaluation center_crop >>>> meanIOU: 59.79

[2023-02-12 21:14:51,517][ INFO] ===========> Epoch: 4, LR: 0.0049, Previous best: 59.79 [2023-02-12 21:14:55,091][ INFO] Iters: 0, Total loss: 0.522, Loss x: 1.038, Loss s: 0.008, Loss w_fp: 0.004, Mask: 0.250 [2023-02-12 21:16:36,610][ INFO] Iters: 43, Total loss: 0.495, Loss x: 0.973, Loss s: 0.027, Loss w_fp: 0.006, Mask: 0.410 [2023-02-12 21:18:18,172][ INFO] Iters: 86, Total loss: 0.486, Loss x: 0.956, Loss s: 0.028, Loss w_fp: 0.006, Mask: 0.381 [2023-02-12 21:19:59,824][ INFO] Iters: 129, Total loss: 0.479, Loss x: 0.942, Loss s: 0.026, Loss w_fp: 0.006, Mask: 0.378 [2023-02-12 21:21:41,471][ INFO] Iters: 172, Total loss: 0.471, Loss x: 0.924, Loss s: 0.027, Loss w_fp: 0.006, Mask: 0.396 [2023-02-12 21:23:23,140][ INFO] Iters: 215, Total loss: 0.463, Loss x: 0.911, Loss s: 0.025, Loss w_fp: 0.007, Mask: 0.409 [2023-02-12 21:25:04,784][ INFO] Iters: 258, Total loss: 0.461, Loss x: 0.907, Loss s: 0.025, Loss w_fp: 0.007, Mask: 0.416 [2023-02-12 21:26:46,563][ INFO] Iters: 301, Total loss: 0.461, Loss x: 0.905, Loss s: 0.026, Loss w_fp: 0.007, Mask: 0.420 [2023-02-12 21:28:28,218][ INFO] Iters: 344, Total loss: 0.460, Loss x: 0.903, Loss s: 0.025, Loss w_fp: 0.007, Mask: 0.430 [2023-02-12 21:28:54,274][ INFO] Evaluation center_crop >>>> meanIOU: 56.80

[2023-02-12 21:28:56,343][ INFO] ===========> Epoch: 5, LR: 0.0049, Previous best: 59.79 [2023-02-12 21:28:59,775][ INFO] Iters: 0, Total loss: 0.445, Loss x: 0.873, Loss s: 0.030, Loss w_fp: 0.003, Mask: 0.285 [2023-02-12 21:30:41,321][ INFO] Iters: 43, Total loss: 0.491, Loss x: 0.965, Loss s: 0.026, Loss w_fp: 0.007, Mask: 0.444 [2023-02-12 21:32:22,979][ INFO] Iters: 86, Total loss: 0.473, Loss x: 0.929, Loss s: 0.028, Loss w_fp: 0.007, Mask: 0.469 [2023-02-12 21:34:04,582][ INFO] Iters: 129, Total loss: 0.469, Loss x: 0.919, Loss s: 0.031, Loss w_fp: 0.008, Mask: 0.475 [2023-02-12 21:35:46,186][ INFO] Iters: 172, Total loss: 0.487, Loss x: 0.954, Loss s: 0.032, Loss w_fp: 0.008, Mask: 0.462 [2023-02-12 21:37:27,891][ INFO] Iters: 215, Total loss: 0.477, Loss x: 0.935, Loss s: 0.031, Loss w_fp: 0.008, Mask: 0.465 [2023-02-12 21:39:09,620][ INFO] Iters: 258, Total loss: 0.475, Loss x: 0.930, Loss s: 0.034, Loss w_fp: 0.008, Mask: 0.463 [2023-02-12 21:40:51,318][ INFO] Iters: 301, Total loss: 0.468, Loss x: 0.915, Loss s: 0.035, Loss w_fp: 0.009, Mask: 0.470 [2023-02-12 21:42:32,988][ INFO] Iters: 344, Total loss: 0.466, Loss x: 0.909, Loss s: 0.035, Loss w_fp: 0.009, Mask: 0.483 [2023-02-12 21:42:58,965][ INFO] Evaluation center_crop >>>> meanIOU: 61.75

[2023-02-12 21:43:02,027][ INFO] ===========> Epoch: 6, LR: 0.0049, Previous best: 61.75 [2023-02-12 21:43:05,378][ INFO] Iters: 0, Total loss: 0.504, Loss x: 0.973, Loss s: 0.056, Loss w_fp: 0.015, Mask: 0.582 [2023-02-12 21:44:46,937][ INFO] Iters: 43, Total loss: 0.446, Loss x: 0.872, Loss s: 0.033, Loss w_fp: 0.008, Mask: 0.551 [2023-02-12 21:46:28,642][ INFO] Iters: 86, Total loss: 0.436, Loss x: 0.853, Loss s: 0.031, Loss w_fp: 0.008, Mask: 0.539 [2023-02-12 21:48:10,332][ INFO] Iters: 129, Total loss: 0.443, Loss x: 0.864, Loss s: 0.034, Loss w_fp: 0.009, Mask: 0.541 [2023-02-12 21:49:52,002][ INFO] Iters: 172, Total loss: 0.448, Loss x: 0.872, Loss s: 0.036, Loss w_fp: 0.010, Mask: 0.550 [2023-02-12 21:51:33,672][ INFO] Iters: 215, Total loss: 0.442, Loss x: 0.861, Loss s: 0.035, Loss w_fp: 0.010, Mask: 0.565 [2023-02-12 21:53:15,459][ INFO] Iters: 258, Total loss: 0.443, Loss x: 0.864, Loss s: 0.036, Loss w_fp: 0.009, Mask: 0.560 [2023-02-12 21:54:57,305][ INFO] Iters: 301, Total loss: 0.436, Loss x: 0.850, Loss s: 0.036, Loss w_fp: 0.009, Mask: 0.565 [2023-02-12 21:56:39,116][ INFO] Iters: 344, Total loss: 0.435, Loss x: 0.847, Loss s: 0.037, Loss w_fp: 0.009, Mask: 0.567 [2023-02-12 21:57:05,077][ INFO] Evaluation center_crop >>>> meanIOU: 61.71

[2023-02-12 21:57:07,247][ INFO] ===========> Epoch: 7, LR: 0.0049, Previous best: 61.75 [2023-02-12 21:57:10,768][ INFO] Iters: 0, Total loss: 0.448, Loss x: 0.883, Loss s: 0.021, Loss w_fp: 0.006, Mask: 0.665 [2023-02-12 21:58:52,386][ INFO] Iters: 43, Total loss: 0.450, Loss x: 0.877, Loss s: 0.037, Loss w_fp: 0.010, Mask: 0.624 [2023-02-12 22:00:33,994][ INFO] Iters: 86, Total loss: 0.440, Loss x: 0.856, Loss s: 0.039, Loss w_fp: 0.010, Mask: 0.613 [2023-02-12 22:02:15,673][ INFO] Iters: 129, Total loss: 0.430, Loss x: 0.836, Loss s: 0.041, Loss w_fp: 0.009, Mask: 0.617 [2023-02-12 22:03:57,410][ INFO] Iters: 172, Total loss: 0.423, Loss x: 0.822, Loss s: 0.040, Loss w_fp: 0.009, Mask: 0.614 [2023-02-12 22:05:39,340][ INFO] Iters: 215, Total loss: 0.424, Loss x: 0.824, Loss s: 0.039, Loss w_fp: 0.008, Mask: 0.619 [2023-02-12 22:07:21,190][ INFO] Iters: 258, Total loss: 0.425, Loss x: 0.827, Loss s: 0.037, Loss w_fp: 0.008, Mask: 0.614 [2023-02-12 22:09:03,019][ INFO] Iters: 301, Total loss: 0.425, Loss x: 0.828, Loss s: 0.037, Loss w_fp: 0.008, Mask: 0.614 [2023-02-12 22:10:44,888][ INFO] Iters: 344, Total loss: 0.419, Loss x: 0.816, Loss s: 0.038, Loss w_fp: 0.008, Mask: 0.617 [2023-02-12 22:11:10,992][ INFO] Evaluation center_crop >>>> meanIOU: 62.56

[2023-02-12 22:11:14,471][ INFO] ===========> Epoch: 8, LR: 0.0048, Previous best: 62.56 [2023-02-12 22:11:17,823][ INFO] Iters: 0, Total loss: 0.441, Loss x: 0.871, Loss s: 0.018, Loss w_fp: 0.005, Mask: 0.563 [2023-02-12 22:12:59,536][ INFO] Iters: 43, Total loss: 0.454, Loss x: 0.890, Loss s: 0.031, Loss w_fp: 0.005, Mask: 0.564 [2023-02-12 22:14:41,376][ INFO] Iters: 86, Total loss: 0.419, Loss x: 0.819, Loss s: 0.032, Loss w_fp: 0.006, Mask: 0.575 [2023-02-12 22:16:23,199][ INFO] Iters: 129, Total loss: 0.413, Loss x: 0.806, Loss s: 0.035, Loss w_fp: 0.006, Mask: 0.608

LiheYoung commented 1 year ago

According to the log, the difference is that I use 1 image per card, while you use 2 images.

The OHEM min_kept hyper-parameter is conditioned on the number of labeled images in each card. Since you have doubled the labeled images per card, I recommend also doubling the min_kept to 200,000. Nevertheless, if you want to fully reproduce our logs, the best practice may be using 1 image per card with 8 cards.

JoyHuYY1412 commented 1 year ago

According to the log, the difference is that I use 1 image per card, while you use 2 images.

The OHEM min_kept hyper-parameter is conditioned on the number of labeled images in each card. Since you have doubled the labeled images per card, I recommend also doubling the min_kept to 200,000. Nevertheless, if you want to fully reproduce our logs, the best practice may be using 1 image per card with 8 cards.

Hi Young. Thank you for your advice. After I tried with 8 GPUs, the initial mask ratio seemed to be normal. However, the final result with 8 GPU is 74.70% (comparable with 4GPU results of 74.49% FYI). I checked the logs and found the highest accuracy before sliding window evaluation is 70.45%, which is also lower than your log.

/home/anaconda3/envs/unimatch2/lib/python3.6/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects --local_rank argument to be set, please change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions

FutureWarning, WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

[2023-02-15 17:22:10,644][ INFO] {'backbone': 'resnet101', 'batch_size': 1, 'conf_thresh': 0.95, 'criterion': {'kwargs': {'ignore_index': 255, 'min_kept': 100000, 'thresh': 0.7}, 'name': 'OHEM'}, 'crop_size': 769, 'data_root': '/BS/SETR/work/Projects/UniMatch_ICON/data/cityscapes', 'dataset': 'cityscapes', 'dilations': [12, 24, 36], 'epochs': 240, 'lr': 0.005, 'lr_multi': 1.0, 'multi_grid': True, 'nclass': 19, 'replace_stride_with_dilation': [False, True, True]}

[2023-02-15 17:22:13,955][ INFO] Total params: 59.5M

[2023-02-15 17:22:27,051][ INFO] ===========> Epoch: 217, LR: 0.0006, Previous best: 70.45 [2023-02-15 17:23:11,050][ INFO] Iters: 0, Total loss: 0.011, Loss x: 0.020, Loss s: 0.001, Loss w_fp: 0.000, Mask: 0.995 [2023-02-15 17:24:08,730][ INFO] Iters: 43, Total loss: 0.060, Loss x: 0.105, Loss s: 0.027, Loss w_fp: 0.005, Mask: 0.947 [2023-02-15 17:25:05,775][ INFO] Iters: 86, Total loss: 0.063, Loss x: 0.109, Loss s: 0.031, Loss w_fp: 0.005, Mask: 0.944 [2023-02-15 17:26:02,629][ INFO] Iters: 129, Total loss: 0.065, Loss x: 0.111, Loss s: 0.034, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:26:59,569][ INFO] Iters: 172, Total loss: 0.066, Loss x: 0.114, Loss s: 0.033, Loss w_fp: 0.005, Mask: 0.942 [2023-02-15 17:27:56,447][ INFO] Iters: 215, Total loss: 0.068, Loss x: 0.115, Loss s: 0.037, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:28:53,283][ INFO] Iters: 258, Total loss: 0.068, Loss x: 0.116, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 17:29:50,094][ INFO] Iters: 301, Total loss: 0.067, Loss x: 0.115, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:30:47,222][ INFO] Iters: 344, Total loss: 0.066, Loss x: 0.114, Loss s: 0.034, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:31:01,883][ INFO] Evaluation center_crop >>>> meanIOU: 69.90

[2023-02-15 17:31:03,703][ INFO] ===========> Epoch: 218, LR: 0.0006, Previous best: 70.45 [2023-02-15 17:31:05,826][ INFO] Iters: 0, Total loss: 0.092, Loss x: 0.176, Loss s: 0.013, Loss w_fp: 0.002, Mask: 0.952 [2023-02-15 17:32:02,543][ INFO] Iters: 43, Total loss: 0.063, Loss x: 0.109, Loss s: 0.030, Loss w_fp: 0.003, Mask: 0.937 [2023-02-15 17:32:59,171][ INFO] Iters: 86, Total loss: 0.067, Loss x: 0.114, Loss s: 0.037, Loss w_fp: 0.004, Mask: 0.940 [2023-02-15 17:33:55,954][ INFO] Iters: 129, Total loss: 0.067, Loss x: 0.115, Loss s: 0.037, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 17:34:52,928][ INFO] Iters: 172, Total loss: 0.065, Loss x: 0.110, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:35:49,831][ INFO] Iters: 215, Total loss: 0.067, Loss x: 0.114, Loss s: 0.037, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:36:46,368][ INFO] Iters: 258, Total loss: 0.067, Loss x: 0.114, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:37:43,108][ INFO] Iters: 301, Total loss: 0.068, Loss x: 0.116, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:38:40,064][ INFO] Iters: 344, Total loss: 0.067, Loss x: 0.115, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:38:54,548][ INFO] Evaluation center_crop >>>> meanIOU: 69.73

[2023-02-15 17:38:56,438][ INFO] ===========> Epoch: 219, LR: 0.0006, Previous best: 70.45 [2023-02-15 17:38:58,564][ INFO] Iters: 0, Total loss: 0.124, Loss x: 0.247, Loss s: 0.000, Loss w_fp: 0.000, Mask: 1.000 [2023-02-15 17:39:55,383][ INFO] Iters: 43, Total loss: 0.058, Loss x: 0.100, Loss s: 0.029, Loss w_fp: 0.003, Mask: 0.948 [2023-02-15 17:40:52,321][ INFO] Iters: 86, Total loss: 0.061, Loss x: 0.105, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.948 [2023-02-15 17:41:49,140][ INFO] Iters: 129, Total loss: 0.063, Loss x: 0.109, Loss s: 0.030, Loss w_fp: 0.003, Mask: 0.946 [2023-02-15 17:42:45,886][ INFO] Iters: 172, Total loss: 0.062, Loss x: 0.108, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 17:43:42,740][ INFO] Iters: 215, Total loss: 0.062, Loss x: 0.107, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:44:39,465][ INFO] Iters: 258, Total loss: 0.062, Loss x: 0.108, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:45:36,309][ INFO] Iters: 301, Total loss: 0.063, Loss x: 0.109, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:46:33,310][ INFO] Iters: 344, Total loss: 0.063, Loss x: 0.109, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:46:47,653][ INFO] Evaluation center_crop >>>> meanIOU: 69.68

[2023-02-15 17:46:49,536][ INFO] ===========> Epoch: 220, LR: 0.0005, Previous best: 70.45 [2023-02-15 17:46:51,657][ INFO] Iters: 0, Total loss: 0.037, Loss x: 0.052, Loss s: 0.041, Loss w_fp: 0.003, Mask: 0.923 [2023-02-15 17:47:48,588][ INFO] Iters: 43, Total loss: 0.064, Loss x: 0.111, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.949 [2023-02-15 17:48:45,578][ INFO] Iters: 86, Total loss: 0.064, Loss x: 0.110, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.945 [2023-02-15 17:49:42,558][ INFO] Iters: 129, Total loss: 0.062, Loss x: 0.108, Loss s: 0.028, Loss w_fp: 0.004, Mask: 0.946 [2023-02-15 17:50:39,713][ INFO] Iters: 172, Total loss: 0.062, Loss x: 0.108, Loss s: 0.028, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:51:36,758][ INFO] Iters: 215, Total loss: 0.061, Loss x: 0.105, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 17:52:33,654][ INFO] Iters: 258, Total loss: 0.061, Loss x: 0.105, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:53:30,614][ INFO] Iters: 301, Total loss: 0.060, Loss x: 0.104, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 17:54:27,575][ INFO] Iters: 344, Total loss: 0.060, Loss x: 0.103, Loss s: 0.029, Loss w_fp: 0.003, Mask: 0.945 [2023-02-15 17:55:44,076][ INFO] Evaluation sliding_window >>>> meanIOU: 74.34

[2023-02-15 17:55:46,811][ INFO] ===========> Epoch: 221, LR: 0.0005, Previous best: 74.34 [2023-02-15 17:55:49,350][ INFO] Iters: 0, Total loss: 0.068, Loss x: 0.110, Loss s: 0.045, Loss w_fp: 0.007, Mask: 0.937 [2023-02-15 17:56:46,078][ INFO] Iters: 43, Total loss: 0.070, Loss x: 0.126, Loss s: 0.024, Loss w_fp: 0.005, Mask: 0.941 [2023-02-15 17:57:42,938][ INFO] Iters: 86, Total loss: 0.068, Loss x: 0.120, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 17:58:39,813][ INFO] Iters: 129, Total loss: 0.066, Loss x: 0.115, Loss s: 0.029, Loss w_fp: 0.005, Mask: 0.944 [2023-02-15 17:59:36,931][ INFO] Iters: 172, Total loss: 0.064, Loss x: 0.112, Loss s: 0.027, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:00:33,852][ INFO] Iters: 215, Total loss: 0.065, Loss x: 0.112, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:01:30,735][ INFO] Iters: 258, Total loss: 0.063, Loss x: 0.109, Loss s: 0.028, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:02:27,547][ INFO] Iters: 301, Total loss: 0.063, Loss x: 0.109, Loss s: 0.028, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:03:24,454][ INFO] Iters: 344, Total loss: 0.063, Loss x: 0.110, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:04:15,368][ INFO] Evaluation sliding_window >>>> meanIOU: 74.55

[2023-02-15 18:04:18,163][ INFO] ===========> Epoch: 222, LR: 0.0005, Previous best: 74.55 [2023-02-15 18:04:20,287][ INFO] Iters: 0, Total loss: 0.078, Loss x: 0.144, Loss s: 0.017, Loss w_fp: 0.007, Mask: 0.959 [2023-02-15 18:05:16,800][ INFO] Iters: 43, Total loss: 0.063, Loss x: 0.104, Loss s: 0.039, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:06:13,388][ INFO] Iters: 86, Total loss: 0.068, Loss x: 0.117, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.939 [2023-02-15 18:07:10,166][ INFO] Iters: 129, Total loss: 0.067, Loss x: 0.115, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.940 [2023-02-15 18:08:06,995][ INFO] Iters: 172, Total loss: 0.065, Loss x: 0.110, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.941 [2023-02-15 18:09:03,605][ INFO] Iters: 215, Total loss: 0.064, Loss x: 0.109, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:10:00,253][ INFO] Iters: 258, Total loss: 0.065, Loss x: 0.112, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:10:56,983][ INFO] Iters: 301, Total loss: 0.066, Loss x: 0.114, Loss s: 0.033, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 18:11:53,954][ INFO] Iters: 344, Total loss: 0.065, Loss x: 0.112, Loss s: 0.034, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 18:12:45,473][ INFO] Evaluation sliding_window >>>> meanIOU: 74.12

[2023-02-15 18:12:47,395][ INFO] ===========> Epoch: 223, LR: 0.0005, Previous best: 74.55 [2023-02-15 18:12:49,464][ INFO] Iters: 0, Total loss: 0.051, Loss x: 0.052, Loss s: 0.095, Loss w_fp: 0.004, Mask: 0.960 [2023-02-15 18:13:46,068][ INFO] Iters: 43, Total loss: 0.060, Loss x: 0.101, Loss s: 0.033, Loss w_fp: 0.004, Mask: 0.941 [2023-02-15 18:14:43,095][ INFO] Iters: 86, Total loss: 0.061, Loss x: 0.105, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.941 [2023-02-15 18:15:40,011][ INFO] Iters: 129, Total loss: 0.061, Loss x: 0.103, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.940 [2023-02-15 18:16:36,947][ INFO] Iters: 172, Total loss: 0.063, Loss x: 0.108, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.941 [2023-02-15 18:17:33,660][ INFO] Iters: 215, Total loss: 0.062, Loss x: 0.107, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 18:18:30,418][ INFO] Iters: 258, Total loss: 0.061, Loss x: 0.105, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:19:27,148][ INFO] Iters: 301, Total loss: 0.061, Loss x: 0.104, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:20:23,818][ INFO] Iters: 344, Total loss: 0.061, Loss x: 0.104, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 18:21:14,834][ INFO] Evaluation sliding_window >>>> meanIOU: 74.48

[2023-02-15 18:21:16,952][ INFO] ===========> Epoch: 224, LR: 0.0004, Previous best: 74.55 [2023-02-15 18:21:19,051][ INFO] Iters: 0, Total loss: 0.046, Loss x: 0.087, Loss s: 0.010, Loss w_fp: 0.002, Mask: 0.973 [2023-02-15 18:22:15,669][ INFO] Iters: 43, Total loss: 0.069, Loss x: 0.121, Loss s: 0.031, Loss w_fp: 0.003, Mask: 0.941 [2023-02-15 18:23:12,323][ INFO] Iters: 86, Total loss: 0.066, Loss x: 0.115, Loss s: 0.030, Loss w_fp: 0.003, Mask: 0.944 [2023-02-15 18:24:09,013][ INFO] Iters: 129, Total loss: 0.066, Loss x: 0.115, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:25:05,655][ INFO] Iters: 172, Total loss: 0.066, Loss x: 0.115, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 18:26:02,636][ INFO] Iters: 215, Total loss: 0.066, Loss x: 0.115, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:26:59,342][ INFO] Iters: 258, Total loss: 0.067, Loss x: 0.117, Loss s: 0.029, Loss w_fp: 0.004, Mask: 0.942 [2023-02-15 18:27:56,125][ INFO] Iters: 301, Total loss: 0.068, Loss x: 0.118, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.941 [2023-02-15 18:28:53,092][ INFO] Iters: 344, Total loss: 0.067, Loss x: 0.116, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.941 [2023-02-15 18:29:44,014][ INFO] Evaluation sliding_window >>>> meanIOU: 74.45

[2023-02-15 18:29:46,033][ INFO] ===========> Epoch: 225, LR: 0.0004, Previous best: 74.55 [2023-02-15 18:29:48,120][ INFO] Iters: 0, Total loss: 0.039, Loss x: 0.066, Loss s: 0.020, Loss w_fp: 0.002, Mask: 0.974 [2023-02-15 18:30:44,912][ INFO] Iters: 43, Total loss: 0.066, Loss x: 0.109, Loss s: 0.043, Loss w_fp: 0.004, Mask: 0.937 [2023-02-15 18:31:41,894][ INFO] Iters: 86, Total loss: 0.063, Loss x: 0.106, Loss s: 0.037, Loss w_fp: 0.004, Mask: 0.938 [2023-02-15 18:32:38,868][ INFO] Iters: 129, Total loss: 0.062, Loss x: 0.103, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.939 [2023-02-15 18:33:35,892][ INFO] Iters: 172, Total loss: 0.063, Loss x: 0.107, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.936 [2023-02-15 18:34:32,888][ INFO] Iters: 215, Total loss: 0.065, Loss x: 0.110, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.936 [2023-02-15 18:35:29,913][ INFO] Iters: 258, Total loss: 0.064, Loss x: 0.108, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.937 [2023-02-15 18:36:26,858][ INFO] Iters: 301, Total loss: 0.064, Loss x: 0.108, Loss s: 0.036, Loss w_fp: 0.004, Mask: 0.936 [2023-02-15 18:37:23,814][ INFO] Iters: 344, Total loss: 0.063, Loss x: 0.107, Loss s: 0.034, Loss w_fp: 0.004, Mask: 0.937 [2023-02-15 18:38:15,571][ INFO] Evaluation sliding_window >>>> meanIOU: 74.38

[2023-02-15 18:38:17,509][ INFO] ===========> Epoch: 226, LR: 0.0004, Previous best: 74.55 [2023-02-15 18:38:19,647][ INFO] Iters: 0, Total loss: 0.046, Loss x: 0.077, Loss s: 0.027, Loss w_fp: 0.004, Mask: 0.969 [2023-02-15 18:39:16,248][ INFO] Iters: 43, Total loss: 0.065, Loss x: 0.116, Loss s: 0.024, Loss w_fp: 0.003, Mask: 0.950 [2023-02-15 18:40:12,942][ INFO] Iters: 86, Total loss: 0.054, Loss x: 0.092, Loss s: 0.028, Loss w_fp: 0.003, Mask: 0.948 [2023-02-15 18:41:09,830][ INFO] Iters: 129, Total loss: 0.056, Loss x: 0.097, Loss s: 0.026, Loss w_fp: 0.003, Mask: 0.947 [2023-02-15 18:42:06,715][ INFO] Iters: 172, Total loss: 0.058, Loss x: 0.099, Loss s: 0.030, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:43:03,686][ INFO] Iters: 215, Total loss: 0.060, Loss x: 0.102, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.945 [2023-02-15 18:44:00,430][ INFO] Iters: 258, Total loss: 0.061, Loss x: 0.104, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:44:57,014][ INFO] Iters: 301, Total loss: 0.062, Loss x: 0.107, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:45:53,783][ INFO] Iters: 344, Total loss: 0.063, Loss x: 0.107, Loss s: 0.034, Loss w_fp: 0.004, Mask: 0.943 [2023-02-15 18:46:44,627][ INFO] Evaluation sliding_window >>>> meanIOU: 74.42

[2023-02-15 18:46:46,535][ INFO] ===========> Epoch: 227, LR: 0.0004, Previous best: 74.55 [2023-02-15 18:46:48,709][ INFO] Iters: 0, Total loss: 0.152, Loss x: 0.292, Loss s: 0.022, Loss w_fp: 0.002, Mask: 0.973 [2023-02-15 18:47:45,475][ INFO] Iters: 43, Total loss: 0.074, Loss x: 0.129, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.947 [2023-02-15 18:48:42,514][ INFO] Iters: 86, Total loss: 0.063, Loss x: 0.110, Loss s: 0.028, Loss w_fp: 0.004, Mask: 0.945 [2023-02-15 18:49:39,331][ INFO] Iters: 129, Total loss: 0.063, Loss x: 0.109, Loss s: 0.031, Loss w_fp: 0.004, Mask: 0.944 [2023-02-15 18:50:36,100][ INFO] Iters: 172, Total loss: 0.066, Loss x: 0.113, Loss s: 0.032, Loss w_fp: 0.004, Mask: 0.945 [2023-02-15 18:51:32,916][ INFO] Iters: 215, Total loss: 0.066, Loss x: 0.113, Loss s: 0.035, Loss w_fp: 0.004, Mask: 0.943

LiheYoung commented 1 year ago

My officially reported result is 75.7, while as I re-run the code, the obtained result is 76.0 (as present in the log). And your result is 74.5. I am sorry that I currently can not think of other reasons for this except randomness and different environments. So could you try other splits? such as 1/8 or 1/4, which will be more stable. If resources are allowed, you can also give a second try to the 1/16 split. Thank you.

LiheYoung commented 1 year ago

According to our later explorations, there is another practice to improve performance on Cityscapes by 1% generally. Please modify the Cityscapes config:

And if you hope to decrease training time and use less GPU memory, please modify the Cityscapes config:

JoyHuYY1412 commented 1 year ago

According to our later explorations, there is another practice to improve performance on Cityscapes by 1% generally. Please modify the Cityscapes config:

  • conf_thresh: 0.95 -> 0

And if you hope to decrease training time and use less GPU memory, please modify the Cityscapes config:

  • replace_stride_with_dilation: [False, True, True] -> [False, False, True]
  • dilations: [12, 24, 36] -> [6, 12, 18]
  • multi_grid: True -> False

Thank you for your reply! I will check other splits and try your suggestions. Thanks again.