The results about 11-1 on cityscapes

[zhangchbin]

Hi, @arthurdouillard We have re-cloned the repo and re-trained the model on cityscapes, but we still cannot reproduce these experiments. We have trained the model on the 1-1 setting, and got the mIoU 34.66 (45.24 in your paper). And we also re-trained the model on the 11-1 setting, and got the mIoU 58.85 (62.05 in your paper). So could you please check the training setting again?

[zhangchbin]

And I attach the results file.

[arthurdouillard]

A problem in your configuration may be that you have a larger amount of GPUs than me (4 vs 2) and a lower batch size (6 vs 12). (1) the optimal learning rate may not be the same, and (2) the batch norm statistics may be poorer for you because we don't sync the statistics across GPUs.

I'm rerunning on my side those two experiments and will come back to you when I have results. But my config will be with batch_size=12 and nb_gpus=2.

[zhangchbin]

A problem in your configuration may be that you have a larger amount of GPUs than me (4 vs 2) and a lower batch size (6 vs 12). (1) the optimal learning rate may not be the same, and (2) the batch norm statistics may be poorer for you because we don't sync the statistics across GPUs.

I'm rerunning on my side those two experiments and will come back to you when I have results. But my config will be with batch_size=12 and nb_gpus=2.

Hi, we conducted experiments again. We apply the standard-setting described as you, batchsize=12, gpus=2. The results are: mIoU = 34.66, 58.86, described as above. I have updated the issue.

[arthurdouillard]

My logs were for 1-1:

INFO:rank0:
Total samples: 695.000000
Overall Acc: 0.921154
Mean Acc: 0.526491
FreqW Acc: 0.860704
Mean IoU: 0.452352
Class IoU:
    class 0: 0.96692383
    class 1: 0.60595274
    class 2: 0.8396912
    class 3: 0.25620675
    class 4: 0.15220715
    class 5: 0.06744852
    class 6: 0.0029509393
    class 7: 0.32076913
    class 8: 0.8602092
    class 9: 0.4685076
    class 10: 0.8987206
    class 11: 0.57019883
    class 12: 0.14706148
    class 13: 0.8104589
    class 14: 0.22377394
    class 15: 0.418797
    class 16: 0.5558119
    class 17: 0.014650934
    class 18: 0.41433895
Class Acc:
    class 0: 0.9843136
    class 1: 0.7646799
    class 2: 0.9475544
    class 3: 0.32678273
    class 4: 0.16397442
    class 5: 0.07432122
    class 6: 0.002961872
    class 7: 0.39156127
    class 8: 0.9406667
    class 9: 0.55477184
    class 10: 0.95123327
    class 11: 0.72136724
    class 12: 0.1578011
    class 13: 0.92616725
    class 14: 0.23514673
    class 15: 0.64418685
    class 16: 0.59930676
    class 17: 0.014668353
    class 18: 0.6018618

INFO:rank0: Closing the Logger.
Last Step: 20
Final Mean IoU 45.24
Average Mean IoU 39.93
Mean IoU first 45.24
Mean IoU last 0.0

and for 11-1 (incorrectly called 10-1 in the code):

Total samples: 695.000000
Overall Acc: 0.940431
Mean Acc: 0.706060
FreqW Acc: 0.892914
Mean IoU: 0.620541
Class IoU:
    class 0: 0.9772094
    class 1: 0.70418805
    class 2: 0.87226355
    class 3: 0.43371245
    class 4: 0.48649183
    class 5: 0.1728502
    class 6: 0.32025278
    class 7: 0.4916692
    class 8: 0.8817237
    class 9: 0.5381132
    class 10: 0.9231286
    class 11: 0.66175914
    class 12: 0.44145837
    class 13: 0.8645326
    class 14: 0.59335023
    class 15: 0.69356126
    class 16: 0.7649851
    class 17: 0.39857766
    class 18: 0.5704467
Class Acc:
    class 0: 0.9894886
    class 1: 0.82313305
    class 2: 0.9535187
    class 3: 0.55524725
    class 4: 0.5560459
    class 5: 0.1998001
    class 6: 0.3846253
    class 7: 0.5757105
    class 8: 0.94679874
    class 9: 0.6382271
    class 10: 0.96305674
    class 11: 0.77320135
    class 12: 0.5875096
    class 13: 0.9436415
    class 14: 0.66980577
    class 15: 0.7844055
    class 16: 0.84219396
    class 17: 0.5033585
    class 18: 0.7253713

INFO:rank0: Closing the Logger.
Last Step: 10
Final Mean IoU 62.05
Average Mean IoU 62.37
Mean IoU first 62.05
Mean IoU last 0.0

Nevertheless, I'm re-running once again my code. I'll keep you updated when I have results.

[zhangchbin]

Yeah, hope for your reply. By the way, we also trained the 11-5 (3 tasks), mIoU = 60.24

[zhangchbin]

Hi, @arthurdouillard Is there any progress now?

[arthurdouillard]

I have trouble reproducing my results for Cityscapes-domain. I'm not sure why since I have logs (see my previous message) showing my reported performances.

I have updated the scripts for Cityscapes Domain with better results, sligthly above MiB and much higher than ILT, but still a bit lower than paper results.

For 11-1:

Final Mean IoU 60.48
Average Mean IoU 59.33
Mean IoU first 60.48
Mean IoU last 0.0
cityscapes_domain_11-1_PLOP_backbone-0.0001-logits-0.0001 On GPUs 0,1
Run in 17781s

For 1-1:

Final Mean IoU 43.15
Average Mean IoU 34.59
Mean IoU first 43.15
Mean IoU last 0.0
cityscapes_domain_1-1_PLOP_backbone-0.0001-logits-0.001 On GPUs 0,1
Run in 17534s

If I find a fix I'll update the code base.

[zhangchbin]

Hi, @arthurdouillard , Could you please re-open this issue? If we have some questions, we can ask for your help in this page.

[arthurdouillard]

Hey, I prefer to have atomic issues. For more questions please open a new issue. Thanks!