I do some ablation experiments!

[Lufei-github]

Sorry to bother you again!!!

I do some ablation experiments for this paper. But I found some strange results.

Firstly, I run your original code for GTA5->Cityscapes. I got the following result:

Then, I remove your attention mechanism. I got the following results:

Finally, after remove your attention mechanism, I remove your self-ensembling method. I got the following results:

So I'm very confused about this. Can you give me some advice and opinion on this? Thanks so much!!!

[Lufei-github]

The first result 35.8 miu is close to your 35.7, it's good.

The second result 35.6 miu is also close to your 35.7, it's strange for me. Because I think it should be around 34.6.

The most strange is my third experiment, my NoAdapt experiment, your paper is about 21.2 miu, however I got 32.7.

[YonghaoXu]

Hi, thank you for sharing these reproduction results.

As mentioned in the Empirical Observations in the README doc, the obtained mIoU results may fluctuate a little bit due to the high randomness of the unsupervised domain adaptation setting (no annotations for the target domain data in the training). Thus, the mIoU improvement obtained from the attention mechanism may be unstable in some reproductions.

For your second concern, we have not implemented the NoAdapt method by ourselves. Instead, the NoAdapt results are directly duplicated from the original paper of Hoffman et al. 2016, as per previous literatures like Zhang et al. 2017.

Hoffman, J.; Wang, D.; Yu, F.; and Darrell, T. 2016. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649. Zhang, Y.; David, P.; and Gong, B. 2017. Curriculum domain adaptation for semantic segmentation of urban scenes. In IEEE International Conference on Computer Vision (ICCV).

[Lufei-github]

Oh, forget to tell you. I use the Cityscapes' s training set to train, use the Cityscapes's val set to test, like AdaptSegNet's setting.

En,still my second concern!

OK, I know the NoAdapt result is directly duplicated from the original paper now. But I still think it's very important for a paper. So can you spend a little time in reruning your code without self-ensembling and attention mechanism? Then get the NoAdapt result?

[YonghaoXu]

Hi, I have removed the self-ensembling and attention modules in the code and got a similar result as you did. The detailed accuracy records are: GTA2Cityscapes_epoch1batch500tgt_miu_232 GTA2Cityscapes_epoch1batch1000tgt_miu_242 GTA2Cityscapes_epoch1batch1500tgt_miu_280 GTA2Cityscapes_epoch1batch2000tgt_miu_293 GTA2Cityscapes_epoch1batch4000tgt_miu_293 GTA2Cityscapes_epoch1batch4500tgt_miu_299 GTA2Cityscapes_epoch3batch1000tgt_miu_309 GTA2Cityscapes_epoch4batch1000tgt_miu_314

[Lufei-github]

Thank you for your prompt reply! Thank you so much!