search

lhoyer / HRDA

[ECCV22] Official Implementation of HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation
Other
235 stars 32 forks source link

About Reproducing the results shown in the Paper #9

Closed kagawa588 closed 2 years ago

kagawa588 commented 2 years ago

Dear Lukas,

I am interested in your recent great work HRDA and thanks for sharing your code. I would like to run the code you provided and reproduce the results.

I followed the setting in "experiments.py" file, but I found that the results I got can not match those provided in the Paper. Should I change some of the default setting to reach the results provided?

The attached image is the experiment data I recorded for Table 1 in paper. I run it on a RTX 6000 GPU. table 1

(The mIoUs for gta5 - cityscapes are 61, 66.92, 63.31 at 3 random seeds, and for synthia - cityscapes are 55.76, 55.17, 56.14 at 3 random seeds.)

I also copied my environment information below:

sys.platform: linux Python: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] CUDA available: True GPU 0: Quadro RTX 6000 CUDA_HOME: /usr/local/cuda NVCC: Build cuda_11.2.r11.2/compiler.29618528_0 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 PyTorch: 1.7.1+cu110 PyTorch compiling details: PyTorch built with:

TorchVision: 0.8.2+cu110 OpenCV: 4.4.0 MMCV: 1.3.7 MMCV Compiler: GCC 9.4 MMCV CUDA Compiler: 11.2 MMSegmentation: 0.16.0+a57d967

Thank you so much!

lhoyer commented 2 years ago

Dear Mu,

Thank you for your interest in our work.

Have you already converted your Synthia->Cityscapes results from 19 to 16 classes as explained by the README.md in Section "Training"? If you haven't done it so far, your results would actually match the ones in the paper after conversion.

However, your GTA->Cityscapes results are indeed off by a large margin. This is rather weird as I used the public code to check reproducibility and it was well reproducible in my environment. I would recommend that you set up a new environment and carefully go through the setup instructions again. Please, make sure that you convert the datasets. Also, please run the provided checkpoint to see if the evaluation gives you the same results as provided in the readme. Further, please try running python run_experiments.py --config configs/hrda/gtaHR2csHR_hrda.py.

If it still does not work, I would additionally recommend trying to reproduce DAFormer (https://github.com/lhoyer/DAFormer), which is the basis for HRDA. The framework and dependencies are very similar and I know that it was successfully reproduced by third parties.

Please, let me know if you were able to solve the problem.

Best regards, Lukas

lhoyer commented 2 years ago

I'm not sure if it is relevant, but your installed library versions differ from the ones used in HRDA. Just to be sure, please use python 3.8.5, CUDA 11.0 (also for MMCV CUDA Compiler), OpenCV 4.4.0.46, and GCC 6.3.0

kagawa588 commented 2 years ago

Dear Lukas,

Thanks so much for your detailed explanation and useful guidance! I actually did not consider that only 16 classes are counted in synthia. I will run the code again according to your recommendation and give you feedback!

Best Regards Mu

kagawa588 commented 2 years ago

Dear Lukas,

Congratulations on your acceptance to ECCV!

I have successfully reproduced the results you show in the DAFormer and HRDA paper. The previous problem was that there may have been a problem with my GTA dataset when it was unpacked (GTA dataset is very large). Thanks so much for your help!

Would you mind if I close this issue?

Best Regards Mu

lhoyer commented 2 years ago

Dear Mu,

Thanks a lot! I'm happy to hear that you were able to reproduce the results.

Best, Lukas