search

mseg-dataset / mseg-semantic

An Official Repo of CVPR '20 "MSeg: A Composite Dataset for Multi-Domain Segmentation"
MIT License
460 stars 78 forks source link

Performance different from reported in paper #43

Open XAVILLA opened 2 years ago

XAVILLA commented 2 years ago

Hi, I tried to run the training code with the 1m config but got significantly worse performance compared to the performance reported in the paper. I notice that you mentioned in the paper that batch size 35 was used but in the 1m config you set batch size to be 14. Can you please explain what should I change to train a model that achieves the performance as you reported with 1m config? Thanks!

johnwlambert commented 2 years ago

Hi @XAVILLA, thanks for your interest in our work. I'll double check on this and get back to you.

XAVILLA commented 2 years ago

Hi John. Thanks for the help! Any update so far?

johnwlambert commented 2 years ago

Hi @XAVILLA, which specific model are you training, on which dataset, at which resolution, and using which config?

Could you share the results you are seeing?

XAVILLA commented 2 years ago

Hi @johnwlambert, I used the HRNet+1m+1080 config given in the repo. link. In this config the batch size is set to 14

I run evaluation on the trained model using 360 ss config and obtained mIOU \~58.5, and 1080 ss config and obtained mIOU \~74, both on cityscapes validation set. The mIOU for 1m model was reported in the repo to be ~80 on cityscapes.

Can you take a look at this issue and let me know what should I fix to obtain the performance reported in the paper? Thanks!

XAVILLA commented 2 years ago

Hi @johnwlambert, any update on this issue? My main question is that in the paper you describe that you used a mini-batch size m = 35, but in the training config here the batch size is set to 35. Can you please clarify which batch size was used in your experiments to produce the reported performance? Thanks!