search

nv-tlabs / GSCNN

Gated-Shape CNN for Semantic Segmentation (ICCV 2019)
https://nv-tlabs.github.io/GSCNN/
Other
915 stars 200 forks source link

Unfair Comparison in your paper #2

Closed LarryBrid closed 4 years ago

LarryBrid commented 4 years ago

After reading your paper and discussion, we have few questions.

  1. In the leaderboard, we can see that you use Mapillary data. However, In the paper of table 6. You compared TKCN and AAF-PSP which didn't use Mapillay data. It is obviously unfair. So what is real result without Maypiilary data and only use fine-data on Cityscapes test set ?
  2. Also we didn't find your claim in using Mapillary data in your paper why didn't mention it?
  3. In the table 3, Why mIoU is so low when use your ResNet101 backbone according other repo. ResNet101 based deeplabv3+ should be 78+. Repo: https://github.com/speedinghzl/pytorch-segmentation-toolbox
tovacinni commented 4 years ago

Thanks for pointing out the missing information on Mapillary dataset in the paper. We will include it in our revision.

Different works has different training setups and it is hard to compare with same settings as previous works. Our ablations (Table 3) show consistent relative improvements with the same training setups that aren't necessarily optimizing for maximum performance.

tovacinni commented 4 years ago

We do mention it on the leaderboard, and we also updated our paper to mention the usage for our camera-ready.

Wangzhuoying0716 commented 4 years ago

@tovacinni Have you tested not using Mapillary dataset and how about the performance? Thanks a lot and will you release the official testing code?

lxtGH commented 4 years ago

@tovacinni Have you tested not using Mapillary dataset and how about the performance? Thanks a lot and will you release the official testing code?