search

clovaai / ECLIPSE

(CVPR 2024) ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning
Other
32 stars 4 forks source link

Unable to reproduce panoptic segmentation results on ADE20k #12

Open Ze-Yang opened 2 weeks ago

Ze-Yang commented 2 weeks ago

Hi, I tried to reproduce the panoptic segmentation results on ADE20k. However, I found 1-2 percent points gap between the reproduced results and the reported one, especially on 100-x series. Moreover, the reproduced results on novel classes are generally lower than the reported one, while the base class results are comparable. Note that I did not do any modifications to the codes. I ran the experiments on two GPUs. Could you kindly advise?

100-x series: 100-5 100-10 100-50
1-100 101-150 all 1-100 101-150 all 1-100 101-150 all
Reported 41.1 16.6 32.9 41.4 18.8 33.9 41.7 23.5 35.6
Reproduced 39.5 14.7 31.2 41.2 16.5 33.0 39.9 21.1 33.6
50-x series: 50-10 50-20 50-50
1-50 51-150 all 1-50 51-150 all 1-50 51-150 all
Reported 45.9 17.3 26.8 46.4 19.6 28.6 46.0 20.7 29.2
Reproduced 46.4 16.8 26.7 47.1 17.7 27.5 46.0 18.6 27.7
qjadud1994 commented 1 week ago

Hi. I used 4 V100 GPUs for most experiments.

And, there were marginal performance variations of about 1~2%. I recommend tuning the delta values for every experiment to produce the best performance.