search

arthurdouillard / CVPR2021_PLOP

Official code of CVPR 2021's PLOP: Learning without Forgetting for Continual Semantic Segmentation
https://arxiv.org/abs/2011.11390
MIT License
145 stars 23 forks source link

can't reproduce Ade20k and don't know the detail of Cityscapes. #11

Closed schuy1er closed 3 years ago

schuy1er commented 3 years ago

Hi, sorry to disturb but i have a few questions as followed. 1) Maybe you have changed some details about the experiments over Cityscapes, since it seems like MiB cannot work on the Cityscapes because there are only GT without pseudo labels. I'd appreciate it if you can tell me what the exact details you have changed over these experiments. 2) I cannot reproduce the performance that you mentioned in your paper about the Ade20k, having seen that you had reproduced the performance of your method over Cityscapes in another issue, I hope that you can reproduce it over Ade20k too and have a further discussion with me. Wish you good luck.

arthurdouillard commented 3 years ago

Hello,

  1. I'm not sure to exactly understand your problem, MiB doesn't use pseudo-labels. But yes indeed, you cannot use UNCE / UNKD (as mentioned in the paper). Therefore MiB is just a classic KD model. I've pushed a script for MiB here: https://github.com/arthurdouillard/CVPR2021_PLOP/blob/main/scripts/cityscapesDomain/mib_cityscapesDomain_1-1.sh

  2. It would help me if you gave me more informations. Have you try the script I provide? Like https://github.com/arthurdouillard/CVPR2021_PLOP/blob/main/scripts/ade/plop_ade_100-5.sh

schuy1er commented 3 years ago

Hello,

  1. Thanks for your sharing! What I really mean is that i wonder if you change any hyper-parameters or something else of the original code of MiB. Nevertheless, I will try this script later and i will contact you if i have meet other problems.
  2. Actually I didn't reproduce 100-5, but the performance on 100-50 is quite aweful, it's only about 18.96 on Ade20k, maybe you can share your script about the 100-50 with me?
arthurdouillard commented 3 years ago
  1. Here is the script: https://github.com/arthurdouillard/CVPR2021_PLOP/blob/main/scripts/ade/plop_ade_100-50.sh . Training ADE is super slow, so you can re-use the initial checkpoint (first step, classes 0-100) that I've uploaded there https://github.com/arthurdouillard/CVPR2021_PLOP/releases/tag/v1.01 . There is instructions commented in the bash script to use weights for the initial task.
schuy1er commented 3 years ago

it seems that this script is for 100-10... not for 100-50 -:)

schuy1er commented 3 years ago

the score of the 100-50 is about 33.26 and here is my log: INFO:rank0: Closing the Logger. Last Step: 1 Final Mean IoU 33.26 Average Mean IoU 37.75 Mean IoU first 42.06 Mean IoU last 15.49 ade_100-50_PLOP On GPUs 0,1,2,3 Run in 78599s

arthurdouillard commented 3 years ago

Oops, you're right, I mistaken 100-50 with 100-10, I'll fix it.

Great, So you got (old / new / all / avg):

While the paper results were:

So I guess, PLOP on ADE20k was correctly reproduced, with even better results than the paper. Happy to hear :)

Therefore I think I've answered your questions, and will close this issue. If you have still have problem, comment here and I'll re-open it. Good luck in your research!