search

jd730 / EOPSN

[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)
Other
52 stars 7 forks source link

Problems of reproducing the results of EOPSN #4

Open HeimingX opened 3 years ago

HeimingX commented 3 years ago

Hi,

Thanks for the interesting paper and open-sourced code.

Recently, I ran the EOPSN method on K20 setting folllowing the given guideline (w/o any editing) and I found the results of unknown things are quite different from the reported one.

Unk | PQ | SQ | RQ -- | -- | -- | -- EOPSN reported | 11.3 | 73.8 | 15.3 EOPSN reproduced | 15.6 | 79.2 | 19.6

From the table, it seems that the released code achieves a much better improvement than the reported one. However, when I further inspect the predictions of class-wise unkown things, it seems that EOPSN's unkown recognition is dominated by the "car" class and other unkown classes are rarely detected. Moreover, the reproduced results may not support the visualization results in Fig5 since several unkown classes are shown to be detected, e.g., stop sign, keyboard, banana, and toilet. So, could you please release the EOPSN checkpoint which supports the reported results? Thanks a lot.

image

BTW, I found that the training of EOPSN requires the pre-trained model of Void-Suppression, but the current released codebase only contains the void-train method. I wonder could you please release the void-suppression code for better reproduction? Thanks again.

FYI, the predictions of class-wise unkown things on Void-Suppression method are as follows and the results are identical to the reported ones in the paper image

jd730 commented 3 years ago

Hi @HeimingX, I apologize for facing the difficulty. However, I am extremely busy these days. I will resolve this as soon as possible.

jd730 commented 3 years ago

Hi @HeimingX, I apologize for the delay. I haven't tested yet and I do not think I have time before the upcoming CVPR deadline... However, it seems that the problem comes from sampling object proposals for clustering or clustering itself. You can relax the condition.

Once, I have time, I will also figure out why the problem was caused while publishing the codebase.

HeimingX commented 2 years ago

Hi, It's been a while since CVPR DDL and I wonder if it is possible for you to release the checkpoint of EOPSN method for further exploration? Many thanks

jd730 commented 2 years ago

Hi @HeimingX, Thank you for reminding me. I just uploaded EOPSN_K10.pth, EOPSN_K05.pth, EOPSN_K20.pth. I did not run the inference again now, so with the little possibility, I could upload the wrong files. If so, please let me know.

HeimingX commented 2 years ago

Hi @HeimingX, Thank you for reminding me. I just uploaded EOPSN_K10.pth, EOPSN_K05.pth, EOPSN_K20.pth. I did not run the inference again now, so with the little possibility, I could upload the wrong files. If so, please let me know.

Hi, thanks for the releasing of EOPSN checkpoints. The EOPSN_K05.pth, EOPSN_K20.pth can produce indentical results as reported in the paper. But EOPSN_K10.pth raises a bug during the model loading and the error log is as follows: image

According to the log, it seems that the released EOPSN_K10.pth does not use the default cluster number(i.e., 128).

jd730 commented 2 years ago

Thank you for letting me know. It seems that the checkpoint is for K20 (weight difference is 8 which is 10% of COCO think class). I will do my best to find the right checkpoint.

HeimingX commented 2 years ago

Hi, many thanks for your kind and prompt response!

Recently, I am conducting some analysis to one of the baseline methods (void-train) and I use this repo to reproduce the void-train method and obtain a slightly worse result:

image

So I'd like to know if it is possible for you to release the checkpoint of void-train baseline for K=20% setting, many many thanks~