search

megvii-research / FSCE

Apache License 2.0
274 stars 47 forks source link

Confusion about the VOC result #26

Open tymanman opened 3 years ago

tymanman commented 3 years ago

Hi, In the Table 2 of the paper, you compare the average ap of mutiple runs of 10 seeds with the results given by TFA which is a average one of 30 seeds.However it is obvious that the former is anyway better than the latter according to the graph shown in TFA, So maybe this seems to be a little unresonable? Look forward to your reply.

linyanhly commented 3 years ago

The same question, I think this comparision may not be reasonable

zshanggu commented 2 years ago

Hi, In the Table 2 of the paper, you compare the average ap of mutiple runs of 10 seeds with the results given by TFA which is a average one of 30 seeds.However it is obvious that the former is anyway better than the latter according to the graph shown in TFA, So maybe this seems to be a little unresonable? Look forward to your reply.

Your observation is right, it should be 30 seeds here. I'd rather believe this is a typo in the paper, but I didn't get same result as in Table2 yet.

Ze-Yang commented 1 year ago

Hi, In the Table 2 of the paper, you compare the average ap of mutiple runs of 10 seeds with the results given by TFA which is a average one of 30 seeds.However it is obvious that the former is anyway better than the latter according to the graph shown in TFA, So maybe this seems to be a little unresonable? Look forward to your reply.

Your observation is right, it should be 30 seeds here. I'd rather believe this is a typo in the paper, but I didn't get same result as in Table2 yet.

I notice that the author makes a tricky test with augmentation TEST.AUG.ENABLED=True while other competing methods did not do so. After removing this, I obtained the results as shown below, which are somewhat lower than the reported numbers. How about your reproduced numbers?

split1 split2 split3
1shot 2shot 3shot 5shot 10shot 1shot 2shot 3shot 5shot 10shot 1shot 2shot 3shot 5shot 10shot
AP50 69.10 68.33 68.11 70.42 72.93 65.03 65.75 67.33 70.14 71.03 69.02 70.18 68.29 72.97 72.66
bAP50 78.44 77.13 74.29 74.67 76.74 79.47 78.06 75.48 78.52 77.85 79.58 78.36 75.66 78.86 77.42
nAP50 41.08 41.92 49.57 57.68 61.51 21.69 28.80 42.87 45.00 50.58 37.31 45.65 46.19 55.31 58.36