search

sud0301 / semisup-semseg

136 stars 30 forks source link

The results are not consisstent #12

Closed chouqin3 closed 3 years ago

chouqin3 commented 3 years ago

I run the command --dataset pascal_voc --checkpoint-dir ./checkpoints/voc_semi_0_125 --labeled-ratio 0.125 --ignore-label 255 --num-classes 21 and achieve 64.11%, that paper says 65.4% image --dataset pascal_voc --checkpoint-dir ./checkpoints/voc_semi_0_02 --labeled-ratio 0.02 --ignore-label 255 --num-classes 21 and achieve 53.59%, that paper says 58.1%, what's the problems? image

sud0301 commented 3 years ago

Hi, Thank you for your interest in the work. Please verify the labeled-ratio used in the command-line arguments. According to your current arguments of labeled-ratio=2%, the expected performance is 58.1% (Table1 without COCO pre-training ) as mentioned in the paper.

chouqin3 commented 3 years ago

I've updated the questions, thanks for your attention, could you explain it?

sud0301 commented 3 years ago
  1. One of the possible reason for the discrepancy in the results could be due to the usage of different splits. Try to run using different splits and average. The reported performance is averaged over 3 random splits.
  2. You should evaluate the validation performance at every 5k iterations. The optimal performance for smaller splits (for eg. with 5% labels) might occur earlier than 35k iterations.