search

JosephKJ / OWOD

(CVPR 2021 Oral) Open World Object Detection
https://josephkj.in
Apache License 2.0
1.04k stars 155 forks source link

Confusion in evaluation. #60

Closed Hrren closed 3 years ago

Hrren commented 3 years ago

Thanks for the nice paper，I have learnt a lot.

But at percent , I still cannot reproduce the result , I use the checkpoint form \owod_backup\models_backup\t1_clustering_with_save\model_final.pth or \owod_backup\models_backup\t2_ft\model_final.pth which from the https://drive.google.com/drive/folders/1Sr4_q0_m2f2SefoebB25Ix3N1VIAua0w?usp=sharing and https://drive.google.com/drive/folders/11bJRdZqdtzIxBDkxrx2Jc3AhirqkO0YV?usp=sharing . When I run the test command python tools/train_net.py --num-gpus 1 --eval-only --config-file ./configs/OWOD/t1/t1_test.yaml SOLVER.IMS_PER_BATCH 1 SOLVER.BASE_LR 0.005 OUTPUT_DIR "./output/t1_final" MODEL.WEIGHTS "/home/appuser/OWOD/output/t1/model_final.pth" or python tools/train_net.py --num-gpus 1 --eval-only --config-file ./configs/OWOD/t2/t2_test.yaml SOLVER.IMS_PER_BATCH 1 SOLVER.BASE_LR 0.005 OUTPUT_DIR "./output/t2_final" MODEL.WEIGHTS "/home/appuser/OWOD/output/t1/model_final_t2.pth" the results are like this: Snipaste_2021-08-13_18-35-11 unknown AP 50 ; unknown Precision50 and Unknown Recall50 are all under 5% Why dose it happen , and where is it wrong ?

JosephKJ commented 3 years ago

Hi @Hrren : You can not fitting the Weibull distribution, nor using it while evaluation. Please follow replicate.sh.

Reopen if issue persists. Thanks!

Hrren commented 3 years ago

Thank you for your reply @JosephKJ Here are two questions I still in trouble： When I finish task2 training, is the model_final.pth in t2_final works better than that in t2_ft? I use the model_final.pth in these two folders , and the test result is different, why this happen? and WI in paper refers to the "Wilderness Impact:{0.8:{50:xxxxxx}}" or others?