[Resolved] Discrepancy in t*_ft.txt

JosephKJ / OWOD

(CVPR 2021 Oral) Open World Object Detection

https://josephkj.in

Apache License 2.0

1.04k stars 155 forks source link

[Resolved] Discrepancy in t*_ft.txt #37

Closed PeizeSun closed 3 years ago

PeizeSun commented 3 years ago

Thanks for your great work !

However, I run the released code but the performance is behind the reported performance in the paper. The first row is the reported performance, the second is my performance. I didn't change any code.

Meanwhile, I find the data split in your logs is different from the released data split in the google drive. For task 2 finetune, the first image is data split in your logs, the second is that in the google drive. They are apparently different.

Looking forwards your reply.

JosephKJ commented 3 years ago

Dear @PeizeSun : I cannot thank you enough for noticing this discrepancy. Apparently, t2_ft.txt was a version that I had used to run an ablation. I have updated the same now. Can you please try now? Unfortunately, I dont have the exact same file with me (the machine on which I ran these is now decommissioned, and I had missed to backup it.), but the current files would suffice.

Please do try it out and let me know.

Thanks, Joseph

LoveIsAGame commented 3 years ago

@JosephKJ i cannot find t1_ft.txt. but in run.sh ,t1_ft is nessary

JosephKJ commented 3 years ago

There is no fine-tuning step required for Task 1. There is nothing like forgetting there. We are learning all the VOC classes in Task 1, and learning to identify unknowns. Please let me know why you think that t1_ft is necessary.

JosephKJ commented 3 years ago

Ah, you might be referring to the old version of run.sh, @LoveIsAGame. Kindly pull the latest tip.

LoveIsAGame commented 3 years ago

Yes, I didn't notice you made a change to run.sh two hours ago. Thanks @JosephKJ

LoveIsAGame commented 3 years ago

@JosephKJ In Task1, I am using 1 GPU, so I set BASE_LR to 0.0025 and here is my result.I don't know what went wrong. The configuration files are executed in the order t1_train t1_val t1_test.

JosephKJ commented 3 years ago

Hi @LoveIsAGame : You need to run it for 8x more iterations as you are running on a single GPU. Please refer to the official Detectron 2 spec. This issue might also be helpful.

dyabel commented 3 years ago

Hi, the new t2_ft.txt is still different from that used in models_backup/t2_ft.The prior contains 1743 images and the latter contains 386. Could you provide the t2_ft.txt which is used to train the model in models_backup? Thx

JosephKJ commented 3 years ago

Can you copy over /datasets/OWOD_imagesets/ to /datasets/VOC2007/ImageSets/Main and try? Please ensure that you are on the latest commit of the code. Thanks!

dyabel commented 3 years ago

Can you copy over /datasets/OWOD_imagesets/ to /datasets/VOC2007/ImageSets/Main and try? Please ensure that you are on the latest commit of the code. Thanks! Yes, I have updated the latest t2_ft.txt which has 1743 images, is that right? but the log.txt of t2_ft in your google drive owod_backup shows there are only 386 images.

JosephKJ commented 3 years ago

There is some confusion here:

1) t2_ft.txt in owod_backup contains 1665 images and not 386 (Please refer to the screenshot in the first comment in this thread by PeizeSun) 2) I believe 386 is what you got after running the previous version of t2_ft.txt 3) If you run the latest t2_ft.txt, you will get 1743 images. Please find my log below:

JosephKJ commented 3 years ago

Kindly reopen if the issue persists.