search

jshtok / RepMet

Few-shot detection for visual categories
Apache License 2.0
108 stars 18 forks source link

Is there any difference between 5-way 1-shot and 5-way 5-shot in the code? #22

Open yang-yk opened 4 years ago

yang-yk commented 4 years ago

Hi, Shtok, sorry to bother you again.

When I run the code of 5-way 1-shot, the experiment results are close to what the paper reports. However, when I run the code with the command 'python fpn/few_shot_benchmark.py --test_name=RepMet_inloc --Nshot=5 --Nway=5 --Nquery_cat=10 --Nepisodes=500' (5-way 5-shot), the AP is much smaller (48.3%) than what the paper reports (68.8%). I wonder why it happens. Can you help me to solve this problem. Thanks very much!

Sincerely, Yukuan Yang

yang-yk commented 4 years ago

@duynn912 Do you have any idea about my questions above. Thanks!

duynn912 commented 4 years ago

Hi @yang-yk,

I also got a result like you for 5 way 1 shot. Actually, I did not experiment 5 way 5 shot so I did not know exactly how many accuracy is. However, according to the paper, when we experiment 5 way 1 shot, it means we have one image from each class, totally 5 images for 5 classes. And you have 5 way 5 shot, you will have 5 images from each class of 5 classes, totally 25 images for 5 classes. At this time, you have more images for your experiment, I think it will have a higher accuracy in comparison to 5 way 1 shot and it will be similar to the paper. So, I was also wondering why it is?

jshtok commented 4 years ago

Hi Yukuyan,

I need to run the test to reconstruct the problem you are describing, and I will get back to you.

Regards, Joseph

On Sun, Jan 19, 2020 at 2:01 PM Yukuan Yang notifications@github.com wrote:

Hi, Shtok, sorry to bother you again.

When I run the code of 5-way 1-shot, the experiment results are close to what the paper reports. However, when I run the code with the command 'python fpn/few_shot_benchmark.py --test_name=RepMet_inloc --Nshot=5 --Nway=5 --Nquery_cat=10 --Nepisodes=500' (5-way 5-shot), the AP is much smaller (48.3%) than what the paper reports (68.8%). I wonder why it happens. Can you help me to solve this problem. Thanks very much!

Sincerely, Yukuan Yang

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/22?email_source=notifications&email_token=ACOBU6VZGT2TWMD5V4VVO4TQ6Q6JLA5CNFSM4KIY2XK2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IHFQPBQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6QVFCWSF4HDSWAOUVLQ6Q6JLANCNFSM4KIY2XKQ .

jshtok commented 4 years ago

Hi @yang-yk , I just executed the line python fpn/few_shot_benchmark.py --test_name=RepMet_inloc --Nshot=5 --Nway=5 --Nquery_cat=10 --Nepisodes=500

and I am getting good results (see .log file below). Sorry I couldn't reconstruct your error.

Regards, Joseph

few_shot_benchmark_RepMet_inloc_5shot_5way_10qpc_500epi_aug_0%3A10_lr_1.0e-04_nms%3A0.30_iou%3A0.70_clust%3A1_exact%3A1_2020-01-27-15-05.log

yang-yk commented 4 years ago

Hi @jshtok , Thanks for your reply. I think the reason why I have met the problem lies in the episode file generated by the code after comparing your log file. My log file and episode file is as follows. I think my episode file (.npz) is different from yours. I wonder whether this is caused by the random seed setting or the NumPy version (1.16.5).

To be clear, I use the original code from Github and the original model parameters (downloaded from the google drive). Can you give me your 1-shot 5-shot and 10-shot episode file (.npz file in the output directory).

Thanks very much and hope to hear from you soon!

Best regards, Yukuan Yang

5-shot result.zip

jshtok commented 4 years ago

Hi Yukuan, Sorry for the late reply. Please see the file attached:

On Sat, Feb 8, 2020 at 5:15 PM Yukuan Yang notifications@github.com wrote:

Hi @jshtok https://github.com/jshtok , Thanks for your reply. I think the reason why I have met the problem lies in the episode file generated by the code after comparing your log file. My log file and episode file is as follows. I think my episode file (.npz) is different from yours. I wonder whether this is caused by the random seed setting or the numpy version (1.16.5).

To be clear, I use the original code from github and the original model parameters (downloaded from the google drive). Can you give me your 5-shot and 10-shot episode file (.npz file in the output directory).

Thanks very much and hoping hearing from you soon!

Best regards, Yukuan Yang

Uploading few_shot_benchmark_RepMet_inloc_5shot_5way_10qpc_500epi_aug_0_10_lr_1.0e-04_nms_0.30_iou_0.70_clust_1_exact_1_2020-01-19-01-24.txt…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/22?email_source=notifications&email_token=ACOBU6TRIKCO3AKQ7AS74M3RB3ECLA5CNFSM4KIY2XK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFUAIA#issuecomment-583745568, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6XC2CCUMUPD5WPWST3RB3ECLANCNFSM4KIY2XKQ .