PASCAL VOC result?

[duynn912]

Hi @jshtok,

Would you mind show us your way to have the test result for PASCAL VOC. In your instruction and create the benchmark, we found that you only provide your way to test on 214 classes of ImageNetLOC by the end-to-end model. Because there are images from ImageNetLOC in voc_inloc_roidb.pkl, not have any images from PASCAL VOC. Do I need to create a same file for PASCAL? If you have one, could you upload it on your github which you use for your paper?

[jshtok]

Hi @duynn912 , I have added a folder 'cache' under 'data' in the Google Drive link, putting the relevant files there. Please try running the training, you should see

num_images 5011 voc_2007_trainval gt roidb loaded from ./data/cache/voc_2007_trainval_gt_roidb.pkl append flipped images to roidb num_images 11540 voc_2012_trainval gt roidb loaded from ./data/cache/voc_2012_trainval_gt_roidb.pkl append flipped images to roidb num_images 544546

[duynn912]

Hi @jshtok,

Thank your for your reply, Actually, your code also creates two files voc_2007_trainval_gt_roidb.pkl and voc_2007_trainval_gt_roidb.pkl. And I have done the training process. What we need here is that we want to evaluate the trained models for PASCAL VOC and reproduce your results of PASCAL VOC on your paper. After training, I have these models like this: But after reading your code, we realized that you only provide a way to evaluate the trained model with ImagnetLOC. Would you mind help us by showing us how to get the results of PASCAL VOC.

[jshtok]

Hi @duynn912 , I didn't read you question well. I have now added the voc_2007_test_gt_roidb.pkl, containing the database for the test data. Is this what you need? Regards, Joseph

[duynn912]

Hi @jshtok, Thank you very much for you quick response.

We actually want to reproduce the result of PASCAL VOC in your paper. My problem is that we do not see anything to do that in your instruction. Therefore, we just wish you to help us to evaluate the model on PASCAL VOC. We are really sorry for bothering you!

Best regards, DuyNN

[jshtok]

Hi DuyNN, In order to run the regular detection test on a specific data, please run the /experiments/fpn_test.py script with the configuration file --cfg=./experiments/cfgs/resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8_test_Pascal.yaml, when the dataset.test_image_set parameter in this yaml is set to '2007_test' or '2012_test'.

[duynn912]

Hi @jshtok, Thank you very much for your reply! This is what I need. You help my life a lot! By the way, I wonder you have re-trained your model again. I can't reproduce the results for ImagnetLOC and PASCAL. You mentioned that "We have observed 55% FG accuracy in end of Epoch 6 and 73% FG accuracy in Epoch 14", but when my training reaches to Epoch 14, it only get 64%. Besides, when I test 1 shot 5 way 500 episodes according to your benchmark , I just get about 25% for each episode

[jshtok]

I am now checking again the training parameters we used for the model our results were obtained with. Possibly some parameters were changed during the experiments. I will keep you updated.

On Thu, 19 Dec 2019, 11:59 duynn912, notifications@github.com wrote:

Hi @jshtok https://github.com/jshtok, Thank you very much for your reply! This is what I need. You help my life a lot! By the way, I wonder you have re-trained your model again. I can't reproduce the results for ImagnetLOC and PASCAL. You mentioned that "We have observed 55% FG accuracy in end of Epoch 6 and 73% FG accuracy in Epoch 14", but when my training reaches to Epoch 14, it only get 64%. Besides, when I test 1 shot 5 way 500 episodes according to your benchmark , I just get about 25% for each episode

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/19?email_source=notifications&email_token=ACOBU6V3OVZLFLXV4A5JXL3QZNAWJA5CNFSM4J3FWB6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJCOXQ#issuecomment-567420766, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6QNOKW6MANFWVUO2ATQZNAWJANCNFSM4J3FWB6A .

[duynn912]

Hi @jshtok,

I was wondering that you have any update for you check of the model? I am really waiting for that, I hope to hear your response soon. Thank you very much!

Best regard, Duynn

[jshtok]

Hi Duynn,

I have trained the model halfway meanwhile. The training stops because of limited server time allocation I have, and I need to restart it. I will restart it today, hopefully it will finish soon. Sorry for the delay, we had holidays in the recent period.

Regards, Joseph

On Mon, Jan 6, 2020 at 2:38 PM duynn912 notifications@github.com wrote:

Hi @jshtok https://github.com/jshtok,

I was wondering that you have any update for you check of the model? I am really waiting for that, I hope to hear your response soon. Thank you very much!

Best regard, Duynn

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/19?email_source=notifications&email_token=ACOBU6QYWAJNPYQ34S7VRS3Q4MQ5HA5CNFSM4J3FWB6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIFKONI#issuecomment-571123509, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6R4NF66NMNDR7N4OILQ4MQ5HANCNFSM4J3FWB6A .

[duynn912]

Hi @jshtok, Thank you for your quick response

I am sorry that you are in holidays of new year. I also hope you to get the same results in your paper. Btw, Happy New Year 2020!

Sincerely, Duynn

[duynn912]

@jshtok,

I am really sorry for bothering you again. I was wondering that you have done your processing of checking your models?

Thank you so much and I am looking forward to hearing from!

Best regards, Duynn

[jshtok]

Hi Duynn ,

The model is still in training, currently at epoch 18 with FG accuracy of 69%. It goes with breaks because I don't always get the GPUs to run and not always catch when the training stops to renew it ( I get the GPUs for allocated time). I suspect the original training was done with batch size = 2 images (=2 GPUs, one image per GPU) and not 4 images. So I started additional training with batch size=2. Also, I train for more than 15 epochs, since the FG accuracy keeps rising. I hope to finish the training and run the tests soon, and I also hope the results will be satisfiable.

Regards, Joseph

On Mon, Jan 20, 2020 at 12:37 PM duynn912 notifications@github.com wrote:

@jshtok https://github.com/jshtok,

I am really sorry for bothering you again. I was wondering that you have done your processing of checking your models?

Thank you so much and I am looking forward to hearing from!

Best regards, Duynn

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/19?email_source=notifications&email_token=ACOBU6QTGW2DSPVAK5JLWSDQ6V5GNA5CNFSM4J3FWB6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJME6PQ#issuecomment-576212798, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6TO7CIPRKUSDBZPVRLQ6V5GNANCNFSM4J3FWB6A .

[duynn912]

Hi @jshtok,

Thank you so much for your quick response!

I really sorry for that! I do not mean to urge you, I actually understand your situation. I also hope you can re-produce a same result in your paper.

Sincerely, Duynn

[jshtok]

Hi, I have reconstructed the .yaml file that has produce the model with reported performance; please see the experiments/cfgs/resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8_orig.yaml

[duynn912]

Hi,

Thank you very much for your validation!

[yang-yk]

Hello, @duynn912 , Can you tell me how to reproduce the VOC experiment. How to change the code and the .yaml file. Thanks very much. Wish to hear from you soon!

Best regards, Yukuan Yang

[duynn912]

Hi @yang-yk,

For me, I only follow the instruction of Readme and run the updated yaml file above ./experiments/cfgs/resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8_orig.yaml

[yang-yk]

@duynn912 Sorry to bother you again! Do you have the '/VOCdevkit/VOCval/ImageSets/Main/partial.txt' file when you do the voc test experiment? Is it created by myself? Thanks!

[duynn912]

Hi @yang-yk,

You're welcome!

Following the author, "in order to run the regular detection test on a specific data, please run the /fpn/test.py script with the configuration file --cfg=./experiments/cfgs/resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8_orig.yaml, when the dataset.test_image_set parameter in this yaml is set to '2007_test' or '2012_test'. For the result of partial, I don't know what it means. And I was waiting for the answer to have the test results on 100 seen classes in table 3 and for ImageNetLOC on table 4.