search

jshtok / RepMet

Few-shot detection for visual categories
Apache License 2.0
108 stars 18 forks source link

Training on my own dataset and questions about fpn_pascal_imagenet-0015.params #31

Closed lx709 closed 3 years ago

lx709 commented 3 years ago

Dear Joseph, thanks for sharing the code of your interesting work. I'm trying to get the results on my own dataset. After preparing the dataset as your format, I got the following error: Box #0 - no overlapping detection boxes found Box #1 - no overlapping detection boxes found Box #2 - no overlapping detection boxes found .... I guess it might be because my dataset (aerial images) is so different from the pascal voc dataset and I need to retrain the network. I checked the file you provided and found a file named 'fpn_pascal_imagenet-0015.params'. Is it the parameters of the pre-trained FPN-DCN network? In my case, if I understand correctly, I need to retrain the FPN-DCN model to get the new parameter file like this one. Please help me to confirm this. Also, would you please provide the code for generating these parameters?

Thanks for your attention.

lx709 commented 3 years ago

Sorry, I found the answer in the README for the training file I need.

jshtok commented 3 years ago

Hi, Indeed the error you are getting is caused by the fact the RPN does not produce any proposals that match (well) the GT boxes. Did you get these at test time, when you try to provide the detector a few-shot episode? Do you have an annotated dataset for training? The code for the regular training is present in the repo (if I remember correctly); if you indeed need to train, I will open the code tomorrow and recall the exact command and arguments.

On Tue, Aug 25, 2020 at 12:12 PM lx709 notifications@github.com wrote:

Dear Joseph, thanks for sharing the code of your interesting work. I'm trying to get the results on my own dataset. After preparing the dataset as your format, I got the following error: Box #0 - no overlapping detection boxes found Box #1 https://github.com/jshtok/RepMet/issues/1 - no overlapping detection boxes found Box #2 https://github.com/jshtok/RepMet/issues/2 - no overlapping detection boxes found .... I guess it might be because my dataset (aerial images) is so different from the pascal voc dataset and I need to retrain the network. I checked the file you provided and found a file named 'fpn_pascal_imagenet-0015.params'. Is it the parameters of the pre-trained FPN-DCN network? In my case, if I understand correctly, I need to retrain the FPN-DCN model to get the new parameter file like this one. Please help me to confirm this. Also, would you please provide the code for generating these parameters?

Thanks for your attention.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/31, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6W3OI2VEVXXYQNWPLDSCN6A7ANCNFSM4QKNMBGQ .

lx709 commented 3 years ago

Yes, I got this error at test time. Right now, I have retrained the model with ./experiments/fpn_end2end_train_test.py. The training process looks good. 2020-08-26 13:27:05,688 Epoch[19] Train-RPNLogLoss=0.001687 2020-08-26 13:27:05,688 Epoch[19] Train-RPNL1Loss=0.005034 2020-08-26 13:27:05,688 Epoch[19] Train-Proposal FG Fraction=0.150667 2020-08-26 13:27:05,688 Epoch[19] Train-R-CNN FG Accuracy=0.861024 2020-08-26 13:27:05,688 Epoch[19] Train-RCNNAcc=0.959115 2020-08-26 13:27:05,688 Epoch[19] Train-RCNNLogLoss=0.098014 2020-08-26 13:27:05,688 Epoch[19] Train-RCNNL1Loss=0.045871 2020-08-26 13:27:05,689 Epoch[19] Train-RepresentativesStats=0.000000 2020-08-26 13:27:05,689 Epoch[19] Train-EmbedLoss=0.003563 2020-08-26 13:27:05,689 Epoch[19] Train-RCNNLinLogLoss=0.099466 2020-08-26 13:27:05,689 Epoch[19] Time cost=263.546.

And the test performance without fine-tuning also looks good. (btw., it works when I changed the init learning rate to 1e-3, rather than 1e-2 during training) 2020-08-26 13:27:05,688 Epoch[19] Train-RPNAcc=0.999561 2020-08-26 14:29:52,866 - my_logger - INFO - RepMet_inloc_nwpu epi 0 5-shot 3-way: #Dets: 26029, #GT: 216 TP: 187 FP: 25842 = 301 wrong + 25541 bkgnd Recall: 0.866 AP: 0.370 2020-08-26 14:32:10,210 - my_logger - INFO - RepMet_inloc_nwpu epi 1 5-shot 3-way: #Dets: 52339, #GT: 404 TP: 351 FP: 51988 = 548 wrong + 51440 bkgnd Recall: 0.869 AP: 0.367 2020-08-26 14:33:49,085 - my_logger - INFO - RepMet_inloc_nwpu epi 2 5-shot 3-way: #Dets: 78771, #GT: 605 TP: 519 FP: 78252 = 760 wrong + 77492 bkgnd Recall: 0.858 AP: 0.357 2020-08-26 14:35:42,822 - my_logger - INFO - RepMet_inloc_nwpu epi 3 5-shot 3-way: #Dets: 104388, #GT: 761 TP: 640 FP: 103748 = 924 wrong + 102824 bkgnd Recall: 0.841 AP: 0.280 2020-08-26 14:37:29,365 - my_logger - INFO - RepMet_inloc_nwpu epi 4 5-shot 3-way: #Dets: 131405, #GT: 940 TP: 790 FP: 130615 = 1130 wrong + 129485 bkgnd Recall: 0.840 AP: 0.285 2020-08-26 14:39:11,728 - my_logger - INFO - RepMet_inloc_nwpu epi 5 5-shot 3-way: #Dets: 157411, #GT: 1125 TP: 939 FP: 156472 = 1348 wrong + 155124 bkgnd Recall: 0.835 AP: 0.226 2020-08-26 14:41:04,093 - my_logger - INFO - RepMet_inloc_nwpu epi 6 5-shot 3-way: #Dets: 183375, #GT: 1334 TP: 1102 FP: 182273 = 1600 wrong + 180673 bkgnd Recall: 0.826 AP: 0.206 2020-08-26 14:42:57,583 - my_logger - INFO - RepMet_inloc_nwpu epi 7 5-shot 3-way: #Dets: 208142, #GT: 1498 TP: 1238 FP: 206904 = 1801 wrong + 205103 bkgnd Recall: 0.826 AP: 0.216 2020-08-26 14:44:35,068 - my_logger - INFO - RepMet_inloc_nwpu epi 8 5-shot 3-way: #Dets: 233460, #GT: 1659 TP: 1373 FP: 232087 = 2005 wrong + 230082 bkgnd Recall: 0.828 AP: 0.203 2020-08-26 14:46:33,811 - my_logger - INFO - RepMet_inloc_nwpu epi 9 5-shot 3-way: #Dets: 260707, #GT: 1855 TP: 1539 FP: 259168 = 2261 wrong + 256907 bkgnd Recall: 0.830 AP: 0.205

But, the problem is when I add "--do_finetune=1 --num_finetune_epochs=5", the test performance gets terrible, 2020-08-27 05:28:19,706 - my_logger - INFO - RepMet_inloc_nwpu epi 2 5-shot 3-way: #Dets: 75738, #GT: 605 TP: 397 FP: 75341 = 794 wrong + 74547 bkgnd Recall: 0.656 AP: 0.011 2020-08-27 05:28:19,706 RepMet_inloc_nwpu epi 2 5-shot 3-way: #Dets: 75738, #GT: 605 TP: 397 FP: 75341 = 794 wrong + 74547 bkgnd Recall: 0.656 AP: 0.011 2020-08-27 05:28:19,884 - my_logger - INFO - AP change due to finetuning: 0.004-->0.011. This looks very strange here.

Would you please give some advice on how to train and test on my own dataset. For example, my dataset has 10 foreground classes, 3 classes (e.g., 8,9,10) are used as the novel classes for model evaluation, the others are used for meta-training. Following is my setting for meta-training.

dataset: NUM_CLASSES: 8 #122 balance_classes: False #true num_ex_per_class: 200 max_num_extra_classes: 8 #122 dataset: NWPU dataset_path: './data/NWPU/imgs' #"/dccstor/leonidka1/data/VOCdevkit;/dccstor/leonidka1/data/imagenet/ILSVRC" cls_filter_files: './data/NWPU/inloc_cls2id_map.pkl:./data/NWPU/inloc_first101_categories.txt' #7 categories image_set: nwpu #2007_trainval+2012_trainval;train_loc per_category_epoch_max: 8 #what's this parameter used for? root_path: "./data" test_image_set: nwpu_val #;val_partial proposal: rpn

And settings in the 'few_shot_benchmark.py' file:

    cfg_fname = root+'/experiments/cfgs/nwpu_resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8.yaml'
    test_classes_fname =root+'/data/NWPU/in_domain_categories.txt'  # 3 categories
    roidb_fname = root+'/data/NWPU/nwpu_inloc_roidb.pkl'

Are these configurations correct? What else do I need to change? Thanks for your attention.

jshtok commented 3 years ago

Hi,

Your parameters look fine; just look at the reduction of loss and the performance on base classes to see if the detector trained well on the base categories. For finetuning, please try to change learning rate and number of iterations; a sanity check is to increase the episode size so that there will be more training data.

Regards, Joseph

On Thu, Aug 27, 2020 at 4:58 AM lx709 notifications@github.com wrote:

Yes, I got this error at test time. Right now, I have retrained the model with ./experiments/fpn_end2end_train_test.py. The training process looks good. 2020-08-26 13:27:05,688 Epoch[19] Train-RPNLogLoss=0.001687 2020-08-26 13:27:05,688 Epoch[19] Train-RPNL1Loss=0.005034 2020-08-26 13:27:05,688 Epoch[19] Train-Proposal FG Fraction=0.150667 2020-08-26 13:27:05,688 Epoch[19] Train-R-CNN FG Accuracy=0.861024 2020-08-26 13:27:05,688 Epoch[19] Train-RCNNAcc=0.959115 2020-08-26 13:27:05,688 Epoch[19] Train-RCNNLogLoss=0.098014 2020-08-26 13:27:05,688 Epoch[19] Train-RCNNL1Loss=0.045871 2020-08-26 13:27:05,689 Epoch[19] Train-RepresentativesStats=0.000000 2020-08-26 13:27:05,689 Epoch[19] Train-EmbedLoss=0.003563 2020-08-26 13:27:05,689 Epoch[19] Train-RCNNLinLogLoss=0.099466 2020-08-26 13:27:05,689 Epoch[19] Time cost=263.546.

And the test performance without fine-tuning also looks good. (btw., it works when I changed the init learning rate to 1e-3, rather than 1e-2 during training) 2020-08-26 13:27:05,688 Epoch[19] Train-RPNAcc=0.999561 2020-08-26 14:29:52,866 - my_logger - INFO - RepMet_inloc_nwpu epi 0 5-shot 3-way: #Dets: 26029, #GT: 216 TP: 187 FP: 25842 = 301 wrong + 25541 bkgnd Recall: 0.866 AP: 0.370 2020-08-26 14:32:10,210 - my_logger - INFO - RepMet_inloc_nwpu epi 1 5-shot 3-way: #Dets: 52339, #GT: 404 TP: 351 FP: 51988 = 548 wrong + 51440 bkgnd Recall: 0.869 AP: 0.367 2020-08-26 14:33:49,085 - my_logger - INFO - RepMet_inloc_nwpu epi 2 5-shot 3-way: #Dets: 78771, #GT: 605 TP: 519 FP: 78252 = 760 wrong + 77492 bkgnd Recall: 0.858 AP: 0.357 2020-08-26 14:35:42,822 - my_logger - INFO - RepMet_inloc_nwpu epi 3 5-shot 3-way: #Dets: 104388, #GT: 761 TP: 640 FP: 103748 = 924 wrong + 102824 bkgnd Recall: 0.841 AP: 0.280 2020-08-26 14:37:29,365 - my_logger - INFO - RepMet_inloc_nwpu epi 4 5-shot 3-way: #Dets: 131405, #GT: 940 TP: 790 FP: 130615 = 1130 wrong + 129485 bkgnd Recall: 0.840 AP: 0.285 2020-08-26 14:39:11,728 - my_logger - INFO - RepMet_inloc_nwpu epi 5 5-shot 3-way: #Dets: 157411, #GT: 1125 TP: 939 FP: 156472 = 1348 wrong + 155124 bkgnd Recall: 0.835 AP: 0.226 2020-08-26 14:41:04,093 - my_logger - INFO - RepMet_inloc_nwpu epi 6 5-shot 3-way: #Dets: 183375, #GT: 1334 TP: 1102 FP: 182273 = 1600 wrong + 180673 bkgnd Recall: 0.826 AP: 0.206 2020-08-26 14:42:57,583 - my_logger - INFO - RepMet_inloc_nwpu epi 7 5-shot 3-way: #Dets: 208142, #GT: 1498 TP: 1238 FP: 206904 = 1801 wrong + 205103 bkgnd Recall: 0.826 AP: 0.216 2020-08-26 14:44:35,068 - my_logger - INFO - RepMet_inloc_nwpu epi 8 5-shot 3-way: #Dets: 233460, #GT: 1659 TP: 1373 FP: 232087 = 2005 wrong + 230082 bkgnd Recall: 0.828 AP: 0.203 2020-08-26 14:46:33,811 - my_logger - INFO - RepMet_inloc_nwpu epi 9 5-shot 3-way: #Dets: 260707, #GT: 1855 TP: 1539 FP: 259168 = 2261 wrong + 256907 bkgnd Recall: 0.830 AP: 0.205

But, the problem is when I add "--do_finetune=1 --num_finetune_epochs=5", the test performance gets terrible, 2020-08-27 05:28:19,706 - my_logger - INFO - RepMet_inloc_nwpu epi 2 5-shot 3-way: #Dets: 75738, #GT: 605 TP: 397 FP: 75341 = 794 wrong + 74547 bkgnd Recall: 0.656 AP: 0.011 2020-08-27 05:28:19,706 RepMet_inloc_nwpu epi 2 5-shot 3-way: #Dets: 75738, #GT: 605 TP: 397 FP: 75341 = 794 wrong + 74547 bkgnd Recall: 0.656 AP: 0.011 2020-08-27 05:28:19,884 - my_logger - INFO - AP change due to finetuning: 0.004-->0.011. This looks very strange here. Would you please give some advice on how to train and test on my own dataset. For example, my dataset has 10 foreground classes, 3 classes (e.g., 8,9,10) are used as the novel classes for model evaluation, the others are used for meta-training. Following is my setting for meta-training. dataset: NUM_CLASSES: 8 #122 balance_classes: False #true num_ex_per_class: 200 max_num_extra_classes: 8 #122 dataset: NWPU dataset_path: './data/NWPU/imgs'

"/dccstor/leonidka1/data/VOCdevkit;/dccstor/leonidka1/data/imagenet/ILSVRC"

cls_filter_files: './data/NWPU/inloc_cls2id_map.pkl:./data/NWPU/inloc_first101_categories.txt'

7 https://github.com/jshtok/RepMet/issues/7 categories

image_set: nwpu #2007_trainval+2012_trainval;train_loc per_category_epoch_max: 8 #what's this parameter used for? root_path: "./data" test_image_set: nwpu_val #;val_partial proposal: rpn And settings in the 'few_shot_benchmark.py' file:

cfg_fname = root+'/experiments/cfgs/nwpu_resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8.yaml'
test_classes_fname =root+'/data/NWPU/in_domain_categories.txt'  # 3 categories
roidb_fname = root+'/data/NWPU/nwpu_inloc_roidb.pkl'

Are these configurations correct? What else do I need to change? Thanks for your attention.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jshtok/RepMet/issues/31#issuecomment-681284412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBU6X2TS5LILJHT2MZ7S3SCW4SVANCNFSM4QKNMBGQ .

lx709 commented 3 years ago

Dear Joseph, thanks for your attention. The performance on base categories is satisfying. According to your suggestions, on my own dataset, I changed the learning rate from 1e-4 to 5e-4 during fine-tuning stage and get better performance.