search

jshtok / RepMet

Few-shot detection for visual categories
Apache License 2.0
108 stars 18 forks source link

About COCO pretrain #4

Closed liyangliu closed 4 years ago

liyangliu commented 4 years ago

Hi, @jshtok, @leokarlin

I noticed that in CVPR paper you mentioned the pretrained model used for few shot detection fine-tuning on ImageNet is pretrained on COCO. I am curious about how you can train a DCN-FPN detection model from scratch without GN or SyncBN. Could you please be so kind as to upload your log on training DCN-FPN model with COCO from scratch? Thanks.

jshtok commented 4 years ago

Dear Liyang,

I am not sure we have the log, I will look for it. What makes you sceptical about this training - does your training not converge well?

Thanks, Joseph

liyangliu commented 4 years ago

Hi, @jshtok,

My model does not converge very well, I used your updated fpn_dcn_coco pretrain model for training on ImageNet 101 classes, but can not reproduce the results reported in your paper for 5 shot 5 way 10 qpc 500 episodes. I can only achieve around 42% mAP.

liyangliu commented 4 years ago

And I tested your uploaded trained model, the mAP for 5 shot 5 way 10qpc 500 eps (without finetuning for episodes) is 66.3%, and that for 10 shot (other setting are the same) is only 64.3%, I don't understand why it is not the same with the results in the paper, especially the results for 10 shot is even lower than that of 5 shot.

liyangliu commented 4 years ago

Hi, @jshtok,

I have trained the model with your code, and get quiet good training accuracy,

2019-10-11 06:17:29,617 Epoch[6] Train-RPNAcc=0.999683 2019-10-11 06:17:29,618 Epoch[6] Train-RPNLogLoss=0.001037 2019-10-11 06:17:29,619 Epoch[6] Train-RPNL1Loss=0.003038 2019-10-11 06:17:29,620 Epoch[6] Train-Proposal FG Fraction=0.036605 2019-10-11 06:17:29,620 Epoch[6] Train-R-CNN FG Accuracy=0.541494 2019-10-11 06:17:29,621 Epoch[6] Train-RCNNAcc=0.979532 2019-10-11 06:17:29,621 Epoch[6] Train-RCNNLogLoss=0.075383 2019-10-11 06:17:29,621 Epoch[6] Train-RCNNL1Loss=0.012629 2019-10-11 06:17:29,621 Epoch[6] Train-RepresentativesStats=0.000000 2019-10-11 06:17:29,621 Epoch[6] Train-EmbedLoss=0.419763 2019-10-11 06:17:29,621 Epoch[6] Train-RCNNLinLogLoss=0.068349

2019-10-11 17:51:17,930 Epoch[12] Train-RPNAcc=0.999844 2019-10-11 17:51:17,931 Epoch[12] Train-RPNLogLoss=0.000545 2019-10-11 17:51:17,931 Epoch[12] Train-RPNL1Loss=0.001710 2019-10-11 17:51:17,932 Epoch[12] Train-Proposal FG Fraction=0.037280 2019-10-11 17:51:17,933 Epoch[12] Train-R-CNN FG Accuracy=0.750570 2019-10-11 17:51:17,933 Epoch[12] Train-RCNNAcc=0.987679 2019-10-11 17:51:17,933 Epoch[12] Train-RCNNLogLoss=0.041537 2019-10-11 17:51:17,933 Epoch[12] Train-RCNNL1Loss=0.007369 2019-10-11 17:51:17,933 Epoch[12] Train-RepresentativesStats=0.000000 2019-10-11 17:51:17,933 Epoch[12] Train-EmbedLoss=0.298569 2019-10-11 17:51:17,933 Epoch[12] Train-RCNNLinLogLoss=0.036943

But when I use this trained model for testing, the mAP is just so low, just around 30%. Do you have any ideas why it is the case? Thanks.

jshtok commented 4 years ago

Hi, I have reconstructed the .yaml file that has produce the model with reported performance; please see the experiments/cfgs/resnet_v1_101_voc0712_trainval_fpn_dcn_oneshot_end2end_ohem_8_orig.yaml