search

ServiceNow / embedding-propagation

Codebase for Embedding Propagation: Smoother Manifold for Few-Shot Classification. This is a ServiceNow Research project that was started at Element AI.
Apache License 2.0
208 stars 21 forks source link

Can not reproduce the results #12

Open sjiajia opened 3 years ago

sjiajia commented 3 years ago

Hello, I cannot reproduce the results of the paper on the miniImagenet dataset. Is there any solution?（thanks very much） I used the pretrained and finetuned model data（backbone-wrn） you provided in another issue. The test results were as follows：miniImagenet: 1shot 74.12% and 5shot 85.21% tieredImageNet: 1shot 84.22% and 5shot 89.40% The results given in the paper are: miniImagenet: 1shot 79.22% and 5shot 88.05%. tieredImageNet: 1shot 83.69% and 5shot 89.34% The performance on the mini dataset is much worse. I pretrained and finetuned the model myself, and I tested it again, the performance was as follows： miniImagenet（wrn）: 1shot 77.12% and 5shot 87.90%. There is still a certain gap between the performance and the result in paper, and I found that in the 1shot experiment on the miniImageNet data set, the performance did not improve after the 8th epoch. Is there any problem here? Finally, I still have a question. According to the results in the article, the performance of epnet-ssl is 9% higher than that of epnet in the case of 1 shot. Is this increase so much? When I did the experiment, the performance of epnetssl was 4% higher than that of epnet. Can you provide a solution? thank you very much!

prlz77 commented 3 years ago

Hi @sjiajia,

First of all, just clarify that the results you are comparing with (79.22, 88.05, 83.69, 89.34) are for the semi-supervised learning experiment (Table 4 https://arxiv.org/pdf/2003.04151.pdf).

In order to debug your SSL set-up. Could you put a breakpoint in the SSL model (https://github.com/ElementAI/embedding-propagation/blob/master/src/models/ssl_wrapper.py#L28), and check if you are using it? Also, could tell me the value of self.exp_dict['unlabeled_size_test']?

Thanks

sjiajia commented 3 years ago

hello @prlz77 , I am sure that I am runing the semi-supervised learning experiment (Table 4). I use the pretrained and finetuned model that you provided in another issue (episodic_miniimagenet_wrn_1-shot/checkpoint_best.pth and episodic_miniimagenet_wrn_5-shot/checkpoint_best.pth) and get the results(74.12 85.21 84.22 89.40). I pretrain and finetune the model on miniImageNet by myself and get the results (77.12 87.90). The exp_dict of mini-wrn-1shot-ssl experiment is as follows: { "dataset_train_root": "mini-imagenet/", "dataset_val_root": "mini-imagenet/", "dataset_test_root": "mini-imagenet/", "model": { "name": "ssl", "backbone": "wrn", "depth": 28, "width": 10, "transform_train": "wrn_finetune_train", "transform_val": "wrn_val", "transform_test": "wrn_val" }, "ngpu": 1, "random_seed": 42, "batch_size": 1, "train_iters": 10, "test_iters": 600, "tasks_per_batch": 1, "dropout": 0.1, "avgpool": true, "n_classes": 64, "collate_fn": "identity", "transform_train": "wrn_finetune_train", "transform_val": "wrn_val", "transform_test": "wrn_val", "dataset_train": "episodic_miniimagenet_pkl", "classes_train": 5, "support_size_train": 1, "query_size_train": 15, "unlabeled_size_train": 0, "dataset_val": "episodic_miniimagenet_pkl", "classes_val": 5, "support_size_val": 1, "query_size_val": 15, "unlabeled_size_val": 0, "dataset_test": "episodic_miniimagenet_pkl", "classes_test": 5, "support_size_test": 1, "query_size_test": 15, "unlabeled_size_test": 100, "predict_method": "labelprop", "finetuned_weights_root": "./logs/finetuning", "embedding_prop": true } Is there any other solutions? Thank you very much!

prlz77 commented 3 years ago

Thanks for the clarification! At first glance I cannot see what is wrong. It may be related to this commit: https://github.com/ElementAI/embedding-propagation/commit/a841b3ff056f16e87e74bc4c36f1c3d6e7f0d9cc

I will investigate it but I am currently busy with the ICML deadline, so I may find a solution after then. If meanwhile, you find the problem, please feel free to make a pull request!

Yoke-L commented 3 years ago

Hello, I cannot reproduce the results of the paper on the miniImagenet dataset. Is there any solution?（thanks very much） I used the pretrained and finetuned model data（backbone-wrn） you provided in another issue. The test results were as follows：miniImagenet: 1shot 74.12% and 5shot 85.21% tieredImageNet: 1shot 84.22% and 5shot 89.40% The results given in the paper are: miniImagenet: 1shot 79.22% and 5shot 88.05%. tieredImageNet: 1shot 83.69% and 5shot 89.34% The performance on the mini dataset is much worse. I pretrained and finetuned the model myself, and I tested it again, the performance was as follows： miniImagenet（wrn）: 1shot 77.12% and 5shot 87.90%. There is still a certain gap between the performance and the result in paper, and I found that in the 1shot experiment on the miniImageNet data set, the performance did not improve after the 8th epoch. Is there any problem here? Finally, I still have a question. According to the results in the article, the performance of epnet-ssl is 9% higher than that of epnet in the case of 1 shot. Is this increase so much? When I did the experiment, the performance of epnetssl was 4% higher than that of epnet. Can you provide a solution? thank you very much!

Hello, do you mind providing the miniImageNet pickle files for me?Thank you very much.