Open silverbottlep opened 5 years ago
Thanks for your comments.
Thanks for your prompt reply.
I also ran other pytorch implementation, and none of them (I tried 2 different codes) approached same level of accuracy as reported in original MAML paper (~48%). Yeap, I understand your pain, and but in many cases where meta-learning is usefule, ensemble might not makse sense, e.g. RL applications, sample efficiency is the one of the main benefits of meta-learning. Anyway, I think it'd be better to mention this in the paper.
I could get ~0.47 with 32 filters with ensemble with 3 different models. But, it sereiously overfits to training sets.
Thanks. I also don't think it would make big difference in terms of final accuracy, either.
yeap, what I meant by was for new task. And, also it seems model-ensemble in RL is not as popular as supervised classification setup.
BTW, do you remember the accuracy without ensemble for 5-shot mini-imagenet task?
Yeah, RL model ensemble doesn't work as well in my experience.
Unfortunately, I don't remember then un-ensembled accuracies for the mini-imagenet 5-shot variants.
Thanks for your quick response, I do really appreciate it. It would be much appreciated if you could share the results of non-ensemble of both MAML++ and MAML. Anyway, your work would be very helpful for many people, thanks!
Ok, I'll run the experiments without any ensembles and let you know.
In the orignal MAML paper I think they use 64 filters not 32 for classification tasks
They used 64 for Omniglot and 32 for Mini Imagenet.
Yeah, thanks for the correction, I ran this experiment "mini-imagenet_1_2_0.01_48_5_0" and ia m getting a test accuracy of 50.30, is this correct because in the MAML paper they get only around 48.50. Thanks a lot for this implementation, by far the cleanest implementation of MAML in pytorch
Hi, thanks for sharing this code! It would be really helpful!
I ran 'mini-imagenet_maml_mini-imagenet_5_way_1_shot.json'. It runs 50,000 iterations, each iteration has 4 tasks (e.g. meta-level batch is 4). It spit out test accuracy ~0.48 at the end, which is close to the one reported in the paper. But, I had some questions.
It seems it does model ensemble of top-3 models, which was selected based on validation accuracy. When I only used top-1 model I could only get ~0.46 accuracy. And, Top validation accuracy was ~0.46, so I think boosting up to ~0.48 might be thanks to the ensemble. But is it valid experiment setup?
cnn_num_filters was set to 48, but I believe original MAML used 32 filters. I ran the experiment with 32 filters, and it gave me ~2% less accuracy.
'first_order_to_second_order_epoch' was set to 10, which enables first-order optimization for the first 10 epoch. you mentioned it increased training stability. But, I believe original MAML didn't used this.
It would be great if you could resolve this issue! Thanks!