About the results you reported in the paper

[molybdenumyz]

Hi,

I downloaded your code and initial weight, and try to reproduce the performance what you reported in the paper. I use the hyperparameters you right in Readme

Here is the result, on the val-set your work performs pretty good but on the test set it works badly.

so I want to know whether there are some tricks not show in the code?

Or show me a good hyperparameters

[Han-Jia]

Hi,

By using the same command, we can get the results reported in the paper:

For MiniImageNet, we use the raw images as the input, and they are center cropped in the data loader.

A simple way to debug is to evaluate the ProtoNet with the pre-trained weights. It gets almost 55% using the euclidean distance and around 62% using the cosine similarity. If lower results are obtained, then the given pre-trained weights are not suitable.

[York-Cheung]

Hi Han Jia!

I ran the command python train_fsl.py --max_epoch 200 --model_class ProtoNet --backbone_class Res12 --dataset MiniImageNet --way 5 --eval_way 5 --shot 1 --eval_shot 1 --query 15 --eval_query 15 --balance 0.01 --temperature 64 --temperature2 64 --lr 0.0002 --lr_mul 10 --lr_scheduler step --step_size 40 --gamma 0.5 --gpu 1 --init_weights ./saves/initialization/miniimagenet/Res12-pre.pth --eval_interval 1 --use_euclidean for reproducing the ProtoNet result you reported in your paper, and got a 64.00 test accuracy. Is it normal? Since it's much higher than 62.39 you reported.

Thanks

[Han-Jia]

Hello,

I think it is normal. ProtoNet is a very strong baseline.

The final performance will vary w.r.t. platforms. Please also ensure that the results are evaluated over 10,000 trials.

[Tsingularity]

Hi,

By using the same command, we can get the results reported in the paper:

For MiniImageNet, we use the raw images as the input, and they are center cropped in the data loader.

A simple way to debug is to evaluate the ProtoNet with the pre-trained weights. It gets almost 55% using the euclidean distance and around 62% using the cosine similarity. If lower results are obtained, then the given pre-trained weights are not suitable.

just wanna double-check, is it a typo here? i think in the paper and readme, 62% acc is got via euclidean distance instead of cosine similarity? please correct me if i am wrong. thanks!

[Han-Jia]

Hi,

The 55% and 62% are the results based on the pre-trained weights directly with distance or cosine similarity, similar to the results in the SimpleShot.

The 62-63% accuracy in the accuracy in the paper is the result of ProtoNet meta-trained with Euclidean distance.