Cannot reproduce the results for 1-shot 5-way Mini-ImageNet

[xwjabc]

I used the command below to perform the experiment (environment: Python 3.6 / TensorFlow 1.10):

# 1-shot 5-way Mini-ImageNet.
python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 15 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 5 --checkpoint ckpt_m15t

The last several lines of output:

batch 99950: train=0.200000 test=0.200000         
batch 99960: train=0.000000 test=0.000000       
batch 99970: train=0.200000 test=0.000000         
batch 99980: train=0.200000 test=0.200000        
batch 99990: train=0.200000 test=0.400000         
Evaluating...                                       
Train accuracy: 0.28016                            
Validation accuracy: 0.26512                    
Test accuracy: 0.24262

TensorBoard output:

However, in the paper, the test accuracy is around 47%.

[unixpickle]

Is there a chance the data you downloaded is incomplete or corrupted somehow? There should be 600 images per class, and there should be 100 classes in total. If you find that the data is in fact invalid (perhaps the download script is broken?), I can send you the data directly.

On Fri, Oct 26, 2018 at 12:51 PM Weijian Xu notifications@github.com wrote:

I used the command below to perform the experiment (environment: Python 3.6 / TensorFlow 1.10):

1-shot 5-way Mini-ImageNet.

python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 15 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 5 --checkpoint ckpt_m15t

The last several lines of output:

batch 99950: train=0.200000 test=0.200000 batch 99960: train=0.000000 test=0.000000 batch 99970: train=0.200000 test=0.000000 batch 99980: train=0.200000 test=0.200000 batch 99990: train=0.200000 test=0.400000 Evaluating... Train accuracy: 0.28016 Validation accuracy: 0.26512 Test accuracy: 0.24262

TensorBoard output: [image: 1-shot 5-way reptile] https://user-images.githubusercontent.com/16413829/47580731-7a509c80-d904-11e8-9424-dd755f26241d.png

However, in the paper, the test accuracy is around 47%.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openai/supervised-reptile/issues/16, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYyBTtPkZn8J8Zf7f_sCD1Q3tGi3Lggks5uoz2TgaJpZM4X8mce .

[xwjabc]

It seems the dataset is indeed incomplete (see the output below). However, it does not lose too many images (38392/38400, 9593/9600, 11996/12000). Thus, I wonder if there are some other reasons behind the performance.

/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./train -name '*.JPEG' | wc -l
38392
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./val -name '*.JPEG' | wc -l
9593
/m/c/w/c/F/R/supervised-reptile/data/miniimagenet
find ./test -name '*.JPEG' | wc -l
11996

By the way, could you send me a copy of the data? Thank you very much!

[unixpickle]

See if 7f815bca856f96a04dc5519b3ad87d99b86d3104 fixes your problem. The command in the README didn't quite match the hyperparameters in the paper. The correct command is:

python -u run_miniimagenet.py --shots 1 --inner-batch 10 --inner-iters 8 --meta-step 1 --meta-batch 5 --meta-iters 100000 --eval-batch 5 --eval-iters 50 --learning-rate 0.001 --meta-step-final 0 --train-shots 15 --checkpoint ckpt_m15t

[xwjabc]

Thank you very much! Will try the new command with complete data.

[xwjabc]

@unixpickle By the way, 7f815bc changes --eval-batch 15 --train-shots 5 to --eval-batch 5 --train-shots 15. I could understand the change of eval-batch, which behaves the same as reducing the learning rate (to 1/3 of original one) in the inner loop during the evaluation stage. However, I wonder why train-shots should be increased. In fact, it seems train-shots=1 matches the best when considering the similarity between the training and the evaluation stage.

[unixpickle]

@xwjabc for whatever reason, we found that training on more "shots" helped Reptile's performance, probably because it allows you to take more diverse gradient steps during each inner-loop. Table 4 of Appendix A in the paper specifies the hyper-parameters, this included.

[xwjabc]

@unixpickle Got it. Thank you for the clarification!

[lampardwk]

@unixpickle @xwjabc I had the same problem with the incomplete miniimagenet data downloaded from fetch_data.sh,most of folder is empty images. Could you send me a complete data？My email address is lampardwk518@163.com ,thanks.

[unixpickle]

I used to have the dataset on my OpenAI machine and in the cloud. Unfortunately, I no longer have access to either copy. I'll see if I can find it sitting anywhere else, but I doubt I can.

[lampardwk]

What a pity! Thanks for your reply