This repo contains a PyTorch implementation of meta-dataset and a unified implementation of some few-shot methods.
Some benefits over original code:
Note that this code also includes the original implementation for comparison (using the PyTorch workaround proposed by the authors). If you wish to use the original implementation, set the option loader_version: 'tf'
in base.yaml
(by default set to pytorch
).
Some cons over original code:
Yet to do:
Please carefully follow the instructions below to get started.
The present code was developped and tested in Python 3.8. The list of requirements is provided in requirements.txt:
pip install -r requirements.txt
To download the META-DATASET, please follow the details instructions provided at meta-dataset to obtain the .tfrecords
converted data. Once done, make sure all converted dataset are in a single folder, and execute the following to produce index files:
make index_files
This may take a few minutes.
export RECORDS='path/to/records'
We provide trained Resnet-18 and WRN-2810 models on the training split of ILSVRC_2012 at checkpoints. All non-episodic baselines use the same checkpoint, stored in the standard
folder. The results (averaged over 600 episodes) obtained with an ImageNet pre-trained Resnet-18 are summarized below. Please take note that results below are not intended to be used as official numbers (they may not follow optimal hyper-parameters), but only to make sure you are able to ensure reproducibility with default options:
Method | aircraft | traffic_sign | quickdraw | dtd | omniglot | fungi | cu_birds | mscoco |
---|---|---|---|---|---|---|---|---|
BDCSPN | 48.4 | 50.93 | 56.03 | 76.79 | 54.97 | 45.43 | 73.96 | 50.75 |
Finetune | 56.12 | 53.92 | 58.89 | 79.67 | 59.41 | 39.7 | 73.67 | 44.85 |
SimpleShot | 49.52 | 51.92 | 56.32 | 76.62 | 55.12 | 44.95 | 73.49 | 50.05 |
TIM_GD | 55.74 | 58.72 | 61.09 | 80.58 | 62.25 | 49.75 | 76.72 | 55.54 |
See Sect. 1.4 and 1.5 to reproduce these results.
In order to train you model from scratch, execute:
make method=<method> arch=<architecture> base=<base_dataset> val=<validation_dataset> train
method
is to be chosen among all method specific config files in config/method, architecture
in /src/models/standard and dataset
among all datasets (as named by the META-DATASET converted folders). Note that the hierarchy of arguments passed to src/train.py
and src/eval.py
is the following: base_config < method_config < opts arguments.
Mutiprocessing : This code supports distributed training. To leverage this feature, set the gpus
option accordingly (for instance gpus: [0, 1, 2, 3]
).
Once trained (or once pre-trained models downloaded), you can evaluate your model on the test split of each dataset by running:
make method=<method> arch=<architecture> base=<base_dataset> test=<test_dataset> eval
Results will be saved in results/<method>/<exp_no>
where
During training, training loss and validation accuracy are recorded and saved as .npy files in the checkpoint folder. Then, you can use the src/plot.py to plot these metrics (even during training).
Example 1: Plot the metrics of the standard (=non episodic) resnet-18 on ImageNet:
python -m src.plot --folder checkpoints/ilsvrc_2012/ilsvrc_2012/resnet18/standard/
Example 2: Plot the metrics of all Resnet-18 trained on ImageNet
python -m src.plot --folder checkpoints/ilsvrc_2012/ilsvrc_2012/resnet18/
For methods that perform test-time optimization (for instance MAML, TIM, Finetune, ...), method specific metrics are plotted in real-time (versus test iterations) and averaged over test epidodes, which can allow you to track unexpected behavior easily. Such metrics are implemented in src/metrics/
, and the choice of which metric to plot is specificied through the eval_metrics
option in the method .yaml config file. An example with TIM method is provided below.
By setting the option visu: True
at inference, you can visualize samples of episodes. An example of such visualization is given below:
The samples will be saved in results/.
All relevant optons can be found in the base.yaml
file, in the EVAL-VISU section.
This code was designed to allow easy incorporation of new methods.
Step 1: Add your method .py file to src/methods/
by following the template provided in src/methods/method.py.
Step 2: Add import in src/methods/__init__.py
Step 3: Add your method .yaml config file including the required options episodic_training
and method
(name of the class corresponding to your method). Also make sure that if your method performs test-time optimization, you also properly set the option iter
that specifies the number of optimization steps performed at inference (this argument is also used to plot the inference metrics, see section 2.2).
Contributions are more than welcome. In particular, if you want to add methods/pre-trained models, do make a PR.
If you find this repo useful for your research, please consider citing the following papers:
@article{triantafillou2019meta,
title={Meta-dataset: A dataset of datasets for learning to learn from few examples},
author={Triantafillou, Eleni and Zhu, Tyler and Dumoulin, Vincent and Lamblin, Pascal and Evci, Utku and Xu, Kelvin and Goroshin, Ross and Gelada, Carles and Swersky, Kevin and Manzagol, Pierre-Antoine and others},
journal=ICLR},
year={2020}
}
@misc{boudiaf2021mutualinformation,
title={Mutual-Information Based Few-Shot Classification},
author={Malik Boudiaf and Ziko Imtiaz Masud and Jérôme Rony and Jose Dolz and Ismail Ben Ayed and Pablo Piantanida},
year={2021},
eprint={2106.12252},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Additionally, do not hesitate to file issues if you encounter problems, or reach out directly to Malik Boudiaf (malik.boudiaf.1@etsmtl.net).
I thank the authors of meta-dataset for releasing their code and the author of open-source TFRecord reader for open sourcing an awesome Pytorch-compatible TFRecordReader ! Also big thanks to @hkervadec for his thorough code review !