Hi, good to see you here! 👋
This is code for "Active Testing: Sample-Efficient Model Evaluation".
Please cite our paper, if you find this helpful:
@article{kossen2021active,
title={{A}ctive {T}esting: {S}ample-{E}fficient {M}odel {E}valuation},
author={Kossen, Jannik and Farquhar, Sebastian and Gal, Yarin and Rainforth, Tom},
journal={arXiv:2103.05331},
year={2021}
}
The requirements.txt
can be used to set up a python environment for this codebase.
You can do this, for example, with conda
:
conda create -n isactive python=3.8
conda activate isactive
pip install -r requirements.txt
To reproduce a figure of the paper, first run the appropriate experiments
sh reproduce/experiments/figure-X.sh
And then create the plots with the Jupyter Notebook at
notebooks/plots_paper.ipynb
(The notebook let's you conveniently select which plots to recreate.)
Which should put plots into notebooks/plots/
.
In the above, replace X
by
123
for Figures 1, 2, 34
for Figure 45
for Figure 56
for Figure 67
for Figure 7Other notes
conf/paper
to modify this property)main.py
is the main entry point into this code-base.
n_runs
active testing experiments for a fixed setup.This repository uses Hydra
to manage configs.
conf/config.yaml
or one of the experiments in conf/...
for default configs and hyperparameters../output/
.See notebooks/eplore_experiment.ipynb
for some example code on how to evaluate custom experiments.
activetesting.visualize.Visualiser
which implements visualisation methods.path
to an experiment in output/path/to/experiment
and explore the methods.A guide to the code
main.py
runs repeated experiments and orchestrates the whole shebang.
n_runs
and acquisition strategies
.experiment.py
handles a single experiment.
model
, dataset
, acquisition strategy
, and risk estimators
.datasets.py
, aquisition.py
, loss.py
, risk_estimators.py
all contain exactly what you would expect!hoover.py
is a logging module.models/
contains all models, scikit-learn and pyTorch.
sk2torch.py
we have some code that wraps torch models in a way that lets them be used as scikit-learn models from the outside.Thanks for stopping by!
If you find anything wrong with the code, please contact us.
We are happy to answer any questions related to the code and project.