This is the code to reproduce the experiments from the Paper: Constructing CIs for 'the' Generalization Error.
The content of this repository is:
./analysis/
contains the code to process the results and to reproduce all the figures../data/
contains some input data and is the folder where generated datasets are stored../datamodels/
contains the code related to the generation of the datasets../experiments
contains the code for the main experiments to investigate the inference methods../figures
is the folder where the figures are stored.
Only the figures from the main paper are stored here, some additional l./inferGE/
is the R package that implements confidence interval methods that are being compared.
This is research code.
If you want to use the well-performing inference methods in R use this repository: https://github.com/mlr-org/mlr3inferr../renv/
and renv.lock
are for the reproducible R environment./results/
contains the final results (such as figures, tables) included in the paperThe instructions to reproduce the experiments are separated into:
./datamodels/README.md
. Note that the resulting datasets are also made available on
OpenML, so the main experiments can be reproduced without this step.
Note: The code and instructions to reproduce the results are still being cleaned up../experiments/README.md
.
Note: The code and instructions to reproduce the results are still being cleaned up../results
subdirectory of this repository.Because of the large size of the benchmark datasets, it is important to download them in parquet format. However, the download might still fail. In this case, simply retry until it works.
In order to evaluate new inference methods, the following steps need to be followed:
mlr3
, you need to implement a new Resampling
class, e.g. by adding it to the inferGE
R package in the folder with the same name.
For an example, e.g. see ResamplingNestedCV
.inferGE
packages.
As an example see the infer_bates.R
file which uses the resample result ccreated by ResamplingNestedCV
../experiments/resample
../experiments/ci.R
Don't hesitate to contact us if you want to reuse this code!
If you don't want to work with R but still work with the results via e.g. python, you can achieve this by:
R
interpreter.rds
file using readRDS(<path>)
write.csv
function.