Open PGijsbers opened 1 year ago
After an offline discussion with @PGijsbers we agreed that this should be an optional feature, i.e. that caching is disabled by default, but can be enabled.
There are definitely cases where this is useful (experimenting with evaluation metrics or ensembling), but the average user that probably doesn't load the same runs many times. Because it would quickly occupy a lot of disk space, we think opt-in is better.
Problem
Predictions of runs are not cached when downloaded. Note that predictions only get downloaded when
get_metric_fn
is called in the first place (this is desired behavior, the description file already contains precomputed evaluations).MWE
CLI:
ls ~/.openml/org/openml/www/runs/10591753/
Output:ls: /Users/pietergijsbers/.openml/org/openml/www/runs/10591753/: No such file or directory
Execute:
output:
CLI:
ls ~/.openml/org/openml/www/runs/10591753/
Output:description.xml
Note that there are no signs of the prediction arff file being present on disk - as you would expect from reading the source code.