Some benchmarks have train and test data, like ARC, or data.jl and data_large.jl like DreamCoder. This information is lost once the module is loaded. It would be nice when one could handle these sets separately, e.g. train only on training problems and evaluate later on the test set.
some benchmarks have large files, like DreamCoder/Lists_tasks/list_tasks2.json (not an actual path to dir), that are over 100mb. We might need to find a smart way splitting those up, OR add github Large file system.
Some benchmarks have train and test data, like ARC, or
data.jl
anddata_large.jl
like DreamCoder. This information is lost once the module is loaded. It would be nice when one could handle these sets separately, e.g. train only on training problems and evaluate later on the test set.