We should ensure all primary components in the code-base are cleaned up to not be overly complicated / hacky, and to enhance ease of contribution and maintenance, while minimizing code-dupe.
[ ] Running new benchmarks / configs (AutoGluon-Benchmark, AutoMLBenchmark)
[x] Simplify generation of BenchmarkContexts for new benchmarks.
[x] Standardize S3 file location + naming
[ ] Automate retries on failed runs
[ ] Repeated cross-validation support for reducing simulation overfitting
[ ] Run configs in nested bagging mode to eliminate validation overfitting
[ ] Add extra model config results: KNN, NN, XGB, FT-Transformer, TabPFN, Linear, VowpalWabbit, etc.
[x] Expand search space for key models (CAT, FASTAI)
[ ] Augment datasets for more information (subsample rows for example)
[ ] Collect more high quality datasets (Ex: Kaggle, MachineHack)
[x] Downloading input files from benchmarks for simulation & loading them (#15)
We should ensure all primary components in the code-base are cleaned up to not be overly complicated / hacky, and to enhance ease of contribution and maintenance, while minimizing code-dupe.