Accompanying repository for the paper Why do tree-based models still outperform deep learning on tabular data?
Create a new environment using python 3.8, then install the requirements:
pip install -r requirements.txt
You can re-run the training using WandB sweeps.
src/configs/wandb_config.py
src
python launch_benchmarks/launch_benchmarks.py
. This will create a csv with the wandb sweep to run, and save
it in src/launch_benchmarks/sweeps/all_benchmarks_medium.csv
.wandb agent <USERNAME/PROJECTNAME/SWEEPID>
in src
. More infos
in the WandB doc.launch_benchmarks/launch_on_cluster.py --filename NAME_OF_THE_CSV_FILE --output_filename FILENAME --n_runs NUMBER_OF_PARALLEL_RUNS_PER_SWEEP --max_runs MAX_NUMBER_OF_RUN_PER_DATASET --monitor
.
You'll need to adapt the
script to your cluster (see the TODO comments in the script). This will automatically launch the sweeps on the cluster
and download the results when they are done.All the R code used to generate the analyses and figures in available in the analyses
folder.
To dowload the random search results used for the figures, run:
curl -L -o analyses/results/benchmark_total.csv https://figshare.com/ndownloader/files/40081681
The datasets used in the benchmark have been uploaded as OpenML benchmarks, with the same transformations that are used in the paper.
import openml
#openml.config.apikey = 'FILL_IN_OPENML_API_KEY' # set the OpenML Api Key
SUITE_ID = 336 # Regression on numerical features
#SUITE_ID = 337 # Classification on numerical features
#SUITE_ID = 335 # Regression on numerical and categorical features
#SUITE_ID = 334 # Classification on numerical and categorical features
benchmark_suite = openml.study.get_suite(SUITE_ID) # obtain the benchmark suite
for task_id in benchmark_suite.tasks: # iterate over all tasks
task = openml.tasks.get_task(task_id) # download the OpenML task
dataset = task.get_dataset()
X, y, categorical_indicator, attribute_names = dataset.get_data(
dataset_format="dataframe", target=dataset.default_target_attribute
)
You can also find these datasets on Hugging Face Hub.
If you want to compare you own algorithms with the models used in this benchmark for a given number of random search iteration, you can use the results from our random searches, by running:
curl -L -o analyses/results/benchmark_total.csv https://figshare.com/ndownloader/files/40081681
To benchmark your own algorithm using our code, you'll need:
src/configs/model_configs/template.py
.