Boosted Graph Neural Networks

The code and data for the ICLR 2021 paper: Boost then Convolve: Gradient Boosting Meets Graph Neural Networks

This code contains implementation of the following models for graphs:

• CatBoost
• LightGBM
• Fully-Connected Neural Network (FCNN)
• GNN (GAT, GCN, AGNN, APPNP)
• FCNN-GNN (GAT, GCN, AGNN, APPNP)
• ResGNN (CatBoost + {GAT, GCN, AGNN, APPNP})
• BGNN (end-to-end {CatBoost + {GAT, GCN, AGNN, APPNP}})

Installation

To run the models you have to download the repo, install the requirements, and extract the datasets.

First, let's create a python environment:

mkdir envs
cd envs
python -m venv bgnn_env
source bgnn_env/bin/activate
cd ..

Second, let's download the code and install requirements

git clone https://github.com/nd7141/bgnn.git 
cd bgnn
unzip datasets.zip
make install

Next we need to install a proper version of PyTorch and DGL, depending on the cuda version of your machine. We strongly encourage to use GPU-supported versions of DGL (the speed up in training can be 100x).

First, determine your cuda version with nvcc --version. Then, check installation instructions for pytorch. For example for cuda version 9.2, install it as follows:

pip install torch==1.7.1+cu92 torchvision==0.8.2+cu92 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

If you don't have GPU, use the following:

pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Similarly, you need to install DGL library. For example, cuda==9.2:

pip install dgl-cu92

For cpu version of DGL:

pip install dgl

Tested versions of torch and dgl are:

• torch==1.7.1+cu92
• dgl_cu92==0.5.3

Running

Starting point is file scripts/run.py:

python scripts/run.py dataset models 
    (optional) 
            --save_folder: str = None
            --task: str = 'regression',
            --repeat_exp: int = 1,
            --max_seeds: int = 5,
            --dataset_dir: str = None,
            --config_dir: str = None

Available options for dataset:

• house (regression)
• county (regression)
• vk (regression)
• wiki (regression)
• avazu (regression)
• vk_class (classification)
• house_class (classification)
• dblp (classification)
• slap (classification)
• path/to/your/dataset

Available options for models are catboost, lightgbm, gnn, resgnn, bgnn, all.

Each model is specifed by its config. Check configs/ folder to specify parameters of the model and run.

Upon completion, the results wil be saved in the specifed folder (default: results/{dataset}/day_month/). This folder will contain aggregated_results.json, which will contain aggregated results for each model. Each model will have 4 numbers in this order: mean metric (RMSE or accuracy), std metric, mean runtime, std runtime. File seed_results.json will have results for each experiment and each seed. Additional folders will contain loss values during training.

Examples

The following script will launch all models on House dataset.

python scripts/run.py house all

The following script will launch CatBoost and GNN models on SLAP classification dataset.

python scripts/run.py slap catboost gnn --task classification

The following script will launch LightGBM model for 5 splits of data, repeating each experiment for 3 times.

python scripts/run.py vk lightgbm --repeat_exp 3 --max_seeds 5

The following script will launch resgnn and bgnn models saving results to custom folder.

python scripts/run.py county resgnn bgnn --save_folder ./county_resgnn_bgnn

Running on your dataset

To run the code on your dataset, it's necessary to prepare the files in the right format.

You can check examples in datasets/ folder.

There should be at least X.csv (node features), y.csv (target labels), graph.graphml (graph in graphml format).

Make sure to keep these filenames for your dataset.

You can also have cat_features.txt specifying names of categorical columns.

You can also have masks.json specifying train/val/test splits.

After that run the script as usual:

python scripts/run.py path/to/your/dataset gnn catboost 

Citation

@inproceedings{
ivanov2021boost,
title={Boost then Convolve: Gradient Boosting Meets Graph Neural Networks},
author={Sergei Ivanov and Liudmila Prokhorenkova},
booktitle={International Conference on Learning Representations (ICLR)},
year={2021},
url={https://openreview.net/forum?id=ebS5NUfoMKL}
}