search

weihua916 / powerful-gnns

How Powerful are Graph Neural Networks?
MIT License
1.18k stars 223 forks source link

Hyperparameters to replicate reported performance #2

Closed Tiiiger closed 5 years ago

Tiiiger commented 5 years ago

Thank you for the great work!

I would like to ask the correct hyperparameters for each of the datasets in order to replicate the paper reported result. Thank you!

gear commented 5 years ago

Hi @Tiiiger, I would like to ask if you have been able to find the parameter settings for all the datasets? I ran MUTAG and found the default setting can (kind of) recreate the result. Given there is early stopping and whatnot, the mean accuracy can be higher and match the paper.

batch_size = 128
num_layers = 2
lr = 0.01
num_mlp_layers = 1
hidden_dim = 64
# --epochs can be way smaller than the default 350 (~90)

mean acc = 0.8775 std acc = 0.0422

Tiiiger commented 5 years ago

I try to tune some hyperparameters but possibly due to high variance, the result averaged over 10 folds is usually worse than the paper reported number. Also, using MLP and 1-layer do not seem to make a difference in my experiments.

gear commented 5 years ago

I see. I am searching for COLLAB hyperparameters now. So far varying the number graph layers around 5 doesn't help. Mean acc = ~60% +- 2%. If I take only the highest test accuracy over all training process (assuming early stopping.....), mean acc = ~70%. There is 10% difference from the paper so please let me know if you found the setting that works for COLLAB.

weihua916 commented 5 years ago

Hi, thanks for your interest. There are number of hyper-parameters to be tuned. Please refer to our paper for the detailed procedure. For COLLAB, did you set --degree_as_tag?

weihua916 commented 5 years ago

Besides, the variance is indeed rather high due to the small data size. Still, all the results should be able to be reproduced by the exact procedure described in our paper.

gear commented 5 years ago

@weihua916 Thanks! I indeed did not set --degree_as_tag for COLLAB since I switched from MUTAG and forgot to use the degree as feature vector. Btw, is the default 350 epochs necessary to achieve the results in the paper?

weihua916 commented 5 years ago

Great! I do not think 350 epochs are necessary for most of the datasets. :)

gear commented 5 years ago

Cool! Thanks for making the code available btw :).