search

snap-stanford / ogb

Benchmark datasets, data loaders, and evaluators for graph machine learning
https://ogb.stanford.edu
MIT License
1.89k stars 398 forks source link

Can‘t achieve reported SIGN performance MAG240M #156

Closed 763337092 closed 3 years ago

763337092 commented 3 years ago

I ran part of the code locally, but can't achieve as high score as written in the README.md. My local SIGN get 0.65 valid accuracy and SGC only get 0.62. Run the same code but don’t know why

rusty1s commented 3 years ago

I'm really sorry. I think the committed hyperparameters are slightly wrong, i.e. try using the model with --layer=2.

Edit: the default --layer=3 is correct.

763337092 commented 3 years ago

I'm really sorry. I think the committed hyperparameters are slightly wrong, i.e. try using the model with --layer=2.

Thanks for your reply! How about the hyperparameters about gnn.py and rgnn.py? The default hyperparameters can't reach convergence.

weihua916 commented 3 years ago

Those hyper-parameters should be correct. Could you paste your tensorboard log + any print output + the command you run? It could be also helpful to give us your package version.

rusty1s commented 3 years ago

I also re-ran the sgc.py script (with --layer=3) and obtained the same results as reported.

weihua916 commented 3 years ago

Same as Matthias: I ran python preprocess_sgc.py; python sign.py --device=0 and it perfectly reproduces the reported SIGN result.

torch==1.7.0, torch_geometric==1.6.3

weihua916 commented 3 years ago

For GNN, the validation curves on the tensorboard will show lower results than reported, since the sampled neighboring node sizes are small during training. You should be able to reproduce the reported validation result once you run the inference code: python gnn.py --device=0 --model=graphsage --evaluate. It uses much larger neighbor sampling sizes.

Closing the issue for now. Let us know if you still cannot reproduce.