search

TUM-DAML / pprgo_pytorch

PPRGo model in PyTorch, as proposed in "Scaling Graph Neural Networks with Approximate PageRank" (KDD 2020)
https://www.daml.in.tum.de/pprgo
Other
127 stars 17 forks source link

Reproducing the paper's result #5

Closed markchen02 closed 2 years ago

markchen02 commented 3 years ago

Hi, I could not reproduce Table 2. Acc: 73.7(39) for PubMed dataset. I got only 67.0%. (See below)

Accuracy: Train: 100.0%, val: 69.5%, test: 67.0% F1 score: Train: 1.000, val: 0.692, test: 0.667 Runtime: Preprocessing: 0.15s, training: 4.20s, inference: 0.05s -> total: 4.40s Memory: Main: 3.87GB, GPU: 0.032GB

Could you let me know how to reproduce 73.7% in Table 2?

gasteigerjo commented 3 years ago

Did you change the teleport probability alpha to 0.25, as described in Appendix A.3?

markchen02 commented 3 years ago

Thank you so much. I missed that part. After I increase alpha to 0.25, the test accuracy did increase. However, the best test accuracy was 70.7% with multiple runs. Could you check if you are able to reproduce 73.7%, and if this is the case, how to reproduce 73.7%? Thank you in advance

gasteigerjo commented 2 years ago

The results in the paper are definitely accurate. I'm sorry, but I can't help you with further debugging your setup.

Note that the paper's results used the TensorFlow version of the code. Also, keep in mind that your result falls within the standard deviation we've reported. A more standard way of writing 73.7(39) would be 73.7 +- 3.9.