THUDM / GATNE

Source code and dataset for KDD 2019 paper "Representation Learning for Attributed Multiplex Heterogeneous Network"
MIT License
527 stars 141 forks source link

Cannot reproduce the performance in the paper #102

Closed Wang-Yu-Qing closed 3 years ago

Wang-Yu-Qing commented 3 years ago

I run the code via python src/main.py --input data/amazon --features data/amazon/feature.txt

And I get the performance:

Optimizing
2021-04-19 11:05:57.456952: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
Training
epoch 0:   0%|| 0/140091 [00:00<?, ?it/s]valid auc: 0.6256135767967163
valid pr: 0.6428760893673962
valid f1: 0.5847500584986922

valid auc: 0.625584291466907:00<?, ?it/s]
valid pr: 0.6428245314390619
valid f1: 0.5844698679691321

valid auc: 0.625586886284441400<?, ?it/s]
valid pr: 0.6428480902602651
valid f1: 0.5844698679691321

valid auc: 0.625579769911787700<?, ?it/s]
valid pr: 0.6428063702870027
valid f1: 0.5847412312832689

valid auc: 0.625559174025684800<?, ?it/s]
valid pr: 0.6427925591891217
valid f1: 0.584881326548049

valid auc: 0.625550895969689500<?, ?it/s]
valid pr: 0.6427865765095414
valid f1: 0.5850214218128291

valid auc: 0.625546650579006500<?, ?it/s]
valid pr: 0.642746090965665
valid f1: 0.5850214218128291
Early Stopping
Overall ROC-AUC: 0.6281
Overall PR-AUC: 0.6443
Overall F1: 0.5863

This is far away from what is list in the origin paper. Is this because the dataset is sampled from the whole dataset?

Wang-Yu-Qing commented 3 years ago

My config is wrong. I can produce the right performance now.