Closed hotchilipowder closed 5 years ago
Hello,
Thanks for your issue and the kind words about our work!
Regarding the PubMed setup, we have found that slight changes to the early stopping strategy were necessary. Namely, we early stop based on the loss only in this case.
Here is a relevant code segment:
if val_acc_avg/vl_step >= vacc_mx or val_loss_avg/vl_step <= vlss_mn:
if val_loss_avg/vl_step <= vlss_mn:
vacc_early_model = val_acc_avg/vl_step
vlss_early_model = val_loss_avg/vl_step
saver.save(sess, checkpt_file)
vacc_mx = np.max((val_acc_avg/vl_step, vacc_mx))
vlss_mn = np.min((val_loss_avg/vl_step, vlss_mn))
curr_step = 0
Hope that helps!
Thanks, Petar
It should also be noted that, for PubMed specifically, a different choice of attention mechanism may yield even better results. Consider the AGNN paper, reporting 79.9% with a simpler attentional construction:
Thank you for your reply. After using your early stopping strategy, I get "0.788 +-0.3%".
Although there is still an error with the results of the paper, it has improved a lot than the previous results("0.777 +- 0.8%").
Hello, I am working on GAT. It's an excellent idea to introduce attention to graph.
But I have some questions about the results on Pubmed dataset which you said "79.0 +- 0.3%".
In the paper, it said :
Since you said "we use 500 additional nodes for validation purposes (the same ones as used by Kipf & Welling (2017))", I copy the data from gcn.
And I changed the hyperparameter to:
and run 100 times, here is the code difference, I get the results " 0.777 +- 0.8%" , which is not able to pass t-test.
Is there anything wrong about the experiments parameters ? Can you help me to reproduce the pubmed dataset results?
Thank you