search

ICT-GIMLab / SeHGNN

100 stars 17 forks source link

Inconsistency between SeHGNN paper results and actual running results. #6

Closed lichaoaaron closed 1 year ago

lichaoaaron commented 1 year ago

Hi, your work is very interesting, and most of the results are solid. However, after I run the command of IMDB, I got five results: 68.80, 66.83 68.33, 66.38 68.35, 66.48 68.96, 67.08 68.01, 66.70 These results are much lower than the reported 69.17, 67.11.

Yangxc13 commented 1 year ago

Thanks for your attention. As the HGB benchmark does not make public the full labels of the IMDB test set, we submit our predictions to HGB's online evaluation website and get those scores posted in our paper.

If you want to re-produce scores in the paper, please try:

  1. Submit your predictions to HGB's website for online evaluation. It seems that your five scores are from local evaluation involving only half of the test set. Please check hgb/Readme.md for more detailed instructions.
  2. Run more trials (e.g. 20 runs with different seeds) and select predictions from the five best runs based on the accuracy and/or the loss on the validation set.
lichaoaaron commented 1 year ago

Thanks for your kind response. Actually, HGB made the test set public in 2023.3.2. So the results are from the full test set. I will run more trials and select the five best runs based on your suggestion. Thank you!