search

THUDM / CogDL

CogDL: A Comprehensive Library for Graph Deep Learning (WWW 2023)
https://cogdl.ai
MIT License
1.72k stars 314 forks source link

Leaderboard for heterogeneous graphs is missing #435

Open amblee0306 opened 1 year ago

amblee0306 commented 1 year ago

In the Simple-HGN code's README (https://github.com/THUDM/CogDL/tree/master/examples/simple_hgn), it mentioned that there is a leaderboard but the link is not working. Do you know where I can find this heterogeneous graphs leaderboard?

cenyk1230 commented 1 year ago

Hi @amblee0306 ,

You can find the cogdl leaderboard from this link. If you want to see the original repo of Simple-HGN, please check this link

amblee0306 commented 1 year ago

Hi @cenyk1230

Thanks for your reply, the leaderboard stated that the models were evaluated using macro-f1 scores, can I check how can I change the code such that the macro-f1 scores are returned? Currently when I run the simple-hgn from cogDL, I believe the test_acc is being utilized instead.

cenyk1230 commented 1 year ago

Hi @amblee0306 ,

It seems that the results in the leaderboard should be micro-f1 scores (the same as accuracy in the multi-class setting) rather than macro-f1.

amblee0306 commented 1 year ago

Hi @cenyk1230

Yes indeed. I would like to get the macro-f1 scores as well. Do you have any advice on how that is possible? Also, the score on the leaderboard, is it only for one instance or it refers to the average micro-f1 score? Cause I ran 10 instances and the average was lower than what was mentioned on the leaderboard.