I cannot reproduce the results of Freebase and AMiner

[ivam-he]

Hi, Ffffffffire I am trying to reproduce the results of HINormer on Freebase and AMiner, using your codes and scripts, but there is a huge gap between the results of mine and those published on your paper. Here are the logs when running:

python run.py --dataset AMiner --num-gnns 3 --len-seq 100 --beta 0.1 --temperature 0.1
repeat: 1
{'micro-f1': 0.8541893362350381, 'macro-f1': 0.7472750110308501}
repeat: 2
{'micro-f1': 0.8524483133841132, 'macro-f1': 0.7513401519532572}
repeat: 3
{'micro-f1': 0.8559303590859632, 'macro-f1': 0.7567903249575831}
repeat: 4
{'micro-f1': 0.8550598476605006, 'macro-f1': 0.7548341771060799}
repeat: 5
{'micro-f1': 0.8596300326441784, 'macro-f1': 0.7515768889725416}
Micro-f1: 0.8555, std: 0.0027
Macro-f1: 0.7524, std: 0.0036

python run.py --dataset Freebase --num-gnns 3 --len-seq 30 --num-layers 3 --dropout 0 --beta 1 --temperature 0.2
repeat: 1
{'micro-f1': 0.6764826175869121, 'macro-f1': 0.6157334967628284}
repeat: 2
{'micro-f1': 0.6666666666666666, 'macro-f1': 0.62344618380074}
repeat: 3
{'micro-f1': 0.6785276073619632, 'macro-f1': 0.5977709087708544}
repeat: 4
{'micro-f1': 0.6768916155419223, 'macro-f1': 0.6315030856090981}
repeat: 5
{'micro-f1': 0.665439672801636, 'macro-f1': 0.6003430074946984}
Micro-f1: 0.6728, std: 0.0062
Macro-f1: 0.6138, std: 0.0146

How can I reproduce the results shown in your paper?

[Ffffffffire]

Thanks for your attention! We noticed that there were some differences on details between our experimental codes and the released codes, so we update our codes for better reproducibility. As we didn't fix the random seed, the performances of HINormer may be unstable in different running. Our results reported in the paper came from hyper-parameter tuning process, and we suggest that you can try different hyper-parameters to test the performances of HINormer. We will also continue to try to improve the stability of model performance. Thanks again!

[OnlineWaAndRe]

i don't know why, i can't run this project, it show an error "***/HINormer/model.py", line 82, in forward graph.update_all(aggregate_fn, fn.sum(msg='m', out='h')) File "/home/dugking/anaconda3/envs/env_zt/lib/python3.9/site-packages/dgl/heterograph.py", line 5110, in update_all ndata = core.message_passing( File "/home/dugking/anaconda3/envs/env_zt/lib/python3.9/site-packages/dgl/core.py", line 393, in message_passing msgdata = invoke_gsddmm(g, mfunc) File "/home/dugking/anaconda3/envs/env_zt/lib/python3.9/site-packages/dgl/core.py", line 289, in invoke_gsddmm x = alldata[func.target][func.in_field] AttributeError: 'SimpleReduceFunction' object has no attribute 'target " could you help me to deal with it? thanks

[OnlineWaAndRe]

help me wuwuwuu File "/media/dugking/runcode/runcode/zt/hin/HINormer/model.py", line 81, in forward AttributeError: 'SimpleReduceFunction' object has no attribute 'target'

[OnlineWaAndRe]

Hi, Ffffffffire I am trying to reproduce the results of HINormer on Freebase and AMiner, using your codes and scripts, but there is a huge gap between the results of mine and those published on your paper. Here are the logs when running:

python run.py --dataset AMiner --num-gnns 3 --len-seq 100 --beta 0.1 --temperature 0.1
repeat: 1
{'micro-f1': 0.8541893362350381, 'macro-f1': 0.7472750110308501}
repeat: 2
{'micro-f1': 0.8524483133841132, 'macro-f1': 0.7513401519532572}
repeat: 3
{'micro-f1': 0.8559303590859632, 'macro-f1': 0.7567903249575831}
repeat: 4
{'micro-f1': 0.8550598476605006, 'macro-f1': 0.7548341771060799}
repeat: 5
{'micro-f1': 0.8596300326441784, 'macro-f1': 0.7515768889725416}
Micro-f1: 0.8555, std: 0.0027
Macro-f1: 0.7524, std: 0.0036

python run.py --dataset Freebase --num-gnns 3 --len-seq 30 --num-layers 3 --dropout 0 --beta 1 --temperature 0.2
repeat: 1
{'micro-f1': 0.6764826175869121, 'macro-f1': 0.6157334967628284}
repeat: 2
{'micro-f1': 0.6666666666666666, 'macro-f1': 0.62344618380074}
repeat: 3
{'micro-f1': 0.6785276073619632, 'macro-f1': 0.5977709087708544}
repeat: 4
{'micro-f1': 0.6768916155419223, 'macro-f1': 0.6315030856090981}
repeat: 5
{'micro-f1': 0.665439672801636, 'macro-f1': 0.6003430074946984}
Micro-f1: 0.6728, std: 0.0062
Macro-f1: 0.6138, std: 0.0146

How can I reproduce the results shown in your paper?

hello

[Ffffffffire]

It may be caused by the different versions of package dgl. The experimental environment is dgl 0.9.0, you should install the old version of dgl to run the code. Thanks!

[OnlineWaAndRe]

非常感谢，但是现在的版本就是0.9， 迷茫啊（非常感谢！！！） 能加个好友嘛（：

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: "Ffffffffire/HINormer" @.>; 发送时间: 2023年5月30日(星期二) 中午12:44 @.>; @.**@.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

It may be caused by the different versions of package dgl. The experimental environment is dgl 0.9.0, you should install the old version of dgl to run the code. Thanks!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[Ffffffffire]

The error lines in your logs were not consistent with the version 0.9.0, it seems like the latest version, please check your dgl version again.

[OnlineWaAndRe]

好消息，没有报找不到target的错误了，这个问题是Networkx版本不一样，（我安装了readme指定的Networkx版本就好了），坏消息是装的dgl版本是cpu版本，但是好像没找到dgl-0.09 GPU版本。能问一下，你的dgl-0.9-gpu怎么装的嘛？ 佬 orz 非常感谢

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: "Ffffffffire/HINormer" @.>; 发送时间: 2023年5月30日(星期二) 中午1:28 @.>; @.**@.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

The error lines in your logs were not consistent with the version 0.9.0, it seems like the latest version, please check your dgl version again.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[OnlineWaAndRe]

大佬，再次打扰一下，我从https://conda.anaconda.org/dglteam/linux-64，下载了

这个版本，并且把之前的版本之前的都给卸载了，然后也用了Networkx=2.8.4版本，但是还是出现没有traget这个报错。好难啊啊，大佬 捞我一把非常感谢

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: "Ffffffffire/HINormer" @.>; 发送时间: 2023年5月30日(星期二) 中午1:28 @.>; @.**@.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

The error lines in your logs were not consistent with the version 0.9.0, it seems like the latest version, please check your dgl version again.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[Ffffffffire]

I never meet the same problem, it is caused by the confusion of message function and reduce function. But I am confused why the 'aggregate_fn' is treated as 'SimpleReduceFunction' in your running. Keep 'aggregate_fn' as a normal 'CopyMessageFunction' should not cause this error. You can replace 'aggregate_fn = fn.copy_src('h', 'm')' with 'fn.copy_u('h', 'm')' as an attempt because ‘copy_src’ is not used in the latest version. If not work, you can give more details about bug, and I will try to find the reason in my spare time.

[OnlineWaAndRe]

感谢大佬，现在我重新下载了项目，然后全部更新了dgl（1.1.0 + 113）版本，然后改了copy_src 到copy_u, 然后运行的时候,不是python run.py,用你readme 提供的 python run.py --dataset DBLP --len-seq 50 --dropout 0.5 --beta 0.1 --temperature 2
这个命令跑，然后就可以了，再次感谢大佬，祝您下一篇更好 （： 谢谢！！！

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: "Ffffffffire/HINormer" @.>; 发送时间: 2023年5月30日(星期二) 晚上9:41 @.>; @.**@.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

I never meet the same problem, it is caused by the confusion of message function and reduce function. But I am confused why the 'aggregate_fn' is treated as 'SimpleReduceFunction' in your running. Keep 'aggregate_fn' as a normal 'CopyMessageFunction' should not cause this error. You can replace 'aggregate_fn = fn.copy_src('h', 'm')' with 'fn.copy_u('h', 'm')' as an attempt because ‘copy_src’ is not used in the latest version. If not work, you can give more details about bug, and I will try to find the reason in my sparce time.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[Ffffffffire]

解决了就好！谢谢！

[OnlineWaAndRe]

能加个联系方式嘛，大佬orz  

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年5月30日(星期二) 晚上10:12 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

解决了就好！谢谢！

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[Ffffffffire]

联系方式邮件发给我

[OnlineWaAndRe]

好滴，谢谢大佬，qq:2856230101, vx: 17839705211 

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年5月30日(星期二) 晚上10:19 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

联系方式邮件发给我

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[OnlineWaAndRe]

大佬，我对您的论文特别感兴趣，我觉得论文质量非常的高，读完之后收益颇深，代码水平也非常的高，但是读完之后我也有很多的疑问，希望大佬抽时间能加个好友讨论一下，非常感谢（： qq:2856230101 vx:17329358698 非常感谢！！！ 

张停 @.***

 

------------------ 原始邮件 ------------------ 发件人: "Ffffffffire/HINormer" @.>; 发送时间: 2023年5月30日(星期二) 晚上10:19 @.>; @.**@.>; 主题: Re: [Ffffffffire/HINormer] I cannot reproduce the results of Freebase and AMiner (Issue #2)

Closed #2 as completed.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>