reproduced results is much lower than what you reported

[tomsonsgs]

We use all your original settings on WN dataset, training epoches for conv is 200 as you set, here are the results of epoch 200 ( the results are nearly stable after ~ epoch 100): hit10: 0.523 hit3: 0.403 hit1: 0.222 MRR: 0.332 mean rank: 2525 All the results are far lower compared with the results you claimed, even the closest hit10 is only at the level of CONVKB or CONVE. Can you provide your training details to reproduce the results on WN? The results is very similar to only using CONVKB which is the base model of your GAT, i wonder that if GAT really works in your model.

[chauhanjatin10]

Hi @tomsonsgs, thanks for showing interest in our paper . We humbly apologise for the inconveniences caused by our mistake while writing the parameters in the paper. We will update the paper soon. Please wait for a while.

[foreseez]

Hi @tomsonsgs, thanks for showing interest in our paper . We humbly apologise for the inconveniences caused by our mistake while writing the parameters in the paper. We will update the paper soon. Please wait for a while.

have you update your code and paper?

[jpainam]

Hi, any update on the code. I also tried to reproduce the results, but they are far lower than the ones reported in the paper. Here is the results

Current iteration time 7215.4601266384125
Stats for replacing head are -> 
Current iteration Hits@100 are 0.6771545827633378
Current iteration Hits@10 are 0.536593707250342
Current iteration Hits@3 are 0.42031463748290016
Current iteration Hits@1 are 0.3009575923392613
Current iteration Mean rank 2806.7089603283175
Current iteration Mean Reciprocal Rank 0.38105117378694725

Stats for replacing tail are -> 
Current iteration Hits@100 are 0.7055403556771546
Current iteration Hits@10 are 0.5454856361149111
Current iteration Hits@3 are 0.42920656634746923
Current iteration Hits@1 are 0.27975376196990426
Current iteration Mean rank 2161.073529411765
Current iteration Mean Reciprocal Rank 0.374568554845391

Averaged stats for replacing head are -> 
Hits@100 are 0.6771545827633378
Hits@10 are 0.536593707250342
Hits@3 are 0.42031463748290016
Hits@1 are 0.3009575923392613
Mean rank 2806.7089603283175
Mean Reciprocal Rank 0.38105117378694725

Averaged stats for replacing tail are -> 
Hits@100 are 0.7055403556771546
Hits@10 are 0.5454856361149111
Hits@3 are 0.42920656634746923
Hits@1 are 0.27975376196990426
Mean rank 2161.073529411765
Mean Reciprocal Rank 0.374568554845391

Cumulative stats are -> 
Hits@100 are 0.6913474692202461
Hits@10 are 0.5410396716826265
Hits@3 are 0.4247606019151847
Hits@1 are 0.29035567715458277
Mean rank 2483.891244870041
Mean Reciprocal Rank 0.37780986431616914

And here the reported results

[liyirui-git]

This is my result, WN is pretty closer to the paper but FB is much lower than the result in paper

- WordNet

Current iteration time 1379.8727445602417
Stats for replacing head are ->
Current iteration Hits@100 are 0.7065663474692202
Current iteration Hits@10 are 0.5629274965800274
Current iteration Hits@3 are 0.466484268125855
Current iteration Hits@1 are 0.3508891928864569
Current iteration Mean rank 2471.312585499316
Current iteration Mean Reciprocal Rank 0.425575541927745

Stats for replacing tail are ->
Current iteration Hits@100 are 0.737688098495212
Current iteration Hits@10 are 0.594733242134063
Current iteration Hits@3 are 0.4945280437756498
Current iteration Hits@1 are 0.36046511627906974
Current iteration Mean rank 1573.5516415868674
Current iteration Mean Reciprocal Rank 0.4447488848113082

Averaged stats for replacing head are ->
Hits@100 are 0.7065663474692202
Hits@10 are 0.5629274965800274
Hits@3 are 0.466484268125855
Hits@1 are 0.3508891928864569
Mean rank 2471.312585499316
Mean Reciprocal Rank 0.425575541927745

Averaged stats for replacing tail are ->
Hits@100 are 0.737688098495212
Hits@10 are 0.594733242134063
Hits@3 are 0.4945280437756498
Hits@1 are 0.36046511627906974
Mean rank 1573.5516415868674
Mean Reciprocal Rank 0.4447488848113082

Cumulative stats are ->
Hits@100 are 0.7221272229822161
Hits@10 are 0.5788303693570451
Hits@3 are 0.4805061559507524
Hits@1 are 0.35567715458276333
Mean rank 2022.4321135430916
Mean Reciprocal Rank 0.4351622133695266

- Freebase

Current iteration time 1022.5329549312592
Stats for replacing head are ->
Current iteration Hits@100 are 0.7704276347979254
Current iteration Hits@10 are 0.5673255700166356
Current iteration Hits@3 are 0.45581759467658284
Current iteration Hits@1 are 0.35844994617868675
Current iteration Mean rank 241.34342890693804
Current iteration Mean Reciprocal Rank 0.42902612382184857

Stats for replacing tail are ->
Current iteration Hits@100 are 0.8045797044720618
Current iteration Hits@10 are 0.6071533418142675
Current iteration Hits@3 are 0.4882082395537724
Current iteration Hits@1 are 0.3831588218025247
Current iteration Mean rank 182.3465114003327
Current iteration Mean Reciprocal Rank 0.4582423684068296

Averaged stats for replacing head are ->
Hits@100 are 0.7704276347979254
Hits@10 are 0.5673255700166356
Hits@3 are 0.45581759467658284
Hits@1 are 0.35844994617868675
Mean rank 241.34342890693804
Mean Reciprocal Rank 0.42902612382184857

Averaged stats for replacing tail are ->
Hits@100 are 0.8045797044720618
Hits@10 are 0.6071533418142675
Hits@3 are 0.4882082395537724
Hits@1 are 0.3831588218025247
Mean rank 182.3465114003327
Mean Reciprocal Rank 0.4582423684068296

Cumulative stats are ->
Hits@100 are 0.7875036696349936
Hits@10 are 0.5872394559154516
Hits@3 are 0.4720129171151776
Hits@1 are 0.3708043839906057
Mean rank 211.84497015363536
Mean Reciprocal Rank 0.4436342461143391

[Ckeai]

0.35844994617868675

Did you get this result after solving the problem of data leakage？