Why performances are not aligned with original work?

[giacoballoccu]

Hi! Congrats on the well-written implementation of this codebase, which really needed a more understandable version. Although being the evaluation side equal to the original, you should get results very similar(almost equal) to them. Have you wondered why this is not happening? Have you done any simplification that could have degraded the performances? Have you changed anything in the dataset? There is a gap of 0.25 points of NDCG@20 for amazon-book between your evaluation and the original one, which is pretty high

[glasses-on]

Hey @giacoballoccu Even i have the same results as you: "Best CF Evaluation: Epoch 0060 | Precision [0.0149, 0.0071], Recall [0.1420, 0.3070], NDCG [0.0756, 0.1130]" Whereas the precision is matching - the ndgc is off by 0.25 points. Did you look into it?

[dbbice]

Precision [0.0150, 0.0071], Recall [0.1443, 0.3105], NDCG [0.0760, 0.1137] I have the same proplem, has anyone sovled it?

[giacoballoccu]

The fact that the precision is fine could mean that NDCG is implemented differently but I'm not sure, I will invite you to check if they correspond. But honestly, when I looked at this code 2 months ago I noticed that it misses some implementations, if you compare this with the original KGAT you can see that, e.g. this repo is missing the KGAT aggregation technique.

[LunaBlack]

@giacoballoccu
Here is the code of aggregation part: https://github.com/LunaBlack/KGAT-pytorch/blob/master/model/KGAT.py#L10 . Did you mean this technique?

[Frankshal]

Has anyone sovled it?👀