calculate_prec_at_k does not fit the description of the paper

[ghost]

According to the paper:

p@10 means the precision of finding the closest graph among the predicted top 10 most similar ones.

However, the code is to calculate precision at the lowest k elements. And ground_truth is normalized GED which means the similarity.

So the code actually calculates the least similar ones, and the precision is thus outrageously high due to the uneven dataset (like AIDS).

https://github.com/Thinklab-SJTU/GENN-Astar/blob/b9e97a67590314598679988ef0a07798753257ac/src/utils.py#L35-L42

https://github.com/Thinklab-SJTU/GENN-Astar/blob/b9e97a67590314598679988ef0a07798753257ac/src/astar_genn.py#L844-L845

https://github.com/Thinklab-SJTU/GENN-Astar/blob/b9e97a67590314598679988ef0a07798753257ac/src/astar_genn.py#L825-L826

https://github.com/Thinklab-SJTU/GENN-Astar/blob/b9e97a67590314598679988ef0a07798753257ac/src/astar_genn.py#L679

[rogerwwww]

Hi there,

Many thanks for pointing this out. The reason for this issue is that we follow the code from another repo and we overlooked this issue in the original code (I would have to admit the code seems sound at first glance). I fixed this issue on my side and did a quick evaluation on GENN. Here are the updated results:

The other A*-based methods need more time to evaluate so I will update a fix once the results are ready.

Finally, permit me to explain some more details. This issue also existed in the repo that we followed in 2020:

https://github.com/gospodima/Extended-SimGNN/blob/602e615176afcedbbfecf9d71bfbe7b987758164/src/utils.py#L46-L59

And gratefully the maintainer of Extended-SimGNN has fixed this issue in a commit in Jul 2021 (https://github.com/gospodima/Extended-SimGNN/commit/d59318cda7c3cc34fafab4b784ce3292464c6703).

[ghost]

Thanks for your quick response and correction. It is a sad story for both of us because my model is based on part of your code. haha

[rogerwwww]

Thanks again and sorry for the inconvenience 😥😥