search

ernestojimenezruiz / logmap-matcher

LogMap: An Ontology Alignment and Alignment Repair System
Apache License 2.0
123 stars 31 forks source link

Hits@K and MRR for OAEI 2022 data #28

Closed Murchana closed 2 years ago

Murchana commented 2 years ago

Hi, I am trying to generate the results for LogMap (Table 4 in https://arxiv.org/pdf/2205.03447.pdf) using the standalone release.

I am using the mappings file generated by LOGMAP ( as stated in https://code.google.com/archive/p/logmap-matcher/wikis/LogMapCommandLine.wiki) for computing all the metrics. Though I am getting precision, recall and f1 score the same as reported in Table 4 in https://arxiv.org/pdf/2205.03447.pdf, I get different values for Hits@K and MRR.

Am I looking at the correct mapping file for generating Hits@K and MRR or do we need to generate the score for the negative candidates separately and then compute these metrics?

For calculating Hits@K and MRR, I follow the same logic as in https://github.com/KRR-Oxford/DeepOnto. Is this correct?

ernestojimenezruiz commented 2 years ago

Hi

Give a look to https://github.com/ernestojimenezruiz/logmap-matcher/blob/master/src/test/java/uk/ac/ox/krr/logmap2/test/oaei/TestNewOAEILargeBio2022.java

LogMap was not meant to be used as a ranker so we implemented a small ranker as a baseline.

Ernesto

Murchana commented 2 years ago

Thanks!