search

KGQA / leaderboard

You can find the most recent KGQA benchmark numbers from publications here.
https://kgqa.github.io/leaderboard/
MIT License
108 stars 16 forks source link

MKQA dataset #34

Closed alexanderpanchenko closed 2 years ago

alexanderpanchenko commented 2 years ago

Thank you for creating the nice leaderboard. Do you plan to integrate the MKQA dataset? https://github.com/apple/ml-mkqa

xixi019 commented 2 years ago

Hey, sorry for late reply. Despite how much we would like to include more datasets, MKQA is more of a open domain question answering dataset than a KGQA dataset. Therefore, I do not think we would integrate MKQA. What do you think @RicardoUsbeck?

RicardoUsbeck commented 2 years ago

I think this is a very interesting special case. In particular, MKQA has answers grounded in a graph. Similar to FreebaseQA. Even though it has no KGQA logical form, e.g. a SPARQL query or a lambda expression, it still is valid. So, adding it makes sense if it fits the other criteria.

@alexanderpanchenko feel free to submit a pull request.

alexanderpanchenko commented 2 years ago

Thank you! From what I understood, part of the queries are answered from KG. But if the question is non-answerable with KG then there is no answer in this form.

Regarding pull request: in case you are not planning to implement yourself but is ready to include we may try to do it on our side, yes.

xixi019 commented 2 years ago

Then it makes sense to add this dataset.

@alexanderpanchenko It would be faster if you add the MKQA. We would publish an instruction video soon, you can check it out. :)

xixi019 commented 2 years ago

fixed by #49

alexanderpanchenko commented 2 years ago

Thank you. We will consider implementing it during fall 2022.

RicardoUsbeck commented 2 years ago

Implementing?

alexanderpanchenko commented 2 years ago

Thanks for integrating it: so far I did not had capacity to do it. It's great you managed to find resources to include it (and RuBQ too).