search

cmavro / ReaRev_KGQA

[EMNLP Findings 2022] ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs
https://aclanthology.org/2022.findings-emnlp.181/
MIT License
21 stars 2 forks source link

KG incompleteness experiments for Webqsp #7

Closed Dinaelin-Yip closed 1 year ago

Dinaelin-Yip commented 1 year ago

Your work is very interesting. I want to conduct experiments for Webqsp 50% KG completeness. However, when I change the parameter 'fact_drop' in parsing.py to 0.5, the result is 71.14 for H1 and 58.62 for F1. Can you tell me what is the correct solution? Thanks a lot!

cmavro commented 1 year ago

Hi,

Thank you for your interest in our work. Could you try this solution https://github.com/cmavro/ReaRev_KGQA/issues/6#issuecomment-1520463167 ?

Dinaelin-Yip commented 1 year ago

Thank you for your kind reply. Do you mean that I should try the solution you have mentioned with the parameter 'fact_drop' be 1?

Dinaelin-Yip commented 1 year ago

Sorry, be 0, I meant.

cmavro commented 1 year ago

Yes, that is right, do not alter fact_drop.

Dinaelin-Yip commented 1 year ago

Thanks for your solution! However, after running the code on Webqsp with your solution, the average of F1/H1 is 66.41/53.41, much higher than the result in your paper with 50% KG completeness, which is 53.4/39.9.

cmavro commented 1 year ago

My misunderstanding; For Webqsp-incomplete (10%, 30%, 50%), we use the incomplete KGs provided by GraftNet.