How can I see the answer is from KB or document?

[pyupya]

Hi, i'm studying in KBQA, and attracted in it. I wanna see how many correct answer is from document or KB. But, I used you code and analyzed, I don't know how I can see some answers from document.

How can I analyze some correct answers are from document or KB?

Sorry for bothering you.

[haitian-sun]

Which dataset are you asking? Answers are provided in the data file.

You can also look at the pred files. They are the probability of each entities being a correct answer. Then you can track it down to the entity-linked retrieved documents or to the KB.

Hope it help.

[pyupya]

Thanks for reply, and it really helped. Thanks for your advise, I looked pred file, it has "dist" and "answers". I think they are the result of the model answered,

In pred file,

1. This file's sequence(I mean question order!) is same with test file?
2. Is "dist" means the entities that could be answer for the question?
3. I understood "Answers" in this file is real answer for the question, then if "dist" doesn't have entity in Answers, is that question wrong?

I'm doing this because I want to see "F1 score" for the model, like in paper! Thanks!

[haitian-sun]

1. Yes, it should be the same as test file.
2. They are logits.
3. That means we fail to retrieve any correct answers in the preprocessing step. The retrieval step is described in Sec 2.2 in the paper.

[pyupya]

I read paper again, I saw to compute F1 score, tuned a threshold on development set to select answers on paper 5.2. Then, can I use pred file's "dist" for classification? If a dist entity's score under threshold, model classify it not correct answer, or other one over threshold, it can be answer? If so, I wonder how much threshold is set. Thanks!

[haitian-sun]

Yes you can.

And yes, the ones above the threshold are correct answers.

The threshold for F1 is tuned on the dev set.

[pyupya]

Thanks for kind reply! Really helped!