What's the meaning of β in Appendix B?

[coderlemon17]

In Appendix B, Eq. (17), the paper writes that: Later the paper states that "We set both β_1 and β_2 to 0.8 ." I was wondering why we can assume P(a=a_R|F) = (1-β_1), and why β_1 = 0.8?

[potsawee]

Hi @coderlemon17

The motivation behind using Bayes' Therom (Appendix B) instead of simple counting (i.e. non_factual_score = count_not_match / (count_match + count_not_match)) is because in our initial experiments when we applied unanswerability scores to filter out bad questions, some instances ended up with a smaller number of questions.

For example, [case1]: count_not_match = 2, count_match = 1 and [case2]: count_not_match = 12, count_match = 6, both give the same nonfact_score (e.g., 2/3) if we do simple counting. In contrast, if we use Bayes', it will take into account the amount of evidence, i.e. count_not_match = 12, count_match = 6 will yield a higher non-factual score than count_not_match = 2, count_match = 1

beta_1 is the probability that the two answers (a' and a_R) are not the same given that the sentence (to be evaluated) is False / non-factual. The value of beta_1 would be 1.0 if we assume that both QG and QA are perfect, while the value will be smaller if we assume that the systems can make errors. In other words, if we set the value to be high (e.g., close to 1.0), the effect of the amount of evidence will be higher. We tried 0.7, 0.8, 0.9 in our initial investigation, and it seems 0.8 was a reasonable value (although 0.7 and 0.9 would work too).

I hope this helps clarify the paper

Potsawee

[coderlemon17]

Hi @potsawee, thanks for your reply.

Yes, I understand why you use Bayes' Theorem here, but my question is "why can we assign an arbitrary value to β_1/2". Like you said, β_1=P(a≠a_R|F) is probability that QAs will make different choices given hallucinated contents.

So my doubt is, since QGs and QAs are fixed models, probability P(a≠a_R|F) should be a fixed (but unknown) value instead of one that can be arbitrarily chosen, right? For example, if we have very poor QGs and QAs, then β_1 should be low (instead of chosen 0.8 in the paper).

[potsawee]

Hi @coderlemon17 ,

Thank you very much for your comment, and I agree with your point that the current setup of selecting beta values is not great.

and yes, QGs and QAs are fixed, and Prob(a != a_R | F) can be computed empirically. But since we only have one evaluation dataset, tuning the beta values on this evaluation dataset would be somewhat cheating. We believe to do this properly we would have to collect some more data points to tune these beta values. In this work, we only tried a few beta values at the beginning and we stick with 0.8.

Best wishes Potsawee

[coderlemon17]

Hi @potsawee, thanks for your explanation😄, closing issue now.