search

shmsw25 / FActScore

A package to evaluate factuality of long-form generation. Original implementation of our EMNLP 2023 paper "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation"
https://arxiv.org/abs/2305.14251
MIT License
238 stars 32 forks source link

Don’t understand how true_score and false_score are represented in logits #17

Closed rubaha96 closed 1 year ago

rubaha96 commented 1 year ago

factscore/factscorer.py lines 219-220, in _get_score:

true_score = logits[5852] false_score = logits[7700]

Is there any intuition about these particular logit indices and how logits are formed?

shmsw25 commented 1 year ago

Hi @rubaha96, these are vocab ids for True and False (in LLAMA tokenizer). So these scores represent how likely the model will complete the sequence with True or False.