Closed athrvkk closed 4 months ago
Thank you for noticing this! We will fix it asap. However, we note that this mistake does not affect the evaluation results.
Thanks for your response @IINemo ! I am not entirely sure how this error would not affect the final results?
The results in the metrics depend on the ordering of the uncertainty scores. Since models have a fixed vocabulary, averaging (i.e. multiplying by a constant) would not affect the final ordering and the final metric scores. Correct me if you note something else.
Oh you mean the final metrics (PRR), then it's correct. Thanks for the clarification!
@rvashurin could you check, please?
Should be fixed now.
Hello,
I am not 100% sure, but I believe the entropy calculation is wrong here: https://github.com/IINemo/lm-polygraph/blob/main/src/lm_polygraph/stat_calculators/entropy.py
On line 43, shouldn't you compute the sum instead of the mean?
Also, the entropy should be calculated with base 2. The log probabilities (logprobs) returned by the HuggingFace language models typically use the natural logarithm (base e).