search

shuoli90 / Rank-Calibration

This is the repo for constructing a comprehensive and rigorous evaluation framework for LLM calibration.
MIT License
8 stars 2 forks source link

About the negative values about degree_u and ecc_u #2

Open wangyifei0047 opened 4 weeks ago

wangyifei0047 commented 4 weeks ago

According to the paper, the uncertainty metrics of degree_u and ecc_u are non-negative. But I ran llama2 on the triviaqa dataset, whose results include negative degree_u and ecc_u. Did I make some configurations wrong? Thanks for your time!

shuoli90 commented 3 weeks ago

Hi wangyifei0047,

First, we recommend that you check the original implementation here: Uncertainty measures, which is the basis for our implementation.

For the degree uncertainty, our implementation is based on this: degree uncertainty, which does not guarantee non-negativeness. So, in theory, the uncertainty scores should be within [0, 1]; depending on the implementation, the range might change.

For the Eccentricity, our implementation is based on this: eccentricity uncertainty, which should be able to ensure the non-negativeness after taking the norm. Is it likely that the negative values are due to numerical issues? For example, -8.0e-7 or even smaller values?

Happy to further discuss.