search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
152 stars 40 forks source link

fix: Context precision score computation #299

Closed awsvmaringa closed 1 day ago

awsvmaringa commented 5 days ago

Description of changes: This code: denominator = sum(model_verdicts) + 1e-10 will lead to inaccurate context precision score when sum(model_verdicts) > 0 and numerator > 0. For example: 1 / (1 + 1e-10) = 0.9999999999 Fixed this by returning 0 if sum(model_verdicts) is 0.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

oyangz commented 4 days ago

Looks good, could you run the linter with ./devtool lint? It will reformat the files so the lint check tests pass.