aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
155 stars 40 forks source link

Update f1 score in QA accuracy eval #166

Closed bilalaws closed 5 months ago

bilalaws commented 5 months ago

Description of changes: Strip the model output and the target text when computing the F1 score. With this change, the strings with an without trailing / leading whitespace characters will be considered equivalent (e.g., "yes" and "yes\n"). We need this change because FMs often include newline characters in their output.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.