aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
152 stars 40 forks source link

feat: fetch log probability jmespath from JS metadata #267

Closed keerthanvasist closed 1 month ago

keerthanvasist commented 2 months ago

Description of changes: Fetch log probability jmespath from JS metadata. This is a non-breaking change.

We were using a constant path, and so far that was also correct. But, going forward, for new models, it might not be. It is better to fetch the jmespath directly from metadata.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.