aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
151 stars 40 forks source link

Support multiple data configs in evaluate #283

Closed athewsey closed 2 weeks ago

athewsey commented 1 month ago

Issue #, if available: #269

Description of changes:

Extend EvalAlgorithmInterface.evaluate() interface to support specifying a list of multiple data_config objects. evaluate() already returns a list of results by dataset, because when run with no data_config argument all applicable built-in datasets are analyzed. As mentioned in the attached issue, it was weird and confusing that users couldn't explicitly specify a set of more than one datasets to use.

Testing done:


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

athewsey commented 2 weeks ago

Rebased to current main. As I understood from the original review there wasn't actually a need for any change, but let me know if this is not the case!

Appreciate if y'all can help get this merged so we can have a more intuitive API 🙏