feat: add col_to_validate to evaluate_dataset

aws / fmeval

Foundation Model Evaluations Library

http://aws.github.io/fmeval

Apache License 2.0

214 stars 46 forks source link

feat: add col_to_validate to evaluate_dataset #294

Closed oyangz closed 5 months ago

oyangz commented 5 months ago

Issue #, if available:

Description of changes:

Add col_to_validate parameter to support instances where the model and model output are not required. See discussion here: https://github.com/aws/fmeval/pull/289#discussion_r1636956969.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

oyangz commented 5 months ago

This change is no longer needed, will directly compute scores and save outputs in the evaluate method for eval algos that don't require the model/model output.