aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
151 stars 40 forks source link

fix: update output record key validation logic in validate_call #254

Closed danielezhu closed 2 months ago

danielezhu commented 2 months ago

Description of changes: This PR relaxes the validation done in validate_call to only check if the keys in the self.output_keys attribute of the Transform are present in the output of __call__. Previously, we checked that the set difference between the current keys and the original keys is exactly equal to self.output_keys, but this validation is too strict for the BYO model outputs use case, where GetModelOutputs should overwrite existing model outputs in the record without validations failing.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.