aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
214 stars 46 forks source link

feat: add configurable param logical_operator (OR/AND) to factual knowledge #307

Closed kirupang-code closed 4 months ago

kirupang-code commented 4 months ago

Description of changes: I created a new configurable parameter called logical_operator that accepts "OR" (as default) and "AND" in FactualKnowledgeConfig. I added checks so that when the logical_operator is "AND", all possible target outputs must be included in model generation in order for the answer to be correct. I also updated all of the current configurations, and created new unit tests to check for behavior given invalid configurations and scores for valid inputs.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.