evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
https://www.evidentlyai.com/evidently-oss
Apache License 2.0
5.44k stars 602 forks source link

Add a new `WordMatch` descriptor to Evidently #1308

Closed elenasamuylova closed 1 month ago

elenasamuylova commented 2 months ago

Add a New WordMatch Descriptor to Evidently

About Hacktoberfest contributions: https://github.com/evidentlyai/evidently/wiki/Hacktoberfest-2024

Description:

Evidently already has an IncludesWords() descriptor that checks if the text contains any (by default) or all specified words, returning a True/False result for each row. However, this descriptor uses a single shared list of words for all rows.

In some cases, such as when evaluating responses against specific ground truth answers, you may need a different list of words for each row. For example, you might want to check if generated responses contain the expected keywords for each row:

Example:

Question Generated Response Expected Words
"Name a primary color." "Red is a primary color." ["blue", "red", "yellow"]

What to Implement:

The new WordMatch() descriptor should:

  1. Accept a with_column parameter: This column contains a list of words specific to each row.
  2. Accept a lemmatize parameter. Default True, to consider inflected and variant words. (Same as IncludesWords() descriptor).
  3. Allow configuration for any or all words present. (Same as IncludesWords() descriptor):
  4. Return True/False for each row` if the specified condition (any or all) is met or not.

References:

jon-bown commented 1 month ago

Along with the with_column parameter, would the column we are getting the generated response from always be named the same? Or would we need it to resemble SemanticSimilarity implementation with two arbitrary columns being compared with WordMatch?