Similarly, Match does not strip the sampled string from white spaces in the front. That causes the evaluation to fail for models using the Completion endpoint, as those are more likely to output spaces in the front. Example:
It would be good to add an argument to Match allowing to require case insensitive behaviour and to determine whether the answer should be stripped of spaces. These can then be specified in the yaml file for a task.
Similar options can be added for the other templates, such as Includes and FuzzyMatch.
Describe the feature or improvement you're requesting
The current implementation of the
Matchbasic eval template is case-sensitive. This leads to results such as:{'correct': False, 'expected': 'no', 'picked': None, 'sampled': 'No', 'options': ['no']}Similarly, Match does not strip the sampled string from white spaces in the front. That causes the evaluation to fail for models using the Completion endpoint, as those are more likely to output spaces in the front. Example:
{'correct': False, 'expected': 'Mumbai', 'picked': None, 'sampled': ' Mumbai', 'options': ['Mumbai']}It would be good to add an argument to
Matchallowing to require case insensitive behaviour and to determine whether the answer should be stripped of spaces. These can then be specified in the yaml file for a task.Similar options can be added for the other templates, such as
IncludesandFuzzyMatch.Additional context
No response