search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
187 stars 42 forks source link

feat: conversion of targets to strings #167

Closed franluca closed 8 months ago

franluca commented 8 months ago

Issue #, if available: Factual knowledge and QA evaluations fail if the targets are not string (because we call split on the targets). Boolean and integer are common types for targets. One practical issue is that arrow's to_json function automatically converts true and false's to boolean even if they are capitalized strings.

Description of changes: Added conversion of targets to strings before calling _get_score.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

franluca commented 8 months ago

@danielezhu yes, casting after reading the dataset would be preferable. could you point me to the part of the code that is responsible for that?

danielezhu commented 8 months ago

Closing this PR since I've raised a separate one to handle things during the data-loading stage.