search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
185 stars 42 forks source link

[Feature] Image fields for multi-modal models #247

Open athewsey opened 5 months ago

athewsey commented 5 months ago

I'm trying to evaluate Claude v3's performance for some document understanding tasks, with a workflow that includes passing the image of the page in as one of the inputs.

Is fmeval considering native handling for image/multi-modal fields in input datasets?

keerthanvasist commented 5 months ago

Hey @athewsey, thanks for reaching out. At this time, we only support LLMs (that text based models).

Adding support for multi-model fields is a future roadmap item.