search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
187 stars 42 forks source link

Evaluation Algorithm for Recommendations #199

Closed mccartni-aws closed 7 months ago

mccartni-aws commented 7 months ago

Thanks for the great work.

I'm working with a customer that is looking to use LLMs for recommendation. We are proactively working on some evaluation metrics/algorithms.

Are there any plans to add any evaluation algorithms for recommendations? I am happy to contribute and submit a PR for this, if not.

danielezhu commented 7 months ago

Hi @mccartni-aws, thanks for reaching out!

An upcoming release will contain changes that make contributing to our codebase a much smoother experience, so if you aren't in a rush, it will be a more pleasant experience for you if you wait until then. However, contributions are always welcome.

mccartni-aws commented 7 months ago

Hey @danielezhu That's fine, I can wait until then 🦖

mccartni-aws commented 6 months ago

Hi @danielezhu Do you have any idea of when this upcoming release is coming?

danielezhu commented 6 months ago

We released v1.0.0 last week! The PRs in the change log of the form "Update [X algorithm]'s implementation to use Transform-based approach" are a good reference for how to use Transforms and TransformPipelines to implement new metrics and evaluation algorithms.