Integration with OpenAI Evals

[JuanmaMenendez]

OpenAI Evals is very popular solution for working with Evals, and it's growing each day (just look at the repo stars).

You should have a clear way to show how to integrate your product with their 'logs'.

Create some tutorials on the documentation. Create a command that allows running your product with the logs easily. Like zuno-evals (which is now unmaintained and kind of deprecated, pointing to all broken versions etc..)

You have a great opportunity of becoming a well-used product if you prioritize openAI evals.

[JuanmaMenendez]

Something like this https://github.com/zeno-ml/zeno-evals-hub which is already using Zeno (an older version) is great.

An easy way to integrate openai evals logs, with zeno-hub.

@cabreraalex any suggestion? I saw a big part of this is thanks to your amazing work ;)

[Sparkier]

Hi @JuanmaMenendez 👋

The repo you've linked is very out of date and only works with an old version of Zeno (we've rewritten most of Zeno from the ground up since then). If you want to integrate OpenAI Evals into Zeno, I'd take a look at our EleutherAI integration. What this provides is basically a script that uploads the results of the evaluation to ZenoHub.

I currently don't have the cycles for this (and I assume @cabreraalex doesn't either probably), but we'd be happy for any external contributions here!