Added v0 of prometheus lm-buddy entrypoint

mozilla-ai / lm-buddy

Your buddy in the (L)LM space.

Apache License 2.0

64 stars 3 forks source link

Added v0 of prometheus lm-buddy entrypoint #75

Closed aittalam closed 7 months ago

aittalam commented 7 months ago

Here's the first version of the lm-buddy prometheus entrypoint!

I tried to keep the script relatively close to the original evaluation script, as I'd like to use it as a use case for a "conversion" of some existing tool (a tutorial is being written...)

I feel it still needs some massaging as I am not 100% sure of the implementation choices I took (format of configs, using one method or another to get artifacts, etc). Happy to receive some early feedback and iterate on it soon!

aittalam commented 7 months ago

Many thanks for your comments! I will go through them (marked all of the actionable ones with a 👍) and also fix the linting issues before asking for approval 🙏