Mixed fixes + support for Mistral API - Githubissues

mozilla-ai / lm-buddy

Your buddy in the (L)LM space.

Apache License 2.0

64 stars 3 forks source link

Mixed fixes + support for Mistral API #112

Closed aittalam closed 2 months ago

aittalam commented 2 months ago

Three main commits:

added allow_remote_code to config => allows us to run Phi models
fixed max_samples bug (now if the value is 0 all samples are considered, if it is larger than len(dataset) we get a warning and all the dataset is used)
added support for Mistral API (created a general APIModelClient class and made OAI and Mistral as two subclasses as so much code was in common)
bonus: added model name in output evals file :-)

Tested manually with different configs (seq2seq models locally, OAI, Mistral) and with pytest.

aittalam commented 2 months ago

LGTM once comments are addressed, 🔥

Thank you! The lm-harness entrypoint still fails tests but we can ignore it for now (it works when running integration tests locally, there might be a library version mismatch to fix)