search

symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
126 stars 5 forks source link

Openrouter Provider preferences #286

Open Munsio opened 1 month ago

Munsio commented 1 month ago

It seems that Openrouter now has a few provider which provide a lower quantization size than others, we need to ensure with our calls to openrouter that we are not going to mix those with multiple requests.

Providers with different quantization: https://openrouter.ai/models/meta-llama/llama-3.1-8b-instruct/status

Munsio commented 1 month ago

Provider Routing for Openrouter: https://openrouter.ai/docs/provider-routing

Munsio commented 1 month ago

This is a derivation of the OpenAI-API which means we need to gate the feature for OpenRouter by either forking the openAI api library we use or switch the API for OpenRouter completely.

Also what we maybe should do is creating our own API for OpenAI by generating everything from the documentation itself, with that we can at least add such custom parts as the original API library does not provide the necessary Public methods to do that.