🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
Currently this API purely acts a proxy so anybody willing to integrate with different providers needs to also implement an SDK for each provider in their code (my use case).
A huge appealing factor would be if we could use only the OpenAI SDK and purely by changing the URLs when initializing a new client get access to the different providers. Side note, it would also play nicely with #40 as the OpenAI client would handle those errors.
Perhaps each provider could be under a separate compatability URL e.g. http://localhost:8002/api/compat/openai/providers/{{ provider_slug }}/
Ollama implemented something similar recently via a middleware which maps OpenAI API objects to their own internal representation if you need some inspiration.
Currently this API purely acts a proxy so anybody willing to integrate with different providers needs to also implement an SDK for each provider in their code (my use case).
A huge appealing factor would be if we could use only the OpenAI SDK and purely by changing the URLs when initializing a new client get access to the different providers. Side note, it would also play nicely with #40 as the OpenAI client would handle those errors.
Perhaps each provider could be under a separate compatability URL e.g.
http://localhost:8002/api/compat/openai/providers/{{ provider_slug }}/Ollama implemented something similar recently via a middleware which maps OpenAI API objects to their own internal representation if you need some inspiration.