🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
Bricks pulls key and provider configurations from PostgreSQL on a periodic basis. The time stamp we used resulted in some inconsistencies in production.
OpenAI proxy does not have a timeout.
Metrics API does not support querying by model and keyId.
Prod Dockerfile has privacy mode turned off.
Token counting is not accurate due to not using the correct tokenizer for certain models.
Health check endpoints are needed for k8s deployment.
Some OpenAI endpoints do not have specific models associated with them. To effectively track OpenAI usage, we need to add path and method.
Tasks
[x] Update memDB to fetch configurations from postgresSQL using last update timestamp.
[x] Add a default OpenAI proxy timeout of 180s.
[x] Add model and keyId filters to the metrics API
[x] Update prod Dockerfile to have privacy mode turned on
[x] Add health check endpoints for both admin and proxy servers
[x] Start recording path and method for each proxy request
Context
Tasks