search

bricks-cloud / BricksLLM

🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
https://trybricks.ai/
MIT License
863 stars 60 forks source link

Health Check Endpoints: Documentation and Usage #68

Closed avukonkepeter-rain closed 2 months ago

avukonkepeter-rain commented 3 months ago

Hello World!

So there is currently a health check endpoint available at: 'api/health'

This API is not well documented and it is not clear if it is an internal API or if it safe for use in self-hosted or managed instances of BricksLLM.

If this endpoint is officially supported could it be included in the official documentation and it's use more thoroughly explained: https://docs.trybricks.ai/docs

Also whether this API differs between managed instances and self-hosted instances as in practice I've observed them to behave differently.

Clarity on the best way to monitor the service is highly valuable for deployments especially to cloud based infrastructure.

spikelu2016 commented 3 months ago

Got the feedback and added the health check endpoints to the documentation.

spikelu2016 commented 3 months ago

For monitoring, we offer datadog integration. Do you want a different integration for your use case?

kisamoto commented 2 months ago

@spikelu2016 - not OP but if we're talking health integrations it would be nice to have something like a prometheus exporter or OTEL. I use Grafana (specifically Grafana Cloud but could equally be relevant to open-source version) to create dashboards of data so an easy way to get this data there would be beneficial.

I can create a new issue if not relevant to this one.

EDIT: #63 seems to mention OTEL already, perhaps it could be wrapped into that

minhoryang commented 2 months ago

Why don't we just use these pipelines with a just simple modification on the BricksLLM?