search

aws-samples / foundation-model-benchmarking-tool

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
https://aws-samples.github.io/foundation-model-benchmarking-tool/
MIT No Attribution
196 stars 31 forks source link

Emit live metrics #6

Open madhurprash opened 9 months ago

madhurprash commented 9 months ago

Emit live metrics so that they can be monitored through Grafana via live dashboard. More information to come on this issue but the goal here is to provide full flexibility to the user to be able to view metrics in ways that best suits the needs of their business and technological goals.

[TBD] --> Some sort of an analytics pipeline sending and emitting live results for different model configurations, their results and different metrics based on the needs of the user.

aarora79 commented 9 months ago

@madhurprash please get details from Shamik and add here in this ticket.

shamikatamazon commented 8 months ago

The intent of this functionality is to use Prometheus to store metrics and use Graphana as a tool to visualize graphs performance of an LLM workload. To achieve this, the client and possibly the serving engine should emit metrics which are scraped by a Prometheus agent and persisted in Prometheus. Graphana uses the data from Prometheus to display metrics in a dashboard that is provided by this project. The steps to deploy a model and the pipeline on an EC2 instance is outlined at https://github.com/shamikatamazon/genai/tree/main/benchmarking