vLLM Metrics for Prometheus

[nacartwright]

The vLLM metrics are not showing correctly. It should show time to first token, number of running requests, cpu/gpu cache...etc as shown here:

https://github.com/vllm-project/vllm/blob/main/vllm/engine/metrics.py

[devpramod]

@kevinintel The metrics explorer in prometheus does not show any vLLM related metrics.

[lvliang-intel]

You should connect to vLLM serving endpoint, not LLM microservice endpoint.

[kevinintel]

please modify endpoint in prometheus.yml ex:
static_configs:

• targets: ["llm-dependency-svc.default.svc.cluster.local:9009"]

[devpramod]

@nacartwright I was able to successfully test on a local machine with the following config for prometheus

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "vllm"
    static_configs:
      - targets: ["external_ip:port"]