search

huggingface / text-generation-inference

Large Language Model Text Generation Inference
http://hf.co/docs/text-generation-inference
Apache License 2.0
8.36k stars 948 forks source link

Prometheus metrics does not work as expected. #2184

Open sywangyi opened 5 days ago

sywangyi commented 5 days ago

System Info

Runtime environment: Target: x86_64-unknown-linux-gnu Cargo version: 1.79.0 Commit sha: N/A Docker label: N/A nvidia-smi: N/A xpu-smi: N/A

Information

Tasks

Reproduction

I want to use https://github.com/huggingface/text-generation-inference/blob/main/docs/source/basic_tutorials/monitoring.md to monitor TGI case. however. I got blank output. I use curl http://127.0.0.1:80/metrics -i to get the metrics output, find the content-length is also 0. see HTTP/1.1 200 OK content-type: text/plain; charset=utf-8 vary: origin, access-control-request-method, access-control-request-headers access-control-allow-origin: * content-length: 0 date: Thu, 04 Jul 2024 07:46:06 GMT

which could explain why I got blank output in https://github.com/huggingface/text-generation-inference/blob/main/docs/source/basic_tutorials/monitoring.md.

this is my prometheus console. not tgi metric output either.

image

Expected behavior

could reproduce what's described in https://github.com/huggingface/text-generation-inference/blob/main/docs/source/basic_tutorials/monitoring.md

bryanSwk commented 5 days ago

I faced the same issue using ghcr.io/huggingface/text-generation-inference:2.1 container.

/metrics route works fine using ghcr.io/huggingface/text-generation-inference:2.0.

sywangyi commented 5 days ago

@OlivierDehaene

birshert commented 5 days ago

Same, no metrics for me starting from 2.1.0. Everything was fine in 2.0.4

sywangyi commented 5 days ago

find it is caused by https://github.com/huggingface/text-generation-inference/pull/2129/files metric could be shown up after downgrading metrics-exporter-prometheus to v0.12.1, metric 0.23.0 could with metrics-exporter-prometheus 0.15.1