prometheus / common

Go libraries shared across Prometheus components and libraries.
Apache License 2.0
259 stars 306 forks source link

"error encoding and sending metric family" on HTTP/2 HEAD request #611

Open es-korneev opened 3 months ago

es-korneev commented 3 months ago

Hi!

I have encountered probably undesirable behaviour of node_exporter when running HTTP/2. node_exporter complained about HTTP/2 write errors when HEAD request method was used. Steps to reproduce:

  1. Run node_exporter with TLS:
    
    $ ./node_exporter --web.listen-address=:9101 --web.config.file=node_exp.yml --log.level=error

node_exp.yml


tls_server_config: cert_file: node_exp.crt key_file: node_exp.key

2. Send a request with `curl`:

$ curl https://localhost:9101 -Ik

`curl` will display 200 OK and exit cleanly, though `node_exporter` will write 299 lines like this:

ts=2024-03-23T18:08:58.844Z caller=stdlib.go:105 level=error msg="error encoding and sending metric family: http2: request body closed due to handler exiting"

3. The error will show up in `promhttp_metric_handler_errors_total` metric:

$ curl https://localhost:9101/metrics -sk | grep encoding promhttp_metric_handler_errors_total{cause="encoding"} 299


I have tracked errors down to `prometheus/common@v0.48.0/expfmt/text_create.go:96`, http2 connection complained about invalid writes `len(metric_families)` times. HTTP/1.1 deals OK with such requests.

Not sure whether it is a problem related to HTTP/2 implementation in Go or error handling issue in `promhttp`, but I feel like some sort of error handling can be added to the library.

Thank you for any comments!

Some info about environment:
### Host operating system: output of `uname -a`

Linux host 6.2.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 6 10:23:26 UTC 2 x86_64 x86_64 x86_64 GNU/Linux


### node_exporter version: output of `node_exporter --version`

node_exporter, version 1.7.0 (branch: master, revision: 6425f079d162ebd22d4c6c4e4d7e4a36ebbe2239) build user: mint@mint-Vostro-14-5410 build date: 20240323-17:52:29 go version: go1.22.1 platform: linux/amd64 tags: netgo osusergo static_build


### node_exporter command line flags

./node_exporter --web.listen-address=:9101 --web.config.file=node_exp.yml --log.level=error

node_exp.yml


tls_server_config: cert_file: node_exp.crt key_file: node_exp.key


### node_exporter log output

repeated 299 times

ts=2024-03-23T18:08:58.844Z caller=stdlib.go:105 level=error msg="error encoding and sending metric family: http2: request body closed due to handler exiting"


### Are you running node_exporter in Docker?
No.

### What did you do that produced an error?

curl https://localhost:9101 -Ik


### `curl --version`

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16 Release-Date: 2022-01-05 Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd