In our specific use case, we were attempting to scrape Prometheus metrics from JupyterHub's hub pod. In our environment, this is done within an Istio Service Mesh. Prometheus attempts to request the /metrics endpoint using HTTP/2 which Tornado does not support and generates error messages like the following:
HTTP message from 127.0.0.6: Malformed HTTP version in HTTP Request-Line: 'HTTP/2.0'
How to reproduce
Deploy JupyterHub to a kubernetes cluster with Istio enabled and the standard hub Service manifest.
Set up a Prometheus ServiceMonitor for the hub Service.
Tail istio-proxy or hub container logs.
Expected behaviour
Allow appProtocol to be set on the hub service (i.e. set to http to "downgrade" requests to the hub service) and specify appProtocol as http.
Actual behaviour
hub application generates errors when HTTP/2 requests are received.
consideRatio
commented
1 month ago
Bug description
In our specific use case, we were attempting to scrape Prometheus metrics from JupyterHub's
hubpod. In our environment, this is done within an Istio Service Mesh. Prometheus attempts to request the/metricsendpoint using HTTP/2 which Tornado does not support and generates error messages like the following:How to reproduce
hubService manifest.hubService.istio-proxyorhubcontainer logs.Expected behaviour
Allow
appProtocolto be set on thehubservice (i.e. set tohttpto "downgrade" requests to thehubservice) and specifyappProtocolashttp.Actual behaviour
hubapplication generates errors when HTTP/2 requests are received.