search

grafana / helm-charts

Apache License 2.0
1.67k stars 2.28k forks source link

[tempo-distributed] gRPC streaming over HTTP not supported by gateway #3218

Open guelfey opened 4 months ago

guelfey commented 4 months ago

The Tempo documentation mentions a configuration setting that enables the querier-frontend to expose the gRPC API for streaming back query results on its normal HTTP port. But the gateway nginx instance doesn't actually support this configuration: in the current configuration, it always uses HTTP 1.1 to talk to the query-frontend; however gRPC only works over HTTP2.

In our case, we simply worked around this by configuring the Grafana datasource to directly connect to the query-frontend Service URL - however, this is only possible since we're running both in the same Kubernetes cluster and don't actually require most of the gateway's functionality like auth. In the general case, incoming traffic from the "outside" should go through the gateway - but then I would also expect it to support this feature constellation.

vaibhhavv commented 1 month ago

Hi @guelfey, I am also facing this problem. I tried to connect directly the grafana to query-frontend but still it give the error and the streaming is not getting enabled. Have you made some more additional changes to enable the functionality?