shunkakinoki
commented
1 year ago Getting the same error as well - @PeterFaiman did you happen to find any solutions
Closed PeterFaiman closed 8 months ago
shunkakinoki
commented
1 year ago Getting the same error as well - @PeterFaiman did you happen to find any solutions
LucioFranco
commented
1 year ago You need to use https://docs.rs/tonic/latest/tonic/transport/channel/struct.Endpoint.html#method.tls_config to enable the tls negotiation iirc
abizjak
commented
10 months ago I am experiencing the same issue, even with enabling TLS config explicitly.
onsails
commented
8 months ago Getting the same error when connecting to a service behind Google Cloud Load Balancer. This endpoint can be successfully queried with grpcurl or buf curl --protocol grpc. Also, connecting with tonic to the same service directly (without TLSConfig, directly to a non-tls port of the service which is behind a GCP LB) – works well.
ajwerner
commented
8 months ago The deal seems to be that tonic is enforcing that the server offer ALPN protocol negotiation. This doesn't seem like a standard thing in the gRPC ecosystem. This can commonly arise if you implement a gRPC service that is not using TLS but then you have a load balancer in front of the service that does the TLS termination. While many of these load balancers offer ALPN options (for example, later nginx in the stream plugin), they can be painful to use in more complex scenarios.
The painful code is here https://github.com/hyperium/tonic/blob/23106dd76882a4ffbf2d3ac59c9c403518e429c9/tonic/src/transport/service/tls.rs#L79-L81.
In some cases, this error might help folks who have pointed their client at the wrong server -- instead of getting garbage errors later, they get a hint that the server is not speaking h2. But, as this issue demonstrates, this is not a panacea.
I think that a reasonable but aggressive change would be to change the default behavior to match that of other clients in the ecosystem -- don't require ALPN, and perhaps allow users to opt into the current behavior.
A more measured solution would be to allow users to opt out of the current behavior. One place in which that's a little bit painful is that most users probably end up relying on the default TLS config and don't set the TlsClientConfig (which is pretty painful to use). I put up this approach in https://github.com/hyperium/tonic/pull/1640.
Bug Report
Version
Original issue: 0.8.3 Still reproducible: 0.9.2
Platform
Linux XXX 6.1.0-0.deb11.5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1~bpo11+1 (2023-03-05) x86_64 GNU/LinuxDescription
I was trying to set up opentelemetry-otlp to connect directly to Grafana Cloud, using the tonic connector, but tonic returns an
H2NotNegotiatederror. As otel uses tonic 0.8.3 at present, I sought to make a minimum reproducible example with the latest tonic (0.9.2).I don't know for sure if this is a bug in tonic or a bug in grpc-go / Grafana Tempo (https://github.com/grafana/tempo), and I don't know enough about HTTP/2, TLS, or ALPN to figure it out myself. In any case, I would expect any 2 gRPC clients to interoperate cleanly. I am filing the issue here since I have trace logs from the client, but not the server. I am sure the server works / is up, as I can send and view traces forwarded through the Grafana Agent to the same endpoint.
I tried this code:
Example cargo project with tracing: https://github.com/PeterFaiman/tonic-issue-minimal-example
I expected to see this happen: connect to the remote server
Instead, this happened: H2NotNegotiated
Full output with TRACE level:
The original code looks like this, in case it matters: