Open remko opened 2 years ago
I would recommend reading https://ahmet.im/blog/grpc-http-mux-go/, I think it addresses this
Hi @howardjohn . Thanks for the link. However, I wanted to avoid using ServeHTTP
because it is experimental and not officially supported, and has performance issues.
I don't think cmux
can handle this. Since its a connection muxer. cloudrun (and Istio, FWIW) will multiplex gRPC and plain http2 methods onto the same connection. So if you use cmux, you might be able to get it to look like it works, but start seeing obscure failures in real world when the connections are reused. That blog was the result of us finding out that problem in prod :stuck_out_tongue:
Without knowing the code or HTTP/2, my guess from what I'm seeing is that the cmux matcher sends a http2 settings frame during negotiation, but when it still fails to match gRPC and falls back to the next 'regular' http2 handler, the http2 handler starts negotiation from scratch and sends its own settings frame, which causes problems
https://github.com/lukeo3o1/cmux/blob/677b4b20e70ac112b035428e88c80dd673af9b59/matchers.go#L252-L289
Maybe fix your guess
Background: Services on Cloud Run can only expose 1 port. If you want to expose a combination of gRPC and a regular REST API, you have to multiplex them. (e.g. using cmux)
In a standard setup, Cloud Run downgrades HTTP/2 requests to HTTP/1, and the setup with cmux seems works fine. However, if you want to support gRPC streaming, you have to enable End-to-end HTTP/2 support on your Cloud Run service. The Cloud Run instructions say:
So, my current setup is:
This unfortunately does not work (even locally, no Cloud Run involved):
hsrv
, usingcurl --http2-prior-knowledge
) works finecurl --http2-prior-knowledge
) no longer works: I get the errorhttp2: server connection error from [::1]:64054: connection error: PROTOCOL_ERROR
Without knowing the code or HTTP/2, my guess from what I'm seeing is that the cmux matcher sends a http2 settings frame during negotiation, but when it still fails to match gRPC and falls back to the next 'regular' http2 handler, the http2 handler starts negotiation from scratch and sends its own settings frame, which causes problems