Compatible issues with HTTP Keep-Alive option, and TCP connection reuse

[lnshi]

I saw the comments/documents on the matchers: https://github.com/soheilhy/cmux/blob/master/matchers.go#L137 (HTTP1HeaderFieldPrefix returns a matcher matching the header fields of the first request of an HTTP 1 connection. If the header with key name has a value prefixed with valuePrefix, this will match.), this will cause issues with HTTP Keep-Alive option, and some connection pool, actually i am already experiencing problem with Chrome and istio-proxy/envoy.

Explanation:

1. I have 1st http server, and which has the matcher cmux.HTTP1HeaderFieldPrefix("User-Agent", "xxx/m.n"), and which has the endpoint /api/a
2. I have 2nd http server, and which has the matcher cmux.HTTP1Fast(), and which has the endpoint /api/b
3. I have 3rd grpc server, and which has the matcher cmux.HTTP2HeaderField("content-type", "application/grpc")

When the client turn on the Keep-Alive, or has the connection pool(only tag the connection by host+port), to reuse the underlying TCP connection, then this will happen:

• First request WITH the "User-Agent": "xxx/m.n", and call /api/a, then the TCP connection will be built and connected to above server 1, it will work fine;
• Second request WITHOUT the "User-Agent": "xxx/m.n", and call /api/b which is supposed to go to above server 2, but since client directly reuse the first connection(host+port is same, and no matcher check will happen), now the request will go to hit the above server 1, which will result 404
• Similar stuff will happen to server 3 also

I know the root cause is the client doesn't tag the connection properly, for our own connection pool we can do that, but for those public client, like Chrome, like those sidecar proxy product, if they are not configurable on how to tag one connection, then we can do nothing.

What is the solution for this kind of case, what are you guys' suggestions? Pls help.

@soheilhy @tamird @acomagu @tmm1

[soheilhy]

CMux is a transparent connection multiplexer. Once a connection is multiplexed, we cannot take back connections that are already multiplexed. In a sense, your problem requires a request multiplexer, which has to be implemented inside your server as requests come in, not outside. The only way to do this in CMux is fake a connection per request which is against the point and is quite expensive. So, I'm afraid, this is not something we can fix or add to CMux.

[pennyluis]

Leave

[lnshi]

@soheilhy thanks for ur reply, i totally agree with ur explanations and thoughts.

pls help to inspire me on what are those best use cases for cmux or more generally to say a layer 4 connection multiplexer?

[soheilhy]

Connection multiplexing is the only use case for Cmux, where you have multiple listeners and you want to use them transparently on a single port. It's not really about layers (i.e., L4 or L7 multiplexing), but about stickiness of multiplexing decision.