wbpcode
commented
4 weeks ago @kyessenov
Open agrawroh opened 4 weeks ago
wbpcode
commented
4 weeks ago @kyessenov
kyessenov
commented
4 weeks ago This is because connections to the internal listener are pooled since the first listener is HTTP. So when you get two HTTP connections to the first listener, they share the connection upstream to the internal listener, and the first one to make that connection gets the filter state. Remember, the transfer of the data happens at TCP level, not HTTP. If you want HTTP level transfer, the best option is to inject a header, and then parse it in the internal listener. Another alternative, is to implement hash() method on the transferred filter state object that will create distinct connection pools to the internal listener based on the value of the hash. Unfortunately, there's no out-of-the-box way right now, envoy.string doesn't implement it, but I think we can fix it and add another envoy.hashable_string that supports a unique hash.
agrawroh
commented
4 weeks ago This is because connections to the internal listener are pooled since the first listener is HTTP. So when you get two HTTP connections to the first listener, they share the connection upstream to the internal listener, and the first one to make that connection gets the filter state. Remember, the transfer of the data happens at TCP level, not HTTP. If you want HTTP level transfer, the best option is to inject a header, and then parse it in the internal listener. Another alternative, is to implement
hash()method on the transferred filter state object that will create distinct connection pools to the internal listener based on the value of the hash. Unfortunately, there's no out-of-the-box way right now,envoy.stringdoesn't implement it, but I think we can fix it and add anotherenvoy.hashable_stringthat supports a unique hash.
@kyessenov Thank you for the insights. I have a couple of quick questions:
max_requests_per_connection to 1 on my upstream cluster, which points to the internal listener, with the expectation that each connection would only serve a single request. Is this correct?kyessenov
commented
4 weeks ago agrawroh
commented
4 weeks ago "It's just standard header mutation rules. There are several ways, and I think there is even standard XFCC support in HCM." - Sorry, I was trying to understand how to pass the headers from outer listener to the internal listener. The only option I could find to do a state transfer is using metadata (Dynamic Metadata and State Filter).
kyessenov
commented
4 weeks ago I mean both terminating and internal listeners are HTTP - you can just decode it per-request and pass data this way.
agrawroh
commented
4 weeks ago I mean both terminating and internal listeners are HTTP - you can just decode it per-request and pass data this way.
In this case the outer listener is getting the encapsulated HTTPS traffic from the client and sending it to the internal listener which then decapsulates it:
route:
cluster: internal_listener_cluster
upgrade_configs:
- upgrade_type: CONNECT
connect_config:
allow_post: trueI tried adding the headers on the request stream like this but internal listener doesn't see these:
request_headers_to_add:
- header:
key: h1
value: v1
append_action: OVERWRITE_IF_EXISTS_OR_ADD
- header:
key: h2
value: v2
append_action: OVERWRITE_IF_EXISTS_OR_ADD@agrawroh I'm sorry, yes, I missed CONNECT termination. Then the path is to define a "hashable" filter state object. I can follow up with a first class support for this, or a custom object would also work.
agrawroh
commented
3 weeks ago @kyessenov I opened a PR following your suggestion to implement a new envoy.hashable_string. Could you PTAL?
kyessenov
commented
3 weeks ago I'm trying to parse the config and not seeing L7 pooling for internal connections. The first listener terminates the CONNECT, and then sends a raw TCP stream to the internal listener. That should mean each CONNECT stream is 1-1 with the internal connection, no? In that case, I don't understand where the sharing can come from.
agrawroh
commented
3 weeks ago I'm trying to parse the config and not seeing L7 pooling for internal connections. The first listener terminates the CONNECT, and then sends a raw TCP stream to the internal listener. That should mean each CONNECT stream is 1-1 with the internal connection, no? In that case, I don't understand where the sharing can come from.
@kyessenov The outer (downstream) listener accepts mixed traffic and based on the route matching rules, terminates the CONNECT and send the raw TCP requests to the internal listener for decapsulating. Internal listener should just be receiving only one type of traffic i.e. with raw TCP which needs decapsulating but given the behavior on the outer listener where it's receiving mixed traffic, is it possible that somehow the connections to internal listener are being shared?
I am not very familiar with this code but implementing the hashing solved the issue and we no longer see any state sharing.
wbpcode
commented
3 weeks ago I think we don't find the root cause. As @kyessenov said, a upgraded connection will never be shared. A possiblity is the clients didn't send requests with connect.
May a connect matcher could be used to ensure only connect requests will be routed to internal listener.
The hashable string may works around the issue, but it would be better to find the actual reason first.
agrawroh
commented
3 weeks ago I think we don't find the root cause. As @kyessenov said, a upgraded connection will never be shared. A possiblity is the clients didn't send requests with connect.
May a connect matcher could be used to ensure only connect requests will be routed to internal listener.
The hashable string may works around the issue, but it would be better to find the actual reason first.
In this case, the client is also an Envoy Proxy and is the only client which is sending requests to the Server (also an Envoy Proxy). Client Envoy is only sending the HTTP POST requests. I agree that we should keep this open and find the actual root-cause.
botengyao
commented
2 weeks ago This is because connections to the internal listener are pooled since the first listener is HTTP. So when you get two HTTP connections to the first listener, they share the connection upstream to the internal listener, and the first one to make that connection gets the filter state. Remember, the transfer of the data happens at TCP level, not HTTP. If you want HTTP level transfer, the best option is to inject a header, and then parse it in the internal listener. Another alternative, is to implement
hash()method on the transferred filter state object that will create distinct connection pools to the internal listener based on the value of the hash. Unfortunately, there's no out-of-the-box way right now,envoy.stringdoesn't implement it, but I think we can fix it and add anotherenvoy.hashable_stringthat supports a unique hash.
@kyessenov, related to HTTP data propagation, do you think we can implement a map in the filter state, like Map<request-id, metadata> from the previous HCM, and then propagate the filter state to the internal listener to be consumed? This needs to make all the filter state be shared.
Description
We're leveraging Envoy to establish an mTLS tunnel, following a setup quite similar to the standard configuration. Additionally, we're utilizing the Set-Filter State HTTP Filter to transmit the XFCC from the outer listener to the internal listener. On the internal listener, we apply a Header Mutation filter to add new headers, using expressions like %FILTER_STATE(tunneling.x-real-ip:PLAIN)%, to access the state data passed by the outer/main listener.
However, we've observed a race condition under high load, where there is a discrepancy between the metadata set by the outer listener and what the internal listener receives. It appears that the states from different requests are getting mixed up, leading to a situation where a request made with Certificate A arrives at the internal listener, but after reading the metadata, the XFCC header is added using Certificate B.
Notes:
We have also tried to limit the number of connection to make only a single request but it didn't help.
Repro Steps
You'll have to generate the cert files and then start both the client & server Envoys:
Start the Python Server for logging:
Now, if you make a ton of requests like this:
And grep the logs from any one of the Python Servers (we have two and each one should only have same type of requests), you'll see that some requests have mismatched values:
Configuration
Simple Server:
Server Envoy Configuration:
Client Envoy Configuration: