Closed clue1ess closed 1 week ago
What MITM proxy are you using?
Can you confirm exactly which v15/v16 versions of Teleport are affected here?
You mentioned 15.0.1 on Slack - can you try server version 15.4.9 (latest v15) and see whether the issue persists?
We are using mitmproxy v10.3.1(latest).
v15 and v16 both are affected. It works with v15.0.1 but does not work with v15.4.9 and v16.
Update from Slack
It works for v15.2.0 Looks like it broke somewhere between 15.2.0 and 15.4.9 Tested following scenarios : Works with v15.3.0 Works with v15.3.6 Works with v15.3.7 Does not work with v15.4.0
@webvictim Updated the description with required details.
Hmm, I'm not seeing any changes in the diff that look relevant: https://github.com/gravitational/teleport/compare/v15.3.7...v15.4.0
Maybeee https://github.com/gravitational/teleport/pull/42192? @greedy52 WDYT?
I can repro this but still debugging the root cause. I have two workarounds ATM before we can patch a proper solution.
Option 1: disable websocket
on mitmproxy: mitmdump --set websocket=false
Option 2: on each teleport agent, use the following env var when starting the agent to avoid websocket TELEPORT_TLS_ROUTING_CONN_UPGRADE_MODE=legacy
I found the root cause. We did a lousy job masking/unmasking websocket frames. When mitmproxy is interpreting websocket, it does it properly. Will put up the fix soon but hopefully it will catch the patch next week
Thanks @greedy52 for fixing this. When can we expect the fix in v16 version ?
Thanks @greedy52 for fixing this. When can we expect the fix in v16 version ?
https://github.com/gravitational/teleport/releases/tag/v16.2.1 released yesterday
We have deployed Teleport v16.1.0 on an Kubernetes cluster in AWS (EKS) behind an ALB and are using a MiTM proxy server in front of teleport agent through which all the traffic goes (from teleport agent to teleport server).
Expected behavior: Teleport agent should be able to connect to the teleport server via MiTM proxy server.
Current behavior: Teleport agent is able to connect to the teleport server via MiTM proxy server but the connection is dropped abruptly.
Logs : Teleport Agent side -
Teleport Server side -
Additionally, our MiTM proxy server is able to intercept TLS traffic and supports ALPN. Tried scenarios as below - [Working] 15 Agent -> MiTM Proxy -> 15 server [Not working] 15 Agent -> MiTM Proxy -> 16 server [Not working] 16 Agent -> MiTM Proxy -> 16 server
After investigation, we found that it broke at Teleport Server v15.4.0
Steps to reproduce :
docker run -d --rm -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p <private_ip_addr>:8081:8080 mitmproxy/mitmproxy mitmdump --set stream_large_bodies=-1