Istio Ambient Mesh Compatibility

[feynmanliang]

Mirrord works great with Istio's traditional sidecar deployment mode, and even gives a helpful message about --steal when it detects service meshes.

However, I'm not sure if it's a bug or not implemented that mirrord doesn't work with Istio's ambient mode. Is this on the roadmap? The agent pod runs fine but ATM I need to remove a istio.io/dataplane-mode: ambient label from the namespace for mirrord to successfully steal traffic.

[aviramha]

Thanks for reporting this. I am looking into it.

[aviramha]

I managed to reproduce it. The setup seems similar to regular mesh (in terms of iptables) though I only see ISTIO_OUTPUT, so not sure if it's really an easy port. mirrord doesn't recognize it as part of the mesh detection (no sidecar, only annotated - maybe we can detect from iptables. can we trust the annotation as way to know?) ambient.istio.io/redirection: enabled

tbh I am not sure we can fix it as fast as we usually do (very busy period) so it'd be good to see if other users need it so we can prioritize it better.

[aviramha]

Hey, @DmitryDodzin from our team tried to reproduce it but couldn't. I tried to re-reproduce it but seems like I'm running into other (just setting up a sample) issues. Do you mind re-testing and seeing if it somehow was fixed since you last tried?

Thanks!

[feynmanliang]

No problem I will try to repro today.

[feynmanliang]

I am having some issues with istio ambient 1.22.1 due to a partial implementation of PROXY protocol (https://github.com/istio/ztunnel/pull/850) - I created https://github.com/istio/ztunnel/issues/1124 and will see if I can repro in the earlier version

[feynmanliang]

OK I verified that the issue is still present in 1.21.0.

I followed the guide in https://istio.io/latest/docs/ambient/getting-started/ to get set up.

Afterwards, I used mirrord to steal traffic from the productpage-v1. When I connect to the HTTPRoute, i expect for this traffic to be stolen and routed to my mirrord process. Instead, it appears the traffic is not stolen and I can see the request appear at productpage-v1.

Not sure if this is helpful or not since I am out of my depth, but FWIW in ambient mode I can see mirrord obtaining a ztunnel connection from logs

│
│ 2024-06-07T19:25:52.095358Z  INFO xds{id=2}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Address" size=1 remove │
│ s=0                                                                                                                                                      │
│ 2024-06-07T19:26:04.505215Z  INFO ztunnel::inpod::statemanager: pod WorkloadUid("fe8e746c-7ca9-402e-9f61-7a97b4f4ba2d") received netns, starting proxy   │
│ 2024-06-07T19:26:04.515067Z  INFO ztunnel::proxy::inbound: listener established address=:15008 component="inbound" transparent=true                      │
│ 2024-06-07T19:26:04.525406Z  INFO ztunnel::proxy::inbound_passthrough: listener established address=:15006 component="inbound plaintext" transparent=tru │
│ e                                                                                                                                                        │
│ 2024-06-07T19:26:04.535869Z  INFO ztunnel::proxy::outbound: listener established address=:15001 component="outbound" transparent=true                    │
│ 2024-06-07T19:26:04.546294Z  INFO ztunnel::proxy::socks5: listener established address=127.0.0.1:15080 component="socks5"                                │
│ 2024-06-07T19:26:05.134103Z  INFO xds{id=2}: ztunnel::xds::client: received response type_url="type.googleapis.com/istio.workload.Address" size=1 remove │
│ s=0                                                                                                                                                      │

[aviramha]

Thanks! Can you share more information about the cluster? is it GKE/EKS/AKS/Local? if so what version/flavor?

[aviramha]

I think I managed to reproduce it. We'll take it internally. Thank you. @DmitryDodzin I created a machine, installed kind + followed the guide and then run

mirrord exec -f mirrord.json --steal -t deployment/productpage-v1 -- python3 -m http.server 9080

and requests weren't stolen I'll give you machine details.