Align EG and Istio to make sure they can work together

[zhaohuabing]

Description:

EG and Istio should be able to work together seamlessly. Users may start with EG at the edge and later want to evolve to Istio service mesh for traffic management, security, and obervablity inside mesh. I think it's crucial to bring this earlier to avoid any confilicts between these two projects and provide user a unified solution.

To achieve this, I see two possible approaches:

1. If the user only uses Kubernetes Gateway API and no EG API or extension is required, they can easily switch to the Istio Ingress Gateway as both EG and Istio use the Gateway API to configure the edge proxy.

2. If the user has used EG API and custom extensions, they should be able to continue using EG at the edge while using sidecar/waypoint proxies inside the mesh.

Overall option 2 should be feasible since the two project already converge at the Gateway API, which sets the foundation for compatibility. However, some areas need to be aligned to ensure they can work together. Below ar some I can think of, but it's definitely not a full list:

• How EG distributes traffic among backends in a multiple-clusters mesh. I know both projects (plan to) support multipe clusters, but the implementation details may differ.
• How EG obtains the client certificates and CAs needed to communicate with the services in the mesh(in ambient mode, this may not be necessary since mtls is handled by ztunnel)

[optional Relevant Links:]

Support Multi-Cluster Services using ServiceImport as a BackendRef

[Xunzhuo]

Thanks for raising it. But this is related to the direction of Envoy Gateway, it is better to collect some ideas from steering-committee and maintainers.

cc @envoyproxy/gateway-steering-committee @envoyproxy/gateway-maintainers

[zhaohuabing]

Sure, that's exactly why I raise this. IMO, keeping EG compatible with Istio is benificial for both projects, as well as the end users.

[arkodg]

hey @zhaohuabing can you please reframe the ask, does this issue pertain to ensuring Envoy Gateway and Istio can run in the same cluster and the user can configure both the control planes using Gateway API (and different GatewayClasses), with a sidecar running next to Envoy Gateway ?

[zhaohuabing]

@arkodg My primary goal is to ensure that EG and Istio are compatible with each other and don't step onto each other's feet. I see EG and Istio are actually complementary rather than competing. In sidecar mode, running a sidecar along with EG is a brilliant idea to achieving this goal, we need to twist the iptables rule a little though. In ambient mode, since the outbound traffic of EG goes through ztunnels and waypoints, it seems that we don't need any change and it would work?

[AliceProxy]

:+1: being incompatible with major service meshes like Linkerd and Istio will be a massive problem for adoption. I would like to make sure we don't only support istio's Ambient mode.

What we do in Emissary to integrate with Istio is (just pulling from the docs here):

1. have the users set these annotations on the Emissary pods:

   # These first two annotations tell Istio not to try to do port management for the
   # Emissary-ingress pod itself. Though these annotations are placed on the Emissary-ingress
   # pods, they are interpreted by Istio.
   traffic.sidecar.istio.io/includeInboundPorts: ""      # do not intercept any inbound ports
   traffic.sidecar.istio.io/includeOutboundIPRanges: ""  # do not intercept any outbound traffic
   
   # We use proxy.istio.io/config to tell the Istio proxy to write newly-generated mTLS certificates
   # into /etc/istio-certs, which will be mounted below. Though this annotation is placed on the
   # Emissary-ingress pods, it is interpreted by Istio.
   proxy.istio.io/config: |
   proxyMetadata:
     OUTPUT_CERTS: /etc/istio-certs
   
   # We use sidecar.istio.io/userVolumeMount to tell the Istio sidecars to mount the istio-certs
   # volume at /etc/istio-certs, allowing the sidecars to see the generated certificates. Though
   # this annotation is placed on the Emissary-ingress pods, it is interpreted by Istio.
   sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-certs"}]'

2. Setup our volume mount for the Istio secrets:

   
   # We define a single storage volume called "istio-certs". It starts out empty, and Istio
   # uses it to communicate mTLS certs between the Istio proxy and the Istio sidecars (see the
   # annotations above).
   volumes:
   - emptyDir:
     medium: Memory
   name: istio-certs

We also tell Emissary-ingress to mount the "istio-certs" volume at /etc/istio-certs in the

Emissary-ingress pod. This gives Emissary-ingress access to the mTLS certificates, too.

volumeMounts:

• name: istio-certs mountPath: /etc/istio-certs/ readOnly: true

  
  3. Emissary establishes a Filesystem watch on the mTLS certificates from /etc/istio-certs (per the AMBASSADOR_ISTIO_SECRET_DIR environment variable) and creates a Kubernetes `Secret` named `istio-certs`.
  4. You can then use that Kuberentes `istio-certs` `Secret` when configuring Emissary to talk to upstreams that are using the istio sidecars.

It's been a long time since the Emissary integration with Istio was written and I don't keep up with the Istio development very much, but if there is a way to get Istio to create the mTLS Kubernetes Secret(s) for us without having to write them to the filesystem and do all that then that would be greatly preferred.

[zhaohuabing]

@AliceProxy Can we just skip the traffic inception at the inbound side of EG and let the Istio proxy manage outbound traffic? By this way, we can eliminate the need for handling certificates. This approach shifts the burden of certificate management to Istio, we can also leverage some capabilities of Istio such as fine-grained traffic routing.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[costinm]

I would add to this bug "support ztunnel HA-Proxy header" so EG can act as a (sandwitched) waypoint. One of the design goals for ambient is to allow L7 and L4 to be swapped. Ztunnel would handle the mesh security/TLS side and L4 policies as well as forwarding of services to a compatible gateway.

While we have the design and consensus - what we lack is another gateway implementation to prove that it works and to guarantee that we don't accidentally couple L4 and L7 again.

AFAIK the main integration point is an extension to the HAProxy ( that envoy supports ) for passing workload identity and few other attributes from ztunnel to waypoint.

[arkodg]

hey @costinm can elaborate more on

• trigger (how does the L7 Gateway get programmed, via its own set of HTTPRoutes and Policies ? )
• setup (what is the exact configuration needed in envoy before relaying to backend) this can always be prototyped with EnvoyPatchPolicy :)

[costinm]

The waypoint ( gateway) uses normal config. It should receive requests from ztunnel, just like Istio would. The intent is to allow other reasons implementations to be swapped - but for this to happen we need some other implementation to try :-)

The L4 ztunnel takes care of crypto - so communication to the backend doesn't change. And it takes care of inbound capture - so requests sent to a Service get redirected to the gateway.

We are still going over the details - and changes are possible, but without some external feedback it'll be hard to achieve the swapping of implementations.

[zirain]

@costinm @arkodg should we wait for ambient beta first?

[costinm]

Ztunnel and L4 are technically beta. As a regular Gateway it should be ready to test.

As a Waypoint, there are quite a few PRs and discussions in progress - if you are interested to use Envoy Gateway as a possible ambient waypoint it may be worth getting involved.

My goal is to see multiple implementations and validate that the 'contracts' between L4 and L7 are good.

[zirain]

@arkodg do you think should EG try to play as a waypoint?

[kyessenov]

HBONE termination is not that difficult in Envoy - the difference between HAPROXY and HBONE variants of xDS is fairly small.

[arkodg]

@arkodg do you think should EG try to play as a waypoint?

@zirain I'm not aware of the implications. Ideally this should be fine if

• the integration can be represented as a doc (similar to the Intel QAT doc)
• the integration can be represented as a generic feature (similar to service routing needed for mesh traffic interception)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.