[EKS/fargate] : Support for ISTIO service mesh sidecars

[studees]

We are very keen to migrate our EKS workloads over to the recently released Fargate integration.

Unfortunately it appears as though it is not currently possible to deploy pods that use ISTIO injected sidecars, which seems to significantly limit the potential use of fargate.

The istio-proxy sidecar itself can be injected successfully but we've not managed to apply the subsequent iptables configuration to direct all traffic via the envoy proxy within the sidecar.

Istio supports two mechanisms for this:

1) istio-init container - this fails as under fargate the pod cannot be deployed within the required NET_ADMIN permission 2) istio-cni - this fails as we cannot schedule daemonsets on fargate, and even if we were to manually reconfigure as a sidecar then we cannot mount the required /opt/cni/bin host volume.

Does the migration to firecracker and the associated move of the fargate agent to outside the VM change the security posture for granting pods NET_ADMIN access? This appears to be the most flexible option, but we'd like to request support for network sidecar proxies be provided by whatever means...

Thank you!

[ofiliz]

Please note that Fargate already supports App Mesh as a managed solution for both ECS and EKS. Customers don't need to run any init containers or additional CNI plugins to setup any iptables rules, which also means that no privileges (NET_ADMIN etc.) are required.

P.S. in 2024: My comment above was in response to the initial issue which requested supporting init containers with NET_ADMIN privileges to run sidecar proxies for any mesh. The issue has since then been updated to ask specifically about ISTIO, hence the answer no longer matches the request.

[dracmic]

Hello @ofiliz ,

I am new in Fargate stuff ... So ask you directly maybe a stupid question: Can I use istio sidecar injector with Fargate? Best regards.

[eightnoteight]

Hello @ofiliz ,

I am new in Fargate stuff ... So ask you directly maybe a stupid question: Can I use istio sidecar injector with Fargate? Best regards.

hey @dracmic , yes, all components of istio which doesn't use istio-proxy are possible to run in fargate, only istio-proxy container needs NET_ADMIN to add the iptable rules

Please note that Fargate already supports App Mesh as a managed solution for both ECS and EKS. Customers don't need to run any init containers or additional CNI plugins to setup any iptables rules, which also means that no privileges (NET_ADMIN etc.) are required.

@ofiliz it's awesome that app mesh as a managed solution is supported in EKS, but I think the support for istio is very much needed because app mesh doesn't offer as much flexibility as istio does in terms of custom protocols, adding filters and many more things.

fargate as a managed solution solves many of management and reliability problems of self-managing ec2 instances but restrictions like being unable to use istio with it is a huge blocker from adopting fargate in production workloads

[vorishirne]

the NET_ADMIN is a priviliged user, whom even I wouldnt allow if fargate does. Hence, Istio now provides ISTIO-CNI, we need to use it. https://github.com/istio/istio/tree/master/cni

[studees]

the NET_ADMIN is a priviliged user, whom even I wouldnt allow if fargate does. Hence, Istio now provides ISTIO-CNI, we need to use it. https://github.com/istio/istio/tree/master/cni

As far as I'm aware Fargate does not allow us to deploy CNI, or any other daemonset either.

Aside from that, relying on daemonsets for single pod worker nodes will be very inefficient, and exarcebates the existing race conditions of the CNI pod being scheduled & starting before the application pod is scheduled

[vorishirne]

@studees can we use this cni plugin from fargate https://github.com/aws/aws-app-mesh-roadmap/issues/3 I can raise a pr in istio, but I am not aware of the aws side permissions.

[wmorgan]

This isn't Istio-specific, the same applies to @Linkerd or to any service mesh that does injection via iptables.

[vorishirne]

Ya we can make atleast istio support locked cloud services

[wmorgan]

@Dhua-v it's not about Istio adding support for Fargate, it's about Fargate adding support for this type of pod network programming, which will allow any service mesh (or other tool) to do proxy sidecar injection.

[vorishirne]

@wmorgan aws seems to have provided this https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ProxyConfiguration.html

But I am not aws guy, cant say how we can utilize that.

[purkhusid]

@ofiliz Is this something that the EKS/Fargate team is looking into or should we not expect anything to happen in the near future? Would love to try out EKS with fargate but we use Linkerd and we don't want to stop using it!

[ofiliz]

@purkhusid Thanks for the feedback. I'll tag @tabern for evaluating this proposal.

[wmorgan]

I know there are lots of AWS Linkerd users, like @purkhusid, who would love to be able to use Fargate. If this issue makes progress maybe we can also remove the word "ISTIO" from the title :)

[HenryYanTR]

How does App Mesh sidecar, also Envoy, get around with this issue? Is there anything AWS can make it generic for other network sidecars?

[CodeFreezr]

How does App Mesh sidecar, also Envoy, get around with this issue? Is there anything AWS can make it generic for other network sidecars?

Anything new on this topic? We are really wanna shift our workload to gRPC/EKS/Fargate with Istio & Kiali on Top. AWS Appmesh is not an option.

[silverlyra]

How does App Mesh sidecar, also Envoy, get around with this issue? Is there anything AWS can make it generic for other network sidecars?

Normally, the App Mesh Kubernetes controller injects an init container, just like Istio does. And just like Istio, in order to configure the Pod’s network traffic to pass through its Envoy proxy, the init container has the NET_ADMIN capability. Fargate rejects any Pods that have that capability – even if you use the same init image App Mesh uses, your Pod will fail to run, with the error:

Pod not supported on Fargate: invalid SecurityContext fields: Capabilities added: NET_ADMIN

But App Mesh can also use CNI to set up the Pod’s networking, where an init container with NET_ADMIN is not needed. When the App Mesh controller processes a Pod that’s running on Fargate, the code contains a special case where the init container is skipped, and it simply adds some annotations to the Pod for the benefit of App Mesh’s CNI implementation.

That code has a comment:

// Fargate platform has appMesh-cni enabled by default

So, to answer the question above:

How does App Mesh sidecar, also Envoy, get around with this issue?

App Mesh uses CNI, in the same way (and for the same security reasons) that Istio can also use CNI instead of a privileged init container. But unlike with Istio, where you need to deploy a DaemonSet so that every node’s kubelet has local access to the Istio CNI implementation, AWS bundles App Mesh’s CNI implementation into the VM image they launch to run Fargate pods.

Is there anything AWS can [do to] make it generic for other network sidecars?

The only reason App Mesh works on Fargate is that AWS built support for it into Fargate. The only way I can think of adding support for Istio (or any third-party CNI plugin) would be to allow DaemonSets to run on Fargate, letting Fargate’s internal kubelet schedule the DaemonSet pods alongside your application pod.

[zeletrik]

Can we get an official answer for that? It's been a long. Will Fargate nodes gonna support privileged containers or at least DaemonSets?

[mreferre]

Can we get an official answer for that? It's been a long. Will Fargate nodes gonna support privileged containers or at least DaemonSets?

This is something we are exploring so we can't answer the specific questions because we don't know yet if/how/when these capabilities are going to be supported.

[wmorgan]

From the Linkerd perspective we'd love to see this and would be happy to help test.

[david-yu]

Also chiming in here as a representative for HashiCorp Consul, in addition to Istio and LinkerD. In general I believe this would heavily benefit the service mesh ecosystem as well as accelerate broader EKS Fargate adoption by allowing a greater percentage of workloads to utilize Transparent Proxy for dialing services within a service mesh.

[Hronom]

Same here, AWS Fargate looks like a great option to save money and probably avoid headache with managing and provisioning EC2 workers for EKS as same as continuous monitoring of "free resources" in the cluster.

[Hronom]

It seems like Amazon playing dirty here and trying to destroy all other Mesh Service solutions expect their home made AWS App Mesh.

Since AWS App Mesh at current state not really competitive(luck of features and community) against Istio and others.

This in general push new companies who didn't started yet use Kubernetes to build zoo of homemade frameworks that will manipulate Fargate based infrastructure in order to re-invent wheel of Service Mesh/Regular Micro-services approaches to save money comparing to EC2 solution

[samhays]

What if all of this was baked into selecting a fully managed Istio CNI? AWS could run all the hidecars and abstract the magic without exposing the wiggly bits?

[dvins]

Three and a half years; no feedback from AWS.

[MateSousa]

4 years guys :1st_place_medal:

[heungh]

Great! I forgot this for a long time

[lisenet]

Istio + EKS user here. 4 years and still no love for Istio sidecars on Fargate? Shame.

[runev-admin]

Indeed, shame AWS!!

[ofiliz]

I no longer work for AWS, but I keep getting notifications on this thread and sometimes frustrated messages directly to me. Before I unsubscribe, out of respect for those who still wait for an answer, I wanted to close the loop because I don't think anyone else will respond.

I cared deeply about Fargate. But you can't make any progress on any long-term vision when your organization changes every six months, your management chain changes every single year, and with it entire roadmaps get thrown out of the window before the ink dries. We had 7 general managers in 6 years! Middle management kept fighting each other over the control of various products to expand their kingdoms. All engineers gave up and already left; I wouldn't expect any updates.

[lisenet]

@ofiliz thank you for your input, appreciated.