[request]: Security feature request to be able to run"proxy/init" container outside the "App mesh Injector"

[Mshahidtaj]

Tell us about your request What do you want us to build? "proxy/init" container requires elevated permission using NET_ADMIN capability and it also requires the "allowPrivilegeEscalation=true" property as well which is the blocker for us to apply PSP restricted policy to API containers in eks cluster.

Expected behaviour : we should be able to run "proxy/init" container functionality out side the "appMesh Sidecar" or pod which allow us to apply restricted PSP on pods without effecting "proxy/init" container functionality.

Which integration(s) is this request for? It is mainly requires for EKS and Kubernetes.

Are you currently working around this issue? How are you currently solving this problem? No work around except manually making sure that API containers don't run with un wanted security contexts inside the cluster.

Additional context Because of appmesh "proxyinit" privilege mode requirement, we need to run whole pod with "allowPrivilegeEscalation=true" . which means we cant apply restricted PSP (pod security policy) on api containers. It is a big security risk for the whole EKS cluster because as a part of our security measure we want to run all api containers in a restricted mode.

is it possible to run appmesh proxy/init container outside the pod as Daemonset ?or any other option to mitigate this issue. istio already have a fix for https://github.com/sabre1041/istio-pod-network-controller.

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[kiranmeduri]

Related issue in Kubernetes https://github.com/kubernetes/kubernetes/issues/69882. This will make it simple to use CNI without running a separate controller.

[kiranmeduri]

https://github.com/kubernetes/kubernetes/issues/69882 is still open.

tldr: CNI in Kubernetes that needs to read Pod annotations to perform its actions currently needs to read pod-info from API server (watch pods). This is logic is not performed in amazon-vpc-cni-plugins.

At this point best bet is to contribute appmesh support under https://github.com/aws/amazon-vpc-cni-k8s. Asking @mogren and @ofiliz for guidance.

[jaypipes]

@kiranmeduri are you recommending that we essentially subsume the functionality of aws-appmesh-proxy-route-manager into the VPC CNI routed-eni plugin? We would look for pod annotations and create the iptables rules as appropriate (basically, what aws-appmesh-proxy-route-manager currently does in-process in the sidecar)?

[mogren]

@kiranmeduri The CNI has a watcher already, scoped to pods on the current node:

https://github.com/aws/amazon-vpc-cni-k8s/blob/5283eaebe94dc6bc02d53c5106011e25133134a3/pkg/k8sapi/discovery.go#L126-L130

What is it that aws-appmesh-proxy-route-manager does?

[ofiliz]

I don't think we should replicate the logic in aws-appmesh CNI plugin. I assume you meant that the IPAMd daemon (not the vpc-cni-k8s CNI plugin itself) can watch the pods. In that case, instead of replicating the logic, IPAMd could either invoke the aws-appmesh binary itself, or use another mux CNI plugin to do so. However that work would be incompatible with CNI2 and vpc-resource-controller.

A much better solution is to do a full working backwards. Until then, use the init container instead of the CNI plugin.

[fawadkhaliq]

Adding this logic to existing CNI plugin would be limiting for multiple reasons 1) what ofiliz@ mentioned 2) we don't want to have a tight coupling/hard dependency between AWS CNI and App Mesh use-uses. There might be clusters which run a different CNI plugin so a CNI with init container only logic chained with other CNI plugins would be better 3) complexity in the VPC CNI (it doing more than one thing), bigger testing matrix, release dependencies across these use-cases