[EKS] [request]: Provide proper PodSecurityPolicy's for AWS provided cluster components

[Niksko]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Currently, EKS ships with a very permissive eks.restricted PodSecurityPolicy. All authenticated users can create pods that use this security policy.

In the interests of maintaining good security posture, it would be preferable to have least-privileged PSPs that accompany AWS provided EKS components. This would include at minimum, the components that are installed on EKS out-of-the-box, i.e.:

• The AWS CNI daemonset
• Kube proxy
• CoreDNS

The AWS CNI daemonset would be the real work here, because the other two are open source and easy to find PSPs for.

At this point, EKS could ship without the eks.privileged PodSecurityPolicy. Alternatively, you could leave a PSP called privileged with the same privileges, and document how to give privileged pods permission to use that PSP. This would result in a better security posture by default in EKS clusters.

Furthermore, it would be great to have a PSP for the Amazon Cloudwatch agent for EKS. This is what prompted this request, because crafting a PSP for this component has proven difficult.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We're attempting to craft a better security posture in our Kubernetes clusters, with least-privileged access for all components. Having a default eks.privileged PSP is insecure, so we are trying to have an individual PSP for each component that needs elevated access, as per this great blog article: https://developer.squareup.com/blog/kubernetes-pod-security-policies/.

Are you currently working around this issue? Trial and error to get a PSP that works for a particular component. The Cloudwatch agent has proven difficult, because even after providing access to the hostPaths that are mounted by the container, the agent seems to be trying to access other files on disk, which it can't do.

[TBBle]

The AWS CNI daemonset is also open-source, see https://github.com/aws/amazon-vpc-cni-k8s. They have an outstanding request for a Pod Security Policy (https://github.com/aws/amazon-vpc-cni-k8s/issues/911).

There was some work on reducing the required permissions at https://github.com/aws/amazon-vpc-cni-k8s/issues/796, so currently the only privileged container used by CNI is an initContainer, which should limit the scope of the PSP to whatever that initContainer needs, plus NET_ADMIN and a set of hostPaths for the daemonset pod.

[ankitwal]

The AWS CNI DaemonSet(https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.5/config/v1.7/aws-k8s-cni.yaml) version 1.7.5 does not declare that it needs NET_RAW explicitly. It relies on the docker defaults(https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). So if the PSP 'requiredDropCapabilities' that capability then daemonset pods get scheduled but fail. On top of that the aws-k8s-agent fails out silently. Took us one whole day to figure out the issue, then trail and error to get the right capability added in the PSP.

[TBBle]

Does that mean you now have a working PSP and securityContext.capabilities for aws-node's initContainer?

[chadlwilson]

@TBBle Yes - however it relies on defaultAddCapabilities given the current default AWS resource YAML does not correctly declare it needs NET_RAW and don't want to hack it. Looks like a PR to here would probably change the default generated resources for those kubectling into their clusters at least.

As an alternative one could kustomize or kubectl patch the daemonset afterwards. Or use the Helm chart where one can customize the values. :-)

The one below is also namespace wide (including kube-proxy + coredns) whereas it should be possible to scope the bindings to the individual service accounts if necessary, to avoid having to, say, give coredns more privileges than it needs. May also be possible to tighten other aspects too (allowPrivilegeEscapation? hostPorts?)

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: kube-system-psp
spec:
  privileged: true
  allowPrivilegeEscalation: true

  allowedCapabilities:
    - NET_BIND_SERVICE # coredns
    - NET_ADMIN # aws-node
    - NET_RAW # aws-node
  requiredDropCapabilities:
    - ALL
  defaultAddCapabilities:
    - NET_RAW # Remove when aws-node properly declares its capabilities

  volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - hostPath

  allowedHostPaths:
    # kube-proxy + aws-node
    - pathPrefix: "/var/log"
    - pathPrefix: "/run/xtables.lock"

    # kube-proxy
    - pathPrefix: "/lib/modules"

    # aws-node 1.7.5
    - pathPrefix: "/opt/cni/bin"
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/var/run/dockershim.sock"
    - pathPrefix: "/var/run/aws-node"

  # For aws-node
  hostNetwork: true
  hostPorts:
    - min: 0
      max: 65535
  hostIPC: false
  hostPID: false
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  readOnlyRootFilesystem: false