[EKS] [request]: Integrated Policy Enforcement

[mikestef9]

Hi Everyone,

We are looking to get input on your experiences and opinions on policy definition and enforcement with EKS. Your feedback will be valuable and help us to build a more integrated product experience for policy enforcement.

Some questions that we have are:

• Are you performing policy enforcement today?
  • If no, why not?
  • If yes, what kind of policies are you applying (pods must have resource requests, pods can't use host networking, pods can't use persistent volumes, etc)
  • If yes, are you applying policies to meet security and compliance requirements, or simply limit what developers can do in a cluster?
• What tool(s) are you using for policy enforcement? (Kyverno, OPA/Gatekeeper, still using Pod Security Policies, other)
• Are you actively blocking pods from being scheduled, or only logging warnings?
• Are you configuring mutating or validating policies?
• If using GitOps, do you perform any policy evaluation at commit time, or only at runtime when manifests are applied to a cluster?
• Within you organization, are policy standards defined and written by a centralized group or by decentralized teams?
• Are you doing Kubernetes Network Policy enforcement?
  • If yes, what tool(s) are you using? (Calico, Cilium, etc)
• Do you use any other AWS services like AWS Config for any non Kubernetes policy enforcement today?
• What challenges are you experiencing with policy enforcement? Any other feedback?

Looking forward to hearing from everyone!

[grosser]

• Are you performing policy enforcement today? Yes
• Rules:
  • Pods must have resource requests
  • Pods can't use host networking/path/ipc/pid
  • Pods must have images from our private registry
  • Pods must have standard tags (and Deployment etc)
  • Don’t use default namespace
  • Don’t use privileged
  • Don’t use external load balancers
  • Don’t use external ingress
  • Don’t use clusterrole
  • Don’t use cluster-critical priority class
  • Don’t use certain tolerations
  • Don’t use nodeport service
  • Don’t run as root
  • Use a readinessprobe when using PDBs
  • Follow istio port naming conventions
  • Project labels must be in internal db
  • Use labels on volumeClaimTemplates
  • Don’t make your PDB unable to drain
  • Don’t use giant env vars (256kb max per object)
  • Correct internal istio config
  • Don’t use service with externalIPs
  • … and a few more internal
• Why: meet security and compliance requirements and limit what developers can do in a cluster (avoid bad patterns / avoid common mistakes)
• What tool(s) are you using for policy enforcement? (Kyverno, OPA/Gatekeeper, still using Pod Security Policies, other) OPA + custom tooling
• Are you actively blocking pods from being scheduled, or only logging warnings? Blocking + allow-list for projects or namespaces
• Are you configuring mutating or validating policies? validating
• If using GitOps, do you perform any policy evaluation at commit time, or only at runtime when manifests are applied to a cluster? Gitops, we validate which resources would violate new policies before deploying them and then add allow-list entries for them to do a smooth rollout, once that’s done we work with teams to remove them from the allow-list Within you organization, are policy standards defined and written by a centralized group or by decentralized teams? Maintained by a single team, but input can come from many places (rego is hard to write, so needs expertise)
• Are you doing Kubernetes Network Policy enforcement? No
• Do you use any other AWS services like AWS Config for any non Kubernetes policy enforcement today? Not sure
• What challenges are you experiencing with policy enforcement? Any other feedback? Validate before apply does not scale when doing locally, so for large clusters it needs to be done in cluster by N pods to get a reasonable turnaround time
  • Auditing policies is important to catch things that are live and would be denied from randomly failing when they restart/scale
  • It’s important for speed to not match on all kinds issue
  • Opa eva is slow

[joebowbeer]

I suggest adding these questions to the next annual AWS Container Security Survey.

Partial answers follow.

• Are you applying policies to meet security and compliance requirements, or simply limit what developers can do in a cluster? Both
• If using GitOps, do you perform any policy evaluation at commit time, or only at runtime when manifests are applied to a cluster? At pull_request time using conftest (.rego)
• Are you doing Kubernetes Network Policy enforcement? Cilium chained to vpc-cni

[mgamsjager]

Any update on this?