Validate that Karpenter is adhering to Pod Security Standards when installing in E2E

[jonathan-innis]

Description

We would like to ensure that proper Pod Security Standards are ensured by the Karpenter controllers that are installed. Today, Karpenter adheres to these Pod Security Standards, but we are currently not ensuring that bar by doing any check in functional or E2E testing.

Currently, we are publishing a warning based on this label that is added to the namespace here: https://github.com/aws/karpenter-provider-aws/blob/main/.github/actions/e2e/install-karpenter/action.yaml#L42. We used to enforce this pod security standard in the namespace rather than just warning on it which was changed here: https://github.com/aws/karpenter-provider-aws/commit/8f500c23be18aa5cb8089a83c43e763303faa9ac. We had to make this change because there are some components in the kube-system namespace which are not adhering to the pod security standard (not Karpenter) that we didn't want to do custom patches to control.

There are a couple options to enforce this pod security standard:

1. Run custom patches on the addons that are added to the E2E test clusters so that they all adhere to the pod security standards of the namespace and we can change the pod security standard back to pod-security.kubernetes.io/enforce=restricted
2. Check the output of the apply operation that happens when Karpenter is installed. Since warnings should be printed by the apiserver if there are violations of the pod security standard, we can just check if the output is empty to ensure that we are adhering to the standard.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[sftim]

Karpenter always meets at least one Pod security standard (privileged) because this is the loosest standard and all Pods that you can define today meet this standard no matter what you do.

Do we also want Karpenter to be able to meet one of the other two standards (baseline and restricted)?

[sftim]

Should karpenter run in kube-system? I typically deploy it into a namespace named karpenter.

:thought_balloon: we could define a namespace for SIG Autoscaling use, to cluster-level components like node autoscaling. That could be a hope for karpenter but also other projects; maybe you have a bootstrap node autoscaler that runs another way and all it does is ensure that there's a node where Karpenter can run.

Would need SIG Architecture confirmation, but we can support additional system namespaces and they don't have to be part of the cluster baseline. If we do that, I also recommend revisiting https://github.com/kubernetes/kubernetes/issues/117807 about putting a label on system namespaces. Happy to explain why - ask on that other issue if the query is about why in general it can help.