Research/Mark necessary pods as privileged

[whitlockjc]

We are currently using the DenyEscalatingExec admission controller to disallow someone using exec or run to run a privileged container. The purpose of this is security. For example, kiln uses the Docker socket directly and should be locked down. But there are also other situations where this could be useful, like Pods that are given access to communicate with Pods outside its own namespace, like the routers.

[noahdietz]

By adding the securityContext container spec property with privileged: true with the aforementioned DenyEscalatingExec admission controller activated, we are able to disallow exec or run commands on the marked container.

This should be applied to any container that has access to or manages sensitive cluster info, including

• kiln: has access to node's Docker host
• k8s-router manages the nginx config containing API key/secrets that shouldn't be seen.
• most things in kube-system namespace

[whitlockjc]

Applying this to k8s-router would make debugging nginx harder but I can understand why we'd do it since our network policies would allow traffic from there to be blindly accepted into any namespace. If that is our reasoning, we should probably do it for any container in any whitelisted namespace.

But we do need to think about this a little more because it would have a huge impact on debugging.

[noahdietz]

It was decided that currently, kiln is the only pod whose potential damage (i.e. access to the host Docker daemon) out-weighs the inability to debug with exec. Therefore it is now marked as privileged.