Improve security (privileged mode being used in kube-system ns)

[jalberto]

Current draftd (0.7.0) run a container in kube-system namespace with privileged flag, meaning it can access k8s system containers.

I currently have no suggestions how to deal with this (capabilities?, isolated namespace for building?)

[bacongobbler]

We did this to support docker multi-stage builds as kubernetes ships with an older version of Docker for runtime requirements. Since Draft relies on only the "container builder" portions of docker, the attack surface is much smaller. Once Kubernetes starts rolling out with docker 17.05 or higher by default we can remove this hack by mounting in the host's docker socket [as we did before](). We'll actually gain a lot of added benefits on certain cloud providers. For example, ACS clusters are automatically authenticated and can pull images from ACRs deployed in the same resource group.

https://github.com/Azure/draft/pull/163

[jalberto]

Correct me if I am wrong, but mounting host's docker socket still dangerous, in particular in kube-systems namespace.

Safest solution i could find (for gitlab-runner with same issue) is to created an isolated namespace and force it to run in an node with not other pods running

[bacongobbler]

It's not the best option, but it's better than what we have today. 😄

docker build has an implicit requirement on docker run to build the image layers, so there's no way around the requirement to run the socket in unprivileged mode. There have been efforts to get docker build to work in a more secure fashion with the proper capabilities set for the container, but AFAIK nobody's been able to successfully document the process to do that.

The best alternative (and my preferred alternative) would be to run builds against a container builder running outside of the cluster, managed by someone else. Using a cloud-specific container builder like Google's Container Builder would be my preferred option, but that is tied specifically to Google Container Registry and is the only solution provided by the big three at this time (no equivalent for AWS/Azure users).

[bacongobbler]

Safest solution i could find (for gitlab-runner with same issue) is to created an isolated namespace and force it to run in an node with not other pods running

curious, what do you mean by this? Do you have an example?

[jalberto]

@backpackerjmk so I created a namespace (testing) with a default serviceAccount with permissions to only access it's own namespace, then I added a small node where only gitlab-runner (and siblings) are deployed (using nodeSelector).

Still insecure, but the privileged container will only be able to access local (node) docker and none other container will be there, so attack surface is reduced.

@bacongobbler I agree, external build service is best option, but no alternative in azure :(

What about deploy "build server" as a independent pod? so risk can be limited like in my gitlab-runner case

[bacongobbler]

Every instance of Draftd should be co-located with the docker daemon for a quick and speedy build. If we split them out as separate pods then they will not share the same network namespace and builds will generally take a not-insignificant performance hit, especially in clusters where nodes can be across different regions.

We could add optional node selectors to draft init so it can be scheduled on a separate worker node if that's what you'd like. That would be a happy medium for the time being.

See https://github.com/kubernetes/helm/pull/2557 for a similar approach.