Open loshz opened 4 years ago
We'd love to improve our documentation around running the agent on Kubernetes, thanks for offering to help out.
As you point out, the security issues are a hurdle. I suspect one of the reasons we haven't managed to publish more official documentation yet is the best approach isn't necessarily obvious.
You're right that mounting the host docker socket has security implications (and it may not always be an option, as some kube hosts experiment with containerd instead of docker). However, you're also correct that the docker-in-docker approach isn't particularly secure either - privileged: true
means any workload run by the agent is able to escape the pod and access the host as root.
One option is we document a way to run the agent on kubernetes without docker. That's a secure configuration, however it requires builds of a very different shape (using something like kaniko to build images, docker-compose can't be used to start a database, etc).
Is docker-in-docker the approach that's working bets for you?
I absolutely agree- running this sort of setup in k8s can be a bit of a minefield and there's definitely not a one solution to fit all!
I would suggest that the new docs start with potential warnings and informs the user of any assumptions made, that way we can keep it as generic as possible.
DinD has been working perfectly for me. No issues so far.
I recently started migrating from a competitor CI to Buildkite and so far I am loving the service! I am running my own agents in a self-hosted Kubernetes cluster with no problems so far.
Even though I am running on AWS, I loosely followed the GCloud docs and got things set up fairly quickly. However, I have a couple of "concerns" about using the approach the docs recommend.
Firstly, from my experience, it is not recommended to mount the host Docker socket (Docker-on-Docker) into a pod as this could potentially expose the following security issues:
kubelet
and could cause scheduling/resource issues.buildkite-agent
with a privileged security context could again mean code ran via a build could potentially have root access on the host machine.I am by no means a security expert, but after using Kubernetes in production for several years I have come to learn that some of the above practices could be dangerous.
That being said, I would like to propose a new, Kubernetes specific, set of installation docs- and I would be more than happy to hep write them alongside anyone else!
I have the following Docker-in-Docker (dind) setup running in one of my clusters:
Happy to provide any further information!