Open mumoshu opened 6 years ago
Hey @mumoshu! Nice to see you here. :)
I think you've summarized the issue and the landscape quite well.
As you mentioned, there are definitely some solutions out there where users are injecting secrets into the build environment without actually committing those to the image itself, usually relying on an external endpoint. In my mind, the true limiting factor here is docker build
and the Dockerfile syntax. Because each step in a Dockerfile is committed as a layer, you cannot inject secrets into the build runtime or else it will either become part of the metadata of the container image, or committed to a layer. Both of which are not good for the reasons you stated above: they become embedded into the image.
Technically speaking, we really just need Docker for building a container image and a registry so the kubelets can pull that image. There's nothing that we truly need that ties us to Docker other than the familiarity that the open source ecosystem now has with Docker, as well as the current Kubernetes ecosystem that relies heavily on Docker tooling to build, run and host container images.
In the past few months there has been work in the open source community to provide alternative build engines like https://github.com/genuinetools/img and https://github.com/projectatomic/buildah, each of which provide users a way to manually play around with the files in the COW filesystem before they are committed as a layer. I really like these tools because they greatly simplify the process around building container images, as well as the fact that they're just CLI tools rather than full blown client/server daemons running as a service. They're lighter weight and therefore should better serve our own use cases.
I know that's a little hand-wavy, but here's the tangible solution of injecting secrets using buildah as an example:
#!/bin/bash
cid=$(buildah from ubuntu)
# mount the container's root filesystem and inject a secret
rootfs=$(buildah mount $cid)
touch $rootfs/mysecretfile
buildah unmount $cid
# do secret stuff with the secret
buildah run $cid -- gem install mysecretgem
# remove the secret from the container filesystem
rootfs=$(buildah mount $cid)
rm $rootfs/mysecretfile
buildah unmount $cid
# Commit this container
buildah commit $cid mysecretcontainer
Existing docker-build alternatives like habitus does support this feature.
habitus definitely seems like it's on the right track. I really like that their build.yml provides a workflow around a Dockerfile's build steps. It allows users to provide additional functionality around the process of building container images, which is why I really like the buildah approach described above as well.
I think there's definitely an opportunity here to provide a process where users can safely copy secrets into the container filesystem before running commands, then removing them before committing the layer. I don't think Draft will necessarily be the one to tackle this problem head on, but if there is a container builder that does happen to tackle this problem, then we could replace the Docker engine with that container builder.
Would it be good to improve draft to support it, too?
Just to clarify, what sort of proposal did you have in mind? Are you thinking of exposing an HTTP server during docker build
which Dockerfiles could fetch secrets from? I'm a little concerned that this would mean that the build would only work on Draft... or at least not without a great amount of effort to stand up your own local "secret service" (see what I did there? :D). I'd definitely be interested to hear your thoughts on this!
Thanks for letting me pick your brain on this! Definitely an interesting conversation, so thank you for your well thought out proposal.
Hi @bacongobbler,
First of all, I basically agree to every point you've noted. Thank you very much for the detailed responses!
I do love img and buildah, and even s2i and imagebuilder for their own sweet-spots and delight futures.
Btw, I've summarized about these tools in https://github.com/moby/moby/pull/36443#issuecomment-369845823 in regard to secret management. PTAL if you have some time.
if there is a container builder that does happen to tackle this problem, then we could replace the Docker engine with that container builder.
Absolutely. I basically agree with this direction.
Are you thinking of exposing an HTTP server during docker build which Dockerfiles could fetch secrets from?
If you mean that draftd does it, no - I fully agree to your concern about build-time dependency on draft. That should be avoided! Back to the description of this issue, my expected prerequisite is to use separate tool for production docker images.
or at least not without a great amount of effort to stand up your own local "secret service"
I don't want to do that either! It would be a needless engineering effort which will become unnecessary once e.g. buildkit matures enough to replace docker-daemon's backend. Or img matures. Root-less image build is the future!!(Sry too emotional but I'd really love that, really)
Given all those points, what I propose is basically keep draftd loosely-coupled to docker-build alternatives.
More concretely, what I propose TODAY is enhancing draft.toml
, so that we can specify a docker-build
-compatible command via draft.toml
, which is then used instead of the current docker-client based builder based on ImageBuild.
Assumed project-structure:
mycustombuilders/habitus-docker-build
(The path is not fixed. It's user's responsibility to decide where to put thisDockerrfile.draftd
:
FROM alpine:3.5
COPY rootfs /
EXPOSE 44135
ADD mycustombuilders/habitus-docker-build /path/to/my/alt-docker-builders/bin/habitus-docker-build
ENTRYPOINT ["/bin/draftd"]
CMD ["start"]
draft.toml
:
[environments]
[environments.development]
name = "my-ruby-app"
namespace = "default"
wait = false
watch = false
watch_delay = 2
build_command = /path/to/my/alt-docker-builders/bin/habitus-docker-build
The habitus-docker-build
would look like:
#!/usr/bin/env bash
set -e
# This decrypts the AWS-KMS-encrypted envvars contained in the local work-tree, may or may not be committed within the git repo
# I personally prefer it to be committed for gitops, but thats an another story.
sops -d build-secrets.enc > .envrc
# .envrc may contains something like:
# export HABITUS_MY_SECRET=mysecret_in_cleartext
# This corresponds to MY_SECRET available thru habitus' secrets server
# Not sure I'd really go with direnv but its essence persists
direnv allow .
RAND_PASS=$(randomly_generated_password_for_secret_sharing) # Not a secret itself
# -authentication-secret-server: enables basic-auth to secrets server
# -secrets: enables the secrets server
# -password: sets the password for basic-auth
# -build SECRET_PASS=$RAND_PASS: passes the password for basic-auth to be used from within docker-build
# -d $DRAFT_BUILD_CONTEXT: Send build context from where draftd keeps it
habitus -d $DRAFT_BUILD_CONTEXT -build SECRET_PASS=$RAND_PASS -env IMAGE=$DRAFT_IMAGE -authentication-secret-server -secrets -password=$RAND_PASS
habitus.yml
:
build:
version: 2016-03-14
steps:
step1:
# _env(IMAGE) is replaced to the value of DRAFT_IMAGE envvar, according to `-env IMAGE=$DRAFT_IMAGE` above
name: _env(IMAGE)
# This references $DRAFT_BUILD_CONTEXT/Dockerfile
# as specified by `habitus -d $DRAFT_BUILD_CONTEXT`
dockerfile: Dockerfile
secrets:
my_env_secret:
type: env
# Corresponds to the HABITUS_MY_SECRET sourced via direnv above
# The prefix "HABITUS_" is required by habitus for security reason
value: MY_SECRET
And in the Dockerfile, I'd get the secret by accessing habitus' secrets server with $SECRET_PASS
.
There are three important pre-requisites:
/path/to/my/alt-docker-builders/bin/habitus-docker-build
must exist within custom draftd docker image. Or maybe you can mount an extra PV containing the command to draftd pod, but a custom docker image seems a bit more straight-forward to me.habitus
binary to be within the custom draftd image, because habitus-docker-build
script runs ithabitus-docker-build
.
Cons:
Pros:
Now that #573 has been merged, we might be able to chew off alternative build runtimes as an experiment. @mumoshu let me know if you'd be interested in taking the lead on this feature!
I would love to explore the proper way to swap out build engines and/or build services in this space.
First up: ACR Build in #691. Now that we have written a container image builder interface, it should be as simple as contributing new interfaces to work with alternative tools.
A lengthy context about the secret management problem in docker: https://github.com/moby/moby/issues/13490
In nutshell, I need draft to support safely passing "secrets" from the local worktree to the remote build context.
A secret can be said to be safely passed when and only when the secret is not exposed to malicious user and not remained in any part of a resulting docker image.
A build secret is required when e.g. you're building a Ruby app depends on a private gem hosted in a private GitHub repository. In this specific case, you need a ssh key to download the private gem.
// Note that this is how
bundle install
orgem
, the defacto dependency manager or the package manager for ruby works.Non-safe cases include:
--build-arg
to pass secrets: NG.docker inspect
.ADD
instruction to pass secrets: NG.docker save $image | tar -x -C out && find out/ -type file | xargs -n 1 grep $secret
rm
ed secret disappear from the resulting docker image. But squashing results in docker-layer-caching useless, no incremental build. You end up re-building the whole docker image from scratch after eachdraft up
...--
Just for clarification, I'm not going to exploit the draft-built docker image itself for production-use. I'll build the production one on my CI/CD system.
So, my concern here is solely about development, which I believe what draft stands for. I don't want to expose my precious secret even in dev env!
Proposal
Today, the most general work-around for the problem seems to be utilizing a temporary tcp/http server last within the single docker build which holds/serves secrets to the build context. You can basically utilize the server like
curl $endpoint/$secret -o ~/.ssh/id_rsa
&& bundle install && rm ~/.ssh/id_rsa`, so that you can pass the secret without polluting the docker image metadata nor the ADD instruction.Existing docker-build alternatives like habitus does support this feature.
Would it be good to improve draft to support it, too?