Safely passing "secrets" to docker build contexts

[mumoshu]

A lengthy context about the secret management problem in docker: https://github.com/moby/moby/issues/13490

In nutshell, I need draft to support safely passing "secrets" from the local worktree to the remote build context.

A secret can be said to be safely passed when and only when the secret is not exposed to malicious user and not remained in any part of a resulting docker image.

A build secret is required when e.g. you're building a Ruby app depends on a private gem hosted in a private GitHub repository. In this specific case, you need a ssh key to download the private gem.

// Note that this is how bundle install or gem, the defacto dependency manager or the package manager for ruby works.

Non-safe cases include:

• Used docker's --build-arg to pass secrets: NG.
  • This results in the docker image has the secrets inside metadata can be seen by e.g. running docker inspect.
• Used ADD instruction to pass secrets: NG.
  • This results in the docker image contains the secret in one of its layers, can be queried by e.g. running docker save $image | tar -x -C out && find out/ -type file | xargs -n 1 grep $secret
  • You can indeed squash the image to a single layer so that ADDed and then rmed secret disappear from the resulting docker image. But squashing results in docker-layer-caching useless, no incremental build. You end up re-building the whole docker image from scratch after each draft up...

--

Just for clarification, I'm not going to exploit the draft-built docker image itself for production-use. I'll build the production one on my CI/CD system.

So, my concern here is solely about development, which I believe what draft stands for. I don't want to expose my precious secret even in dev env!

Proposal

Today, the most general work-around for the problem seems to be utilizing a temporary tcp/http server last within the single docker build which holds/serves secrets to the build context. You can basically utilize the server like curl $endpoint/$secret -o ~/.ssh/id_rsa && bundle install && rm ~/.ssh/id_rsa`, so that you can pass the secret without polluting the docker image metadata nor the ADD instruction.

Existing docker-build alternatives like habitus does support this feature.

Would it be good to improve draft to support it, too?

[bacongobbler]

Hey @mumoshu! Nice to see you here. :)

I think you've summarized the issue and the landscape quite well.

As you mentioned, there are definitely some solutions out there where users are injecting secrets into the build environment without actually committing those to the image itself, usually relying on an external endpoint. In my mind, the true limiting factor here is docker build and the Dockerfile syntax. Because each step in a Dockerfile is committed as a layer, you cannot inject secrets into the build runtime or else it will either become part of the metadata of the container image, or committed to a layer. Both of which are not good for the reasons you stated above: they become embedded into the image.

Technically speaking, we really just need Docker for building a container image and a registry so the kubelets can pull that image. There's nothing that we truly need that ties us to Docker other than the familiarity that the open source ecosystem now has with Docker, as well as the current Kubernetes ecosystem that relies heavily on Docker tooling to build, run and host container images.

In the past few months there has been work in the open source community to provide alternative build engines like https://github.com/genuinetools/img and https://github.com/projectatomic/buildah, each of which provide users a way to manually play around with the files in the COW filesystem before they are committed as a layer. I really like these tools because they greatly simplify the process around building container images, as well as the fact that they're just CLI tools rather than full blown client/server daemons running as a service. They're lighter weight and therefore should better serve our own use cases.

I know that's a little hand-wavy, but here's the tangible solution of injecting secrets using buildah as an example:

#!/bin/bash

cid=$(buildah from ubuntu)

# mount the container's root filesystem and inject a secret
rootfs=$(buildah mount $cid)
touch $rootfs/mysecretfile
buildah unmount $cid

# do secret stuff with the secret
buildah run $cid -- gem install mysecretgem

# remove the secret from the container filesystem
rootfs=$(buildah mount $cid)
rm $rootfs/mysecretfile
buildah unmount $cid

# Commit this container
buildah commit $cid mysecretcontainer

Existing docker-build alternatives like habitus does support this feature.

habitus definitely seems like it's on the right track. I really like that their build.yml provides a workflow around a Dockerfile's build steps. It allows users to provide additional functionality around the process of building container images, which is why I really like the buildah approach described above as well.

I think there's definitely an opportunity here to provide a process where users can safely copy secrets into the container filesystem before running commands, then removing them before committing the layer. I don't think Draft will necessarily be the one to tackle this problem head on, but if there is a container builder that does happen to tackle this problem, then we could replace the Docker engine with that container builder.

Would it be good to improve draft to support it, too?

Just to clarify, what sort of proposal did you have in mind? Are you thinking of exposing an HTTP server during docker build which Dockerfiles could fetch secrets from? I'm a little concerned that this would mean that the build would only work on Draft... or at least not without a great amount of effort to stand up your own local "secret service" (see what I did there? :D). I'd definitely be interested to hear your thoughts on this!

Thanks for letting me pick your brain on this! Definitely an interesting conversation, so thank you for your well thought out proposal.

[mumoshu]

Hi @bacongobbler,

First of all, I basically agree to every point you've noted. Thank you very much for the detailed responses!

I do love img and buildah, and even s2i and imagebuilder for their own sweet-spots and delight futures.

Btw, I've summarized about these tools in https://github.com/moby/moby/pull/36443#issuecomment-369845823 in regard to secret management. PTAL if you have some time.

if there is a container builder that does happen to tackle this problem, then we could replace the Docker engine with that container builder.

Absolutely. I basically agree with this direction.

Are you thinking of exposing an HTTP server during docker build which Dockerfiles could fetch secrets from?

If you mean that draftd does it, no - I fully agree to your concern about build-time dependency on draft. That should be avoided! Back to the description of this issue, my expected prerequisite is to use separate tool for production docker images.

or at least not without a great amount of effort to stand up your own local "secret service"

I don't want to do that either! It would be a needless engineering effort which will become unnecessary once e.g. buildkit matures enough to replace docker-daemon's backend. Or img matures. Root-less image build is the future!!(Sry too emotional but I'd really love that, really)

Given all those points, what I propose is basically keep draftd loosely-coupled to docker-build alternatives.

More concretely, what I propose TODAY is enhancing draft.toml, so that we can specify a docker-build-compatible command via draft.toml, which is then used instead of the current docker-client based builder based on ImageBuild.

Assumed project-structure:

• draft.toml
• Dockerfile
• Dockerfile.draftd (May or may not be within the project. Could be managed by an infrastructure team
• build.yml (for habitus)
• .envrc.enc (encrypted version of direnv file )
• .sops.yml (mozilla/sops config)
• mycustombuilders/habitus-docker-build (The path is not fixed. It's user's responsibility to decide where to put this

Dockerrfile.draftd:

FROM alpine:3.5

COPY rootfs /

EXPOSE 44135

ADD mycustombuilders/habitus-docker-build /path/to/my/alt-docker-builders/bin/habitus-docker-build

ENTRYPOINT ["/bin/draftd"]
CMD ["start"]

draft.toml:

[environments]
  [environments.development]
    name = "my-ruby-app"
    namespace = "default"
    wait = false
    watch = false
    watch_delay = 2
    build_command = /path/to/my/alt-docker-builders/bin/habitus-docker-build

The habitus-docker-build would look like:

#!/usr/bin/env bash

set -e

# This decrypts the AWS-KMS-encrypted envvars contained in the local work-tree, may or may not be committed within the git repo
# I personally prefer it to be committed for gitops, but thats an another story.
sops -d build-secrets.enc > .envrc

# .envrc may contains something like:
# export HABITUS_MY_SECRET=mysecret_in_cleartext
# This corresponds to MY_SECRET available thru habitus' secrets server

# Not sure I'd really go with direnv but its essence persists
direnv allow .

RAND_PASS=$(randomly_generated_password_for_secret_sharing) # Not a secret itself

# -authentication-secret-server: enables basic-auth to secrets server
# -secrets: enables the secrets server
# -password: sets the password for basic-auth
# -build SECRET_PASS=$RAND_PASS: passes the password for basic-auth to be used from within docker-build
# -d $DRAFT_BUILD_CONTEXT: Send build context from where draftd keeps it
habitus -d $DRAFT_BUILD_CONTEXT -build SECRET_PASS=$RAND_PASS -env IMAGE=$DRAFT_IMAGE -authentication-secret-server -secrets -password=$RAND_PASS

habitus.yml:

build:
  version: 2016-03-14
  steps:
    step1:
      # _env(IMAGE) is replaced to the value of DRAFT_IMAGE envvar, according to `-env IMAGE=$DRAFT_IMAGE` above
      name: _env(IMAGE)
      # This references $DRAFT_BUILD_CONTEXT/Dockerfile
      # as specified by `habitus -d $DRAFT_BUILD_CONTEXT`
      dockerfile: Dockerfile
      secrets:
        my_env_secret:
          type: env
          # Corresponds to the HABITUS_MY_SECRET sourced via direnv above
          # The prefix "HABITUS_" is required by habitus for security reason
          value: MY_SECRET

And in the Dockerfile, I'd get the secret by accessing habitus' secrets server with $SECRET_PASS.

There are three important pre-requisites:

• /path/to/my/alt-docker-builders/bin/habitus-docker-build must exist within custom draftd docker image. Or maybe you can mount an extra PV containing the command to draftd pod, but a custom docker image seems a bit more straight-forward to me.
• habitus-specific: requires the habitus binary to be within the custom draftd image, because habitus-docker-build script runs it
• Prior to docker-build, you need to give draftd pod an appropriate permission to decrypt your secret while running e.g. habitus-docker-build.
  • For cloudproviders supports KMS, this requires you to configure some kind of server-authn on node or pod-level. For example, you'd annotate your draftd pod with kube2iam/kiam annotation to attach IAM role with KMS access. For other cases, you'd need to pass a private key(like gpg key?) for decryption via PV.

---

Cons:

• Configuration would be a mess.
• Documentation must be challenging

Pros:

• Change to draftd is minimum. Only extracting a image-builder interface, refactoring the existing logic to form the default bulder impl, and implementing a "command" builder impl.
• draftd will be kept less-coupled to any docker-build alternative

[bacongobbler]

Now that #573 has been merged, we might be able to chew off alternative build runtimes as an experiment. @mumoshu let me know if you'd be interested in taking the lead on this feature!

[squillace]

I would love to explore the proper way to swap out build engines and/or build services in this space.

[bacongobbler]

First up: ACR Build in #691. Now that we have written a container image builder interface, it should be as simple as contributing new interfaces to work with alternative tools.