Automatically surface artifacts produced by a build

[imjasonh]

When a Build produces a Docker image in a registry, (or any artifact, like a JAR in object storage, etc.) that information is not surfaced anywhere in the Build itself. This information can be useful in a chain-of-custody type scenario, when you need to determine the provenance of an image, how exactly it was built, and from what source. Since this ties to security, we need to make sure it's hard to forge or modify build provenance of an image.

It would be useful to have this information automatically collected by the Build system, possibly using an egress proxy that inspects network traffic leaving the Build's Pod. This could watch for traffic that looks like an image push, and update the Build resource with the image reference and digest that was pushed. It could also push this provenance information to a service like Grafeas, which was built for exactly this kind of audit information.

[julz]

This makes me wonder: why does the Build take an explicit source but rely on a step to deal with output? Wouldn't it make more sense (and nicely solve this issue) to have an output section which would specify a docker registry?

Thinking out loud, presumably the main issue with that would be that we'd need to specify the path and disk layout of the image inside the shared workspace (or another shared volume) in order for our implicit final init container to know how to grab it and push it?

[imjasonh]

That's a good question. I think when we talk about "outputs" what we're actually talking about is two types of things (at least) that it's useful to know your build produced, and they're each handled separately.

For container images, it's hard in Knative Build today because we expect steps themselves to push those, and we don't get much visibility into whether/where they pushed. If the build wrote an image to the docker daemon, we could push those images at the end of the build and have a solid record that the images were pushed, because we were responsible for it. (This is what GCB does, build configs have an images field that tells us which images in the local Docker daemon to push and record). On-cluster, however, you don't want to mount the Docker daemon, so only unprivileged builders (buildah, kaniko, jib, buildpacks, etc.) are recommended. When one of these builders pushes an image, we don't have any visibility into the details of that push today. I think an egress proxy slurping all outgoing traffic for image pushes would help, and I'm open to other ideas.

For other artifacts (jars, tars, zips, debs, logs, etc.), we could do something like this. GCB also supports non-container artifacts, where the build specifies a pattern of files in the /workspace to upload to a specified location in Cloud Storage, and after the build steps complete we run one more container with /workspace mounted that uploads files matching the patterns and records information about the upload (final location, object generation, file digests). We could support this use case in Knative Build in pretty much the same fashion today, by adding an artifacts field to the build spec, and allow users to specify patterns of files to upload to their preferred cloud provider. That's basically what you've described, and if users want it, we can definitely design and implement that.

Basically the essential difference is that for unprivileged image builders, container images are constructed and pushed entirely during a build step's execution, and don't necessarily write any persistent data to the /workspace for the Build system to inspect (some could, I think buildah does?).

Does that distinction help clarify the problem, and my thinking on possible solutions?

[julz]

Yeah, thanks for the response! I think that basically confirms my guess that the basic problem is how the builder can organise for an image to be in a format where we're able to upload it. I definitely don't think you'd want to expose the docker socket in to a step or have a privileged builder, but actually once you realise that you don't want to have privileged builders on a cluster I think the problem might simplify a bit.

Given you are using an unprivileged builder, you almost by definition aren't getting any benefit of layered filesystems or any magic like that (because you'd need privileges to use any of that). That means your final 'image' is just files on disk. Which means (it seems, potentially), you could just build on to a volume.

As a blurry straw-man, we already have a shared /workspace volume mounted in to each step, if we had an /output volume the unprivileged builders could - instead of building and uploading an image directly - build and save it in to /output in OCI format (or list the path to the OCI image in artefacts, as you say). Then we'd deal with upload ourselves generically, which would also let us update the job metadata etc. I worry a fair amount about egress slurping as it seems dangerously close to magic, and dangerously dependent on our system keeping in sync with external APIs (it's pretty sad if the docker registry API changes and my CI automation breaks).

[imjasonh]

I like the idea of builder writing images to volumes for us to push ourselves later, I'm just not sure why they would if they can push directly themselves already today, but let's assume we motivate them since most of them are...us :smile: .

For builders that use go-containerregistry it probably wouldn't be hard to add a switch that uses tarball.Write instead of remote.Write -- buildpacks seem to already support this switch for remote-vs-daemon.

If we go that route, I think we'd have to find some way to refactor tarball.Write to not write layers locally that can be remotely mounted when we do the eventual push. This means that we can skip downloading those layers entirely, and some builders (FTL, Jib) rely on this optimization for fast incremental image builds. Related: https://github.com/google/go-containerregistry/pull/209

@mattmoor @jonjohnsonjr @dlorenc

[jonjohnsonjr]

Just thinking out loud:

How much do we trust the builders? If we need to change them to write to disk, we might as well allow them to just self-report what they pushed.

E.g. we could just scrape logs for something in the form: gcr.io/jonjohnson-test/ubuntu:latest: digest: sha256:958eaeb7e33e6c4f68f7fef69b35ca178c7f5fb0dd40db7b44a8b9eb692b9bc5 size: 1357

Docker and ggcr already output this, so it would be easy to do but pretty brittle :/

I don't think there's enough information in the normal docker save format tarball manifest to actually do a full push if we don't write the layers to the tarball. There is enough information to do the final manifest PUT, though... we could do something, but it's definitely going to feel nonstandard and strange :) I'm not sure how to make this compose nicely with existing tools.

/shrug, needs more 🤔

[imjasonh]

I don't think we should trust the builders to report what they pushed. An earlier alternative which was discussed was to scrape step status text and just report that, but I think the danger is too high that some system would explicitly trust that output, and that a malicious user could either omit reporting that it pushed a bad image, or report that it pushed a good image when it pushed a bad one.

By having builders write images locally and having the build do the push itself at the end, we can at least verify that the image we claim to have pushed was pushed, since we did it.

The egress proxy approach is I think useful because it wouldn't be possible to trick it into reporting false pushes, though it might be possible to confuse it into missing a push done by the builder.

Agreed, needs more :thinking:

[jonjohnsonjr]

What if we have the builders do all the pushing, since they have all the context required to do it efficiently (without trying to serialize that context to disk), but have them also write out the registry manifest they (would have) pushed. If we then do the final (perhaps additional) PUT of that manifest, we can guarantee that it exists and that we have write access to push it.

Re: egress proxy, I doubt we can completely avoid missing some pushes, but SGTM if you think it would work and if I don't have to maintain it 😅

[mattmoor]

Let's consider three options for provenance:

• Builder X puts an image into the docker daemon, post-step pushes from the daemon.
• Builder Y produces a docker save tarball, post-step pushes the tarball.
• Builder Z publishes the image and reports what it published.

I'd argue that the only thing X and Y give you that Z does not is confirmation that the image was written by the identity the build is running as (this can often also be checked if outputs are declared, and if all credentials and protocols are known by the system).

Let's suppose hypothetically, that I don't trust these:

• Builder X maliciously pulls an image and tags it instead of building, you push the bad image.
• Builder Y does the same, but saves it to a tarball, you push the bad image.
• Builder Z writes out that it published a bad image.

Network jails make this (and everything) harder, but not impossible. I still need to trust that the builder doesn't have an image tarball embedded in its filesystem.

I think that trust in the Builder (at some level) is required for provenance to work. I don't think that necessarily means I need to trust all Builders, but I think that provenance should sign the outputs in the context of the builder SHAs, and that policy can judge provenance assertions based on builder reputation.

I'm surprised @vbatts isn't here :)

[evankanderson]

A secondary item in favor of storing metadata in a local output volume is that you can standardize much of the post-processing (e.g. if you want to also submit the image to a scanning service prior to upload, you can do that in a post-step without needing to teach each builder how to do so). The drawback is that you end up with 0-2 extra sets of disk I/O for the produced image; depending on the work done this could be large or tiny, though I'd expect that the large ones (e.g. kaniko) will have done a lot of other I/O work anyway.

On Thu, Aug 9, 2018 at 3:53 PM Matt Moore notifications@github.com wrote:

Let's consider three options for provenance:

• Builder X puts an image into the docker daemon, post-step pushes from the daemon.
• Builder Y produces a docker save tarball, post-step pushes the tarball.
• Builder Z publishes the image and reports what it published.

I'd argue that the only thing X and Y give you that Z does not is confirmation that the image was written by the identity the build is running as (this can often also be checked if outputs are declared, and if all credentials and protocols are known by the system).

Let's suppose hypothetically, that I don't trust these:

• Builder X maliciously pulls an image and tags it instead of building, you push the bad image.
• Builder Y does the same, but saves it to a tarball, you push the bad image.
• Builder Z writes out that it published a bad image.

Network jails make this (and everything) harder, but not impossible. I still need to trust that the builder doesn't have an image tarball embedded in its filesystem.

I think that trust in the Builder (at some level) is required for provenance to work. I don't think that necessarily means I need to trust all Builders, but I think that provenance should sign the outputs in the context of the builder SHAs, and that policy can judge provenance assertions based on builder reputation.

I'm surprised @vbatts https://github.com/vbatts isn't here :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/knative/build/issues/215#issuecomment-411923129, or mute the thread https://github.com/notifications/unsubscribe-auth/AHlyN58Rlzfg-mhYA1l-Nm_RHRyhHbHAks5uPL1OgaJpZM4U7cKg .

-- Evan Anderson argent@google.com

[julz]

As @evankanderson says the nice thing about using a local OCI image is you can share the upload steps between multiple builds, and standardise post-processing. It also seems a bit more declarative for tooling to be able to see in the spec what will be uploaded by the build in a standard way (rather than implicitly assuming a particular step is the upload step). The other maybe nice advantage of using local OCI images as the step outputs is that steps could interact with the image produced by previous steps, for example running vulnerability checks on the image from a previous step, or adding extra layers (I guess this is really just a special case of the above, though).

@vbatts would be way way better than me to say this for sure, but fwiw I think the avoiding-double-IO problem with having the step build a local image is pretty soluble with the OCI format. The normal path (with the potential double disk IO problem) is to have the step produce an image in OCI format with all the needed layer blobs referenced by the manifest present in the blobs directory, which it's easy to then push to a registry. To avoid an extra copy of layers that are already in the registry I think the step can just produce an OCI image whose manifest references blobs that it doesn't bother to put in the blobs folder: as long as those are already in the registry, the uploader can skip uploading them and never notice they're not there. (@vbatts feel very free to say if this is crazy! :))

[mattmoor]

I'm not sure this is the most productive forum for this, perhaps we should add to next week's Build WG agenda?

[vbatts]

@ImJasonH @julz I think having an ./output/ volume is a fine approach for a general builder use-case. Though that is the implicit feeling I got from an $IMAGE variable. (and for any serverless/FaaS use-case that seems a more minimal path, though would need a basic vetting that the image exposes some port, etc).

@jonjohnsonjr said

I don't think there's enough information in the normal docker save format tarball manifest to actually do a full push if we don't write the layers to the tarball. There is enough information to do the final manifest PUT, though... we could do something, but it's definitely going to feel nonstandard and strange :) I'm not sure how to make this compose nicely with existing tools.

There ought to be enough there. There is an age-old effort to have a docker save option for the OCI image-layout https://github.com/moby/moby/pull/33355 which would be standardized except for pushback. Then provenance metadata could stashed in annotations and the digest of the image calculated. No scraping of logs needed. Very deterministic.

This is the original and primary use-case for skopeo is to copy from an OCI (or docker save) layout up to a remote registry, etc.

@ImJasonH said

his information can be useful in a chain-of-custody type scenario, when you need to determine the provenance of an image, how exactly it was built, and from what source.

To this point, I was wonder the other day about a boilerplate plumbing that could enable builders (buildah, img, buildkit, etc) to have information for stashing in image annotations/LABELS like the git commit of source built from, digest of the BUILDER_IMAGE, etc. This info ought not be quite so ephemeral in that it is stashed in the signable artifact.

@mattmoor yea this github issues make for sloppy design conversations.

lol as I'm writing replies here and continuing to read down the issue, I'm seeing others have the same feedback :+1:

Also also wik, good to see you again @julz ;-)

[jchesterpivotal]

This could watch for traffic that looks like an image push, and update the Build resource with the image reference and digest that was pushed. ... I think an egress proxy slurping all outgoing traffic for image pushes would help, and I'm open to other ideas.

Wouldn't this require the proxy to MITM the HTTPS connection to the registry? If so that makes me profoundly uncomfortable -- it would be a giant shining bullseye for attackers looking to perform supply chain attacks.

I can imagine having private repos as write-through proxies, which will suit a lot of enterprise folks (many of whom already do this). But it won't fly with folks who want to use dockerhub, GCR etc directly, because you will not (I hope) be able to present valid certificates in their place.

[jonjohnsonjr]

@vbatts ack, the OCI image layout would work perfectly for a sparsely populated tarball 👍

[cppforlife]

i'm curious about how this plays with remote builders. for example, imagine if google builder would be implemented via a container (any reason why it shouldn't be decoupled?) that just talks to the remote apis after it uploaded some source code. since image isnt locally built there wouldnt be anything to inspect (unless it's downloaded, but that's expensive). i see several approaches:

• add new api for bringing remote builders (knative build will pick up metadata from the builder via some contract)
• keep on adding remote builders into build repo (similar to current google build impl)
• use in-cluster builder and have remote builder shipped as steps (image) and somehow pick up info without having access to the image (some metadata file?)
• have remote builders be separate controllers looking for matching builds -- controllers will kick off the build and populate metadata by updating crd