What are the issues with Proposal D?

[chris-crone]

Reading through the proposals, I believe that Proposal D is directionally the one that is best. I think this because it does not require changing the data model of registries which means that getting adoption will be easier.

Reading the meeting notes from when this proposal was reviewed, the issues identified appear to be:

1. Updating an OCI index is racy
2. Indexes are limited to 4 MB
3. Filtering cannot be done server side
4. Garbage collection needs to be done client side
5. There are use cases where people want to keep the digest the same but add metadata (signing?)

I believe that (1) is a real issue that the OCI should engage with. I can see arguments for either being explicit that registries are expected to be eventually consistent and clients must take this into account or providing some kind of transactional API for index manipulation.

(2) says to me that we should push people not to inline large blobs into OCI indexes. 4 MB should be enough to link objects together by reference (😅 I hope this doesn't come back to haunt me!).

I'm not sure I understand (3). If one wants to query a reference, they just need to fetch (by digest or have kept) the root OCI index. Registries could provide reverse lookup APIs (digest -> related objects) which are arguably no worse than the APIs proposed in Proposal E.

I'm not sure I understand (4). I would argue the opposite: Only Proposal D provides a working server-side GC model.

I've heard that there are use cases for keeping the digest of an object constant (5). I have not seen a concrete example so I don't understand it. It is by design that digests change when the content changes and tags are the current method of working around this.

Are these the only issues with Proposal D? I'd be happy to discuss others and to get more clarity on (5).

[sudo-bmitch]

5. There are use cases where people want to keep the digest the same but add metadata (signing?)

I think the best way I can describe this one is a combination of "don't break existing workflows" (if possible) and "keep tooling modular".

Existing workflows to me includes image mirroring tooling that compares digests between a given tag on two registries to detect when the content is out of date, and also deployments that are pinned to a digest. The ideal for me is if the digest for an image changes when the image is different, and not when metadata that references that image changes. Especially when runtimes don't even see that metadata. E.g. if every day at 3am we redeploy every container in the cluster to track that the vulnerability scan has finished and attached new metadata, that feels like a bad UX.

Modular tooling means I can generate an SBOM, sign the image, and do other activities, without each activity needing to coordinate with the other tooling. We've seen that need with multi-platform image builds that want to perform a build in parallel on different hardware and push when each platform is done. When we talk about signing tools, I think there's a value in being able to sign an image without breaking the tool that was mirroring the image from upstream, or from needing to track previous states of the digest to follow audit of an image from build, to unit test, to SBOM generation, to CI, to signing, to production.

[chris-crone]

5. There are use cases where people want to keep the digest the same but add metadata (signing?)

I think the best way I can describe this one is a combination of "don't break existing workflows" (if possible) and "keep tooling modular".

Existing workflows to me includes image mirroring tooling that compares digests between a given tag on two registries to detect when the content is out of date, and also deployments that are pinned to a digest. The ideal for me is if the digest for an image changes when the image is different, and not when metadata that references that image changes. Especially when runtimes don't even see that metadata. E.g. if every day at 3am we redeploy every container in the cluster to track that the vulnerability scan has finished and attached new metadata, that feels like a bad UX.

Breaking the use cases down:

1. Syncing by tag across registries: I believe one would want a sync to be triggered when metadata is updated. Note it would only update the index and fetch the new metadata not the image manifests or layers.
2. Deployments pinned by digest and scan results: I think the issue here is actually that scan results are not the same as SBOM and signatures. Scan results are mutable data that is expected to change regularly. SBOMs and signatures should not change while an image is deployed. A mutable reference should be used for scan results (tag/API), and an immutable one for SBOM/signatures (OCI index).

For (2), tooling should pin to index digests so that they can then find all referenced objects or vendors could provide a reverse lookup API from manifest digest to parent index.

Also for (2), if a user has specified an image digest, they are explicitly telling the runtime to be dumb and do what it's told and implicitly saying that they have tooling/process to decide which exact image they want deployed. This tooling/process can have all the intelligence for checking scan results, signatures, whatever.

Modular tooling means I can generate an SBOM, sign the image, and do other activities, without each activity needing to coordinate with the other tooling. We've seen that need with multi-platform image builds that want to perform a build in parallel on different hardware and push when each platform is done.

This is the issue I raised as C1. Given that it has come up in the context of references and multi platform tooling, the OCI should seriously consider looking at it.

When we talk about signing tools, I think there's a value in being able to sign an image without breaking the tool that was mirroring the image from upstream, or from needing to track previous states of the digest to follow audit of an image from build, to unit test, to SBOM generation, to CI, to signing, to production.

I'm not sure that I understand this problem. Existing tooling that copies OCI objects between registries is not broken if we use existing OCI objects like indexes. It is broken if we add something like "refers" to manifests.

There are multiple ways to handle promotion flows. This could be by moving objects between repositories or using tags. Storing the attestations of the process that the artifact followed as part of the final artifact (signatures, SBOMs, etc.) would also be desirable for auditing purposes.

I really appreciate having the use cases because I think it makes the data model and UX issues more clear. Thank you @sudo-bmitch for adding them 😄

EDITS:

1. Added a thanks for the use cases, I disagree but appreciate them to clarify points!
2. Added a clarification to what pinning by digest means

[sudo-bmitch]

1. Syncing by tag across registries: I believe one would want a sync to be triggered when metadata is updated. Note it would only update the index and fetch the new metadata not the image manifests or layers.

That handles when the metadata changes come from upstream, but now when downstream adds their own metadata that they don't want overwritten. To give an example, a lot of environments I work with want their own keys to control what runs in their production environment, if they don't sign it, it doesn't get deployed. I don't think Docker Hub is updating the latest Alpine image to say ACME Rockets has signed this image, that's going to be internal to ACME Rockets, but now the digests are different between what ACME Rockets is deploying in their environment and what's upstream, so existing tooling that would track that they are running old images or synchronizing mirrors would need to be redesigned to handle the image signing tooling changing the copied image's digest.

2. Deployments pinned by digest and scan results: I think the issue here is actually that scan results are not the same as SBOM and signatures. Scan results are mutable data that is expected to change regularly. SBOMs and signatures should not change while an image is deployed. A mutable reference should be used for scan results (tag/API), and an immutable one for SBOM/signatures (OCI index).

It's a "signature-as-policy" style solution, where you're not necessarily attaching the scan results, but a signature saying this image is approved as of the latest scan, and we're communicating that with an image signature. There are pro's and con's to the approach. The biggest pro to me is each tool doesn't need to know how to communicate with every other tool, it just needs to be able to create or validate signatures, so the admission controller doesn't have to talk to aqua, snyk, anchore, etc. to get the approval from which ever tool the security team happens to be running. Not only does it give us modularity between components, but it avoids points of failure with the asynchronous signing/validation. The security scanning tool can be offline for as long as the signatures are valid minus the scan frequency without triggering any downtime.

Also for (2), if a user has specified an image digest, they are explicitly telling the runtime to be dumb and do what it's told and implicitly saying that they have tooling/process to decide which exact image they want deployed. This tooling/process can have all the intelligence for checking scan results, signatures, whatever.

That concerns me when thinking about how someone could use it to run unsigned/untrusted workloads in a cluster.

I'm not sure that I understand this problem. Existing tooling that copies OCI objects between registries is not broken if we use existing OCI objects like indexes.

Mutating the index with every change to the metadata is the concern for me. That digest coming out of the build has value to users, it's something they can track. So if adding metadata in the pipeline keeps changing that digest, I can see users upset that we've broken their workflow, especially if some of those changes come from periodic jobs completely outside of a pipeline.

It is broken if we add something like "refers" to manifests.

At present, I'm only aware of ECR restricting unknown fields, and I think they're on board with incorporating this if approved by OCI. Are there other registries we need to be worried about?

[dlorenc]

I agree the approach of using an index works in some cases where you either only have one signature or don't care about deploying by digest. That's fine though, because you can already accomplish this today without any changes to the registry.

Here's one concrete use case we have that doesn't work with this flow though, and why this group is exploring changes to the registry.

• We always reference all images and indexes by digest, from build time onward. We never use tags.
• We sign images at build time by attaching a signature to the image (or index) digest to attest to the build system they came from
• We sign images again before deployment indicating their approval to be deployed into that specific environment.
• Both signatures are required for the image to run.

We can't do this with the index based approach today without passing extra information out of band.

Similar use cases:

• multi-party approval (m of n signatures from different SREs on a team) required before deployment
• Attaching scan results and then a manual approval or attestation that those scan results meet policy
• Probably lots more!

The group did an extensive enumeration of required scenarios which were discussed, debated, and agreed on here before design work began. The designs here are being evaluated against the agreed on requirements and scenarios.

The index approach is definitely fine and the simplest approach for some scenarios, so IMO it's fine to use it there. But it just doesn't work for others. If it worked for all cases, or those other cases didn't matter, we wouldn't need to be here!

[chris-crone]

@sudo-bmitch and @dlorenc Thank you both for being patient and poking holes in Proposal D! I realize that I'm coming late to this working group and do not have all the context that has been shared in meetings– specifically on the signing use cases.

For signing, I can see how having detached signatures would be useful if there's a hard requirement on pinning to digests.

Re-reading Proposal C and the discussion about it (1, 2), I think that using an OCI index instead of a new type (as proposed in the 2022-03-29 meeting) would solve existing registries understanding that the objects are linked together and, as a new index would be pushed with the metadata, the initial image/index digest would not change. This does not solve for finding metadata from an image digest which would require a registry API or some other mechanism. Nested indexes are within spec but I agree that enforcing nesting depth would require some work.

My goal is the same as the working group's: To find a good solution to attach metadata to artifacts stored in registries. I strongly believe that the best path to this is requiring no or very few changes to existing structures and APIs. Any changes will take time to propagate through the ecosystem and delay getting this functionality into users' hands.

[tonistiigi]

We always reference all images and indexes by digest, from build time onward. We never use tags.

We sign images again before deployment indicating their approval to be deployed into that specific environment.

It somewhat seems logical that adding an additional signature would change the digest as well as the behavior or the image has changed (depends on the image definition of course). The more interesting part is what practical flows this digest change breaks. The design of all of these objects is that wrapping is cheap. Eg. for a docker pull/run workflow this doesn't really change anything, only top-level manifest is repulled on now previous image has the additional signature without user possibly even realizing it.

Your case seems to be that you already stored the digest, then resigned but after signing you only have access to the previous digest. In that case, isn't is conceptually the resigner's job to also notify that it has changed the identity of the image. Being able to detect that this identity has changed might actually be quite important as different components might need to react to this change and invalidate the previous decision they made based on the previous collection of signatures/attestations.

If the original unsigned digest is still the only digest that defines the object in your flow then registry could provide a way to find the objects who have (signed) descriptors pointing to specified digest. Basically it is similar to the refers idea but instead of unsigned digest pointing to a list of signatures/attestations, it is pointing to wrappers of itself. So in practical terms, an unsigned multi-arch image index would point to image index that has a signature and a descriptor with the unsigned digest. If registry is updated it could calculate these backreferences on its own, if it doesn't then a (opt-in) fallback tagging scheme could give access to updated signature from a tag that contains the unsigned digest string.

[imjasonh]

Thanks for your feedback @tonistiigi. I think it's worth settling once and for all whether we should consider it a requirement that "alpine" and "alpine signed by Alice" and "alpine signed by Bob" should be considered "the same image" (i.e., have the same digest), since either way we go I think that limits our options in useful ways. Prior to this discussion I think we've mostly agreed that they are the same image, and that Alice and Bob's signatures are attached outside of the image, thus not affecting its digest.

I'm not sure I understand the part about wrappers, is that to say that in the act of Alice signing an image, she creates an index manifest that contains a descriptor of her signature, and a descriptor of the (possibly multi-arch) image she signed? That seems like it will wreak havoc on container runtimes that, historically at least, don't handle nested manifests well at all, or at least consistently. And if Alice signs alpine then Bob signs her wrapper, you've got deep nesting, which gets messy fast. That's a big reason we've mostly avoided it. But again, perhaps I'm misunderstanding the point.

Re: reacting to a signature by possibly invalidating a previous decision; this isn't something we've really discussed I think. I'm struggling a bit to find a real use case for it, a policy that wants to only admit images Alice hasn't signed? That sounds a bit like revocation, which I believe we've also considered mostly out of scope.

In any case, I'm really glad to have your perspectives here, and I'd love to get more of them. If either of you (@chris-crone) can make the WG meeting on Tuesdays to discuss more, that would be awesome. If the time doesn't work I'm sure we can find a one-off or move it if there's interest.

[tonistiigi]

That seems like it will wreak havoc on container runtimes that, historically at least, don't handle nested manifests well at all,

Indeed, seems that not all runtimes handle the nested index 😢 . We can fix it but not having support for old versions is a problem. Need to think if there is a workaround. Posting the rest of my comment that I already wrote before testing this.

And if Alice signs alpine then Bob signs her wrapper, you've got deep nesting, which gets messy fast.

I guess from the data structures standpoint, unlimited nesting should be allowed like it is in plain indexes, but in practical terms, I thought it more like a two-step process.

Let's say the unsigned image index has digest sha256:abc, and we wrap it with a signer manifest.

sha256:bcd
{ ... 
  "manifests": [
    {"digest": "sha256:abc",}
    {"digest": "sha256:123", "annotations": {"ref": "sha256:abc", "signature": ...}} // or annotations in main descriptor, main thing is that all signatures are in this file
  ]
}

Now we want to turn it into an image with 2 signatures. Generate a new root object.

sha256:cde
{ ... 
  "manifests": [
    {"digest": "sha256:abc"}
    {"digest": "sha256:123", "annotations": {"ref": "sha256:abc", "signature": ...}}
    {"digest": "sha256:234", "annotations": {"ref": "sha256:abc", "signature": ...}} 
  ]
}

New root replaces the old one. Now the definition of the image is only image with 2 signatures.

If you have a use case where you stored unsigned digest sha256:abc and want to find its latest signatures, then (by the protocol) the process that signed the image again also updated the tag. :sha256-abc.<hash>.sig that now points to :sha256:cde.

Re: reacting to a signature by possibly invalidating a previous decision; this isn't something we've really discussed I think. I'm struggling a bit to find a real use case for it, a policy that wants to only admit images Alice hasn't signed?

You could even use it as a deploy trigger. Image does not deploy because it doesn't have the deploy key, once it is added the image is valid.

Apart from signatures, the same mechanism would be used for attestations where the examples are probably more basic. Eg. SBOM being uploaded or updated would mean that the image's vulnerability report changes etc.

that "alpine" and "alpine signed by Alice" and "alpine signed by Bob" should be considered "the same image" (i.e., have the same digest), since either way we go I think that limits our options in useful ways.

I think Docker use cases are more aligned with image being self-contained, with signature(s), attestations etc. You can build and sign it without pushing it into a registry first. You can copy it like a regular image, load/save etc. You can check attestations offline. You can also understand the history of images that may have had the same container bits but different sets or signatures or attestations.

For example, consider a case where I build an image with provenance attestation. Later I build a new image that generates the same container artifacts. The provenance for the two builds was not the same: different commands can generate the same artifact, and provenance also logs the exact times of the build. The same is true for signatures that are timestamped and signed with a certificate only valid for 10 min. I'd argue the correct way to think about this case is not that I have an image and this image happens to have N provenance and signatures, but that I built 2 images that happened to have the same container bits. Both of them have one signature(even if the subject of the signature is the same) and one attestation.

This might be something completely different, but if I run cosign tree gcr.io/distroless/static-debian11, it is signed 34 times with 34 certificates that look very similar except for their different 10-minute validity periods. If I run cosign verify gcr.io/distroless/static-debian11 it takes 2 minutes to complete with the current implementation. I'm sure it can be optimized by some fraction, but if this is the same case, then it does show the problem. My assumption is that 34 builds have built the same container bits. Only one of these signatures belongs to gcr.io/distroless/static-debian11; the rest are either other images or previous builds of static-debian11 that don't matter anymore.

If we want 90% of images to be signed and not 1%, then we also need to think about the validation performance. In case of Docker that would mean validating every docker pull (without opt-in) while keeping performance change unnoticeable. This can't happen if we check an evergrowing list of random signatures. In the image index example above, I would actually prefer all signatures to be inlined in the root manifest to avoid the extra requests(attestations can't be inlined of course).

[dlorenc]

This might be something completely different, but if I run cosign tree gcr.io/distroless/static-debian11, it is signed 34 times with 34 certificates that look very similar except for their different 10-minute validity periods.

This one is definitely a pathological case. The image is bit for bit reproducible but gets built automatically in a git repository with dozens of other images that are also reproducible on every change. So each build will have one image that's different but a bunch that are the same.

[dlorenc]

This can't happen if we check an evergrowing list of random signatures. In the image index example above, I would actually prefer all signatures to be inlined in the root manifest to avoid the extra requests(attestations can't be inlined of course).

The recently merged data field will help here. Then we can do both - append signatures but still fetch them all in one request.

[imjasonh]

Pathological or not, that's a case that's clearly possible, and I agree we should make sure performance isn't terrible in case it happens. Reproducible builds should hopefully become more prevalent, and we shouldn't punish them for it 😄. FWIW cosign verify gcr.io/distroless/static-debian11 takes ~45s for me, not 2+ minutes, but I agree that's still too long.

I don't think we should encourage or assume clients will have to validate every attached signature every time they pull an image. Instead, users should configure a policy about what images they want to be able to pull, i.e., which signatures they care about or not, and have only matching signatures be verified. Attaching a bad signature to an image shouldn't make it unpullable, for instance, if the policy doesn't care about that signature.

This is roughly what the cosign policy controller (neé cosigned) does on Kubernetes, and I think it's worth exploring how to bring that behavior to docker pull too, so policies can be written and distributed to block untrusted containers from running on dev machines too. This policy can then also extend to things like SBOMs or attestations (or any future attached thing the community comes up with), without having to pull and inspect every bit of every attached thing before pulling the image.

[jdolitsky]

Closing since the outcome of these discussions resulted in Proposal F