Define immutable tags

[AkihiroSuda]

A tag MAY be recreated to refer to a diferrent content. However, a tag with -immutable suffix, such as 1.2.3-immutable, SHOULD not be recreated with a different content. (Not "MUST not", because a registry implementation might not be aware of the history of the tag)

Closes #241

[sudo-bmitch]

Should a tag delete be allowed, or deleting the manifest that the tag points to? If it is, could the tag be deleted and recreated to point to different content? And if it's not, is this a potential attack where an attacker with short term access to a repository is able to force that repository to permanently host their content?

I'm also conflicted if we should be defining a standard tag naming scheme vs only defining the http response to various API calls when an immutable tag is encountered. The latter would leave the decision of when to enforce immutability up to individual registries. E.g. many users may only want their release tag with a version to be immutable, without needing to change their versioning tag to match this.

[imjasonh]

One difficulty with this convention is that it's not clear whether the registry will enforce it -- today, no registries enforce it, and rolling out support will inevitably be slow, especially to all the on-prem registries. Some registries will just never support it.

The only thing worse than mutable tags is the illusion of immutable tags. 😆

Perhaps we should have the format of an immutable tag be something that's not a currently valid tag, so it's clear to users that any registry that accepts the tag will definitely enforce that it's immutable.

Throwing out ideas: a tag can't start with - today, so maybe we could say if a tag starts with - it's immutable? Or some other character, maybe # or %? (@ is probably not ideal, . seems too easy to miss)

• reg.example.com/repo/app:-v1.2.3
• reg.example.com/repo/app:#v1.2.3
• reg.example.com/repo/app:%v1.2.3

The downside of course is that clients would have to update their image reference validation logic to accept and understand these new semantics, and rollout of that would also be slow. At least support for the feature would be unambiguous.

[amouat]

@sudo-bmitch deletion is an interesting one. I would say no, but for legal reasons, you probably have to allow some form of delete. Instead of normal deletion, once "deleted" the tag could return an error. Which means you have to keep track of old tags. Also, it's arguably not immutable...

[sudo-bmitch]

Something I've wanted to do for a while is improve the tag listing API, outputting a map of tags to descriptors instead of only an array of tag names. Perhaps adding a way for registries to indicate a tag is immutable in that list would be useful. This would be more of a long term option since updating that API would take time.

[AkihiroSuda]

@sudo-bmitch

Should a tag delete be allowed, or deleting the manifest that the tag points to? If it is, could the tag be deleted and recreated to point to different content? And if it's not, is this a potential attack where an attacker with short term access to a repository is able to force that repository to permanently host their content?

Deletion is allowed, but a deleted tag shouldn't be repurposed to refer to another content. As discussed below, this convention is just for hinting clients. A registry server implementation may still accept mutating -immutable tags.

I'm also conflicted if we should be defining a standard tag naming scheme vs only defining the http response to various API calls when an immutable tag is encountered. The latter would leave the decision of when to enforce immutability up to individual registries. E.g. many users may only want their release tag with a version to be immutable, without needing to change their versioning tag to match this.

Immutability has to be known ahead of calling the HTTP API, so it has to be encoded in the tag strings.

---

@imjasonh

One difficulty with this convention is that it's not clear whether the registry will enforce it -- today, no registries enforce it, and rolling out support will inevitably be slow, especially to all the on-prem registries. Some registries will just never support it.

Nothing will be enforced by this spec. The -immutable suffix is just for providing a hint to client implementations.

E.g., a future version of docker build can be implemented to raise an error during processing FROM golang:x.y.z-immutable , when the digest of golang:x.y.z-immutable is different from the known digest recorded in Dockerfile.sum: https://github.com/moby/buildkit/issues/2794 .

But it will never raise such an error for images that do not have -immutable suffix, for compatibility reason.

Throwing out ideas: a tag can't start with - today, so maybe we could say if a tag starts with - it's immutable? Or some other character, maybe # or %? (@ is probably not ideal, . seems too easy to miss)

Looks too cryptic 😅 And also incompatibility with existing client implementations.

[AkihiroSuda]

Updated PR for rewording

[imjasonh]

A registry server implementation may still accept mutating -immutable tags.

I think this is a fundamental flaw with this approach. If you want to enforce that -immutable tags mean something to your team, you already can. You can write scripts and hooks and proxies to alert/block on changes to -immutable tags. But when the spec recommends it as the way to signal immutability, that may or may not be enforced by clients or registries, I think it only makes the situation worse.

E.g., a future version of docker build can be implemented to raise an error during processing FROM golang:x.y.z-immutable , when the digest of golang:x.y.z-immutable is different from the known digest recorded in Dockerfile.sum: moby/buildkit#2794 .

Sure, but a future version of docker build could also be implemented to understand that golang:%x.y.z means it's immutable, and would have the added benefit of the registry knowing it too. A registry that doesn't know about immutability would unambiguously fail to accept that ref at all.

I'm not saying it's ideal, it's definitely a large undertaking for both registries and clients to rollout support (and again, some never will!), but if the goal is immutable image refs, I don't see any other way to safely and unambiguously support that.

[AkihiroSuda]

that may or may not be enforced by clients or registries

The immutability isn't really expected to be enforceable by the registry, as the registry nodes may have a distributed database that does not guarantee serializable consistency.

And even if the registry claims to be capable of enforcing immutability, the client should't trust that claim.

Sure, but a future version of docker build could also be implemented to understand that golang:%x.y.z means it's immutable, and would have the added benefit of the registry knowing it too. A registry that doesn't know about immutability would unambiguously fail to accept that ref at all.

Registries do have conformance bugs. Some of the current registries may already accept golang:%x.y.z without support for immutability.

I updated the PR to let the GET /v2/_extensions/list endpoint return a JSON document like {"org.opencontainerd.distribution-spec/immutable-tags": true}. A client may refuse to pushing an immutable tag to a registry that does not support this "org.opencontainerd.distribution-spec/immutable-tags" extension. Same applies to pulling an immutable tag, too.

I'm not saying it's ideal, it's definitely a large undertaking for both registries and clients to rollout support (and again, some never will!), but if the goal is immutable image refs, I don't see any other way to safely and unambiguously support that.

Rolling out the new spec will probably take more than 3 or 5 years :confused: So I still prefer to retain compatibility with the current spec.

[imjasonh]

The immutability isn't really expected to be enforceable by the registry, as the registry nodes may have a distributed database that does not guarantee serializable consistency.

And even if the registry claims to be capable of enforcing immutability, the client should't trust that claim.

Sure, but I'd say that becomes an issue between the user and their registry vendor at that point. Same as if a registry claims to reliably persist data, and doesn't. If a registry doesn't (or can't) reliably enforce immutability, it shouldn't advertise that it does.

Registries do have conformance bugs. Some of the current registries may already accept golang:%x.y.z without support for immutability.

That's true, but I'd prefer to have data to suggest that, rather than rely on speculation when discussing this change.

I think the extension idea is a step in the right direction. The extension mechanism might even mean this is something that doesn't need to be specified in this exact spec, but only in some "official" extension defined... somewhere?

[sudo-bmitch]

I've been thinking over this one a bit. Is there a need for the client to know the tag is immutable without contacting the registry? I.e. does it need to be in the syntax of the tag, or can it be discovered by an API? If discovery by API is enough, then I'd lean towards not defining the tag schema, and instead having the extension API to identify which tags are immutable, update the tag listing API, or even do both. That gives the flexibility of defining which tags can be immutable back to the user and registry operators.

[AkihiroSuda]

Is there a need for the client to know the tag is immutable without contacting the registry? I.e. does it need to be in the syntax of the tag, or can it be discovered by an API?

Yes, this should be in the syntax of the tag just like example.com/foo:1.2.3-immutable@sha256:deadbeef...

Otherwise we will have to update a bunch of other specs to support pulling an image tag that does not match the foreknown digest.

e.g., - `docker (pull|create|run)` CLI spec and the corresponding Docker daemon REST API ```console $ docker run --immutable-tag example.com/foo:1.2.3@sha256:deadbeef... ``` - `docker-compose.yaml` ```yaml services: foo: image: "example.com/foo:1.2.3@sha256:deadbeef..." immutable_tag: true ``` - Dockerfile ```dockerfile FROM --immutable-tag example.com/foo:1.2.3@sha256:deadbeef... ``` - Kubernetes Pod yaml ```yaml apiVersion: v1 kind: Pod metadata: name: foo annotations: # Eventually this will be moved from an annotation to the regular Pod spec immutable-tags.security.alpha.kubernetes.io/foo: true spec: containers: - name: foo image: "example.com/foo:1.2.3@sha256:deadbeef..." ``` - Trivy CLI ```console $ trivy image --immutable-tag example.com/foo:1.2.3@sha256:deadbeef... ``` - ...

It should be noted that even if the immutability is discoverable via the OCI Distribution Spec API, the CLIs and the YAMLs have to have "immutable-tag: true" field as the OCI registry server might be compromised and might return false information about immutability.

[AkihiroSuda]

Implementing the current proposal for containerd:

• https://github.com/containerd/containerd/pull/7074

The tag part in example.com/hello-world:42.0@sha256:deadbeef had been just ignored. The tag was never verified with the specified digest because the tag was considered to be mutable. Even the existence of the tag was never verified, and it was causing a lot of confusions like issue https://github.com/containerd/containerd/issues/6450.

This commit enables verification of the tag string that matches the ImmutableTagMatcher function. The matcher functions defaults to HasSuffix(tag, "-immutable") || HasSuffix(tag, "_immutable") || HasSuffix(tag, ".immutable").

e.g., example.com/hello-world:42-immutable@sha256:deadbeef is a ref string with an immutable tag.

When the tag matches, the resolver always verifies the existence of the tag, and also verifies the digest if the digest is present in the ref string.

i.e., pulling example.com/hello-world:42-immutable@sha256:deadbeef now fails if the tag 42-immutable does not exist or exists with an unexpected digest on example.com/hello-world.

The matcher convention is being submitted to the OCI Distribution Spec in opencontainers/distribution-spec#320.

Fixes https://github.com/containerd/containerd/issues/6450

[sudo-bmitch]

Thinking through the containerd example, I think there are some other options to consider:

Option 1:

1. containerd pulls the manifest for the tag
2. if there is an annotation or some other metadata in the manifest that indicates it's immutable, and the digest doesn't match, fail
3. if the digest doesn't match and there's no immutable flag in the manifest, pull the digest

This is probably the easiest to implement since we just need to define something like an annotation. Security isn't perfect since a malicious registry could still change what the tag points to. The other option, since we may not want to pull the manifest twice (e.g. rate limit concerns):

Option 2:

1. pull the manifest by digest
2. query the tag using a head request
3. if the digest mismatches, look for a header or other extension from the registry indicating the tag should have been immutable

This isn't great since we'd be stuck waiting for the registry to support a new header. The last option that I like the best is to not handle this with immutability. Instead, defer to signing solutions:

Option 3:

1. Signer would sign the image along with a list of canonical names/tags that may be used by the image (I'm avoiding saying we sign a specific name since images can be copied and have multiple tags, e.g. staging -> prod and v1 + v1.2 + v1.2.10)
2. Runtime pulls the digest, and then validates the signature for the digest
3. Signature verification also checks if the name/tag matches one of the canonical names from the signer, and the signer is trusted for the given canonical name (they'd also want a policy in the signing tool allowing the image to be copied to another location)

[kzys]

Late to the party. I agree with @sudo-bmitch -

I'm also conflicted if we should be defining a standard tag naming scheme vs only defining the http response to various API calls when an immutable tag is encountered. The latter would leave the decision of when to enforce immutability up to individual registries. E.g. many users may only want their release tag with a version to be immutable, without needing to change their versioning tag to match this.

Tags have been opaque strings. Introducing semantics like that is technically a breaking change. People shouldn't name mutable tags as "xxx-immutable", but we really cannot control how they have used containers.

[AkihiroSuda]

Thank you @sudo-bmitch

The background of my proposal is to let docker build raise an error during processing FROM golang:x.y.z-immutable , when the digest of golang:x.y.z-immutable is different from the foreknown digest recorded in a database like Dockerfile.pin (Initially called Dockerfile.sum):

• https://github.com/moby/buildkit/issues/2816

The option 1 might be acceptable if we can store that immutability annotation in the digest database, but I still think it is easier to use when the immutability is encoded in the image reference string.

Option 4: encode immutability into the “query” part of the image ref string, e.g., docker.io/library/golang:1.23.4?immutable=1. (Cc @dmcgowan https://github.com/containerd/containerd/pull/7074) The ref string spec isn’t currently formalized in the OCI spec though.

[kzys]

Regarding the pinning use case, shouldn't the immutability enforced by the end user who write Dockefile and Dockerfile.pin?

Let's say I'm writing Dockerfile and want to enforce immutability regarding images, I don't want to wait registries and image vendors to support the immutable notation. Clients such as Docker Engine could enforce that without waiting them.

[AkihiroSuda]

Let's say I'm writing Dockerfile and want to enforce immutability regarding images, I don't want to wait registries and image vendors to support the immutable notation. Clients such as Docker Engine could enforce that without waiting them.

Option 4 doesn’t need any modification to the reg and the images, but clients should share the same reference spec. WG for establishing the proper reference spec (not just for immutability) is now being discussed in the OCI Slack.

[flavianmissi]

It should be noted that even if the immutability is discoverable via the OCI Distribution Spec API, the CLIs and the YAMLs have to have "immutable-tag: true" field as the OCI registry server might be compromised and might return false information about immutability.

Maybe a dumb question but why can't we trust the registry server about tag immutability? Registries could deny pushes to tags it knows as immutable, ensuring they stay that way.

If this is a security problem, then @sudo-bmitch's option 3 seems better suited.

If this is about pinning images on a Dockerfile then why not refer to their digest instead?

I'm late to the discussion so probably missing something.

[AkihiroSuda]

why can't we trust the registry server about tag immutability?

Because the registry server can be compromised

If this is about pinning images on a Dockerfile then why not refer to their digest instead?

This is actually expected to be used with the digest, either manually or automatically with human-friendly digest locking utilities such as Dockerfile.pin. Pulling golang:x.y.z-immutable@sha256:deadbeef will fail either if the tag does not exist or exists with an unexpected digest.

[flavianmissi]

Because the registry server can be compromised

Would you mind elaborating? What kinds of risks do you see?

This is actually expected to be used with the digest, either manually or automatically with human-friendly digest locking utilities such as Dockerfile.pin. Pulling golang:x.y.z-immutable@sha256:deadbeef will fail either if the tag does not exist or exists with an unexpected digest.

I wonder if we really need immutability to accomplish this? Does matching the digest of a manifest pulled with image:tag@digest against the provided digest accomplish the same thing? I.e pulling golang:latest@sha256:a2b3 and getting anything other than the manifest with digest sha256:a2b3 causes an error.

Immutability could then be tackled separately, and used together with the above referencing "style".

To clarify my position: I like the idea of tag immutability - it's something we have discussed in Quay and want to eventually support - I'm just trying to understand this.

[sudo-bmitch]

It may help to define the various scenarios for immutable tags:

1. Ensuring a tag can only ever point to one digest. From a registry administration view, this is more policy than a guarantee, since there's usually ways to alter a server, even with this sort of policy (e.g. administrator disables immutability, replace the tag, reenable immutability).
2. Detect if a tag+digest points to an unknown, possibly malicious digest. E.g. someone creates a deployment for registry.example.com/app:v2@sha256:1234... but sha:1234... actually points to v1 and user's don't realize the wrong image is being run.
3. Detect if a tag+digest has an altered tag and the digest should be trusted.

And those scenarios have various use cases:

1. Pinning deployments for a runtime.
2. Pinning dependencies for image builds.
3. Auditing deployments that may no longer exist in production or on a registry.

What other scenarios/use cases am I missing? Drawing these out and identifying the possible attacker scenarios may help identify the different options we have to implementing a soltuion.

[flavianmissi]

that's very helpful @sudo-bmitch, thanks!