[V1.31.0] Error: checking signature : Docker references with both a tag and digest are currently not supported

[Nemric]

What happened?

Hi, I did the upgrade from K8S & Cri-O from v1.30.4 to v1.31.0 I don't know if this issue is really related to Cri-O but I can't install ingress-nginx because of this behavior taken from kubectl get events when applying https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/baremetal/deploy.yaml

pod/ingress-nginx-admission-create-s6x86         Error: checking signature of "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3": creating docker:// reference for "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3": Docker references with both a tag and digest are currently not supported

Crio logs looks like : Sep 13 09:10:35 Marie crio[2048]: time="2024-09-13 09:10:35.088965171Z" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:ce263a8653f9cdabdabaf36ae064b3e52b5240e6fac90663ad3b8f3a9bcef242,RepoTags:[],RepoDigests:[registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:1b792367d0e1350ee869b15f851d9e4de17db10f33fadaef628db3e6457aa012 registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3],Size_:55821706,Uid:&Int64Value{Value:65532,},Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=e2ce05ae-4b61-4ab6-b7f1-8b940feb277c name=/runtime.v1.ImageService/ImageStatus

and kubelet logs looks like : Sep 13 09:10:35 Marie podman[2232]: E0913 09:10:35.103521 2247 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"patch\" with CreateContainerError: \"checking signature of \\\"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3\\\": creating docker:// reference for \\\"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3\\\": Docker references with both a tag and digest are currently not supported\"" pod="ingress-nginx/ingress-nginx-admission-patch-d2hvn" podUID="ae27ff5e-a382-4a6b-a942-2ed1e4975914"

What did you expect to happen?

Ingress-nginx installation to run well like on v1.30.4

How can we reproduce it (as minimally and precisely as possible)?

On a K8S cluster v1.31.0 with cri-o, calico, openelb try to apply the yaml file depending of your environment from : https://kubernetes.github.io/ingress-nginx/deploy/

Anything else we need to know?

My cluster is a baremetal one with Fedora CoreOS 40.20240825.3.0 nodes

CRI-O and Kubernetes version

```console $ crio --version crio version 1.31.0 GitCommit: ac758bb6183ef69cd47c663eb916953217a97fb3 GitCommitDate: 2024-09-09T22:22:50Z GitTreeState: dirty BuildDate: 1970-01-01T00:00:00Z GoVersion: go1.22.5 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: static netgo osusergo exclude_graphdriver_btrfs seccomp apparmor selinux exclude_graphdriver_devicemapper LDFlags: unknown SeccompEnabled: true AppArmorEnabled: false ``` ```console $ kubectl version --output=json { "clientVersion": { "major": "1", "minor": "31", "gitVersion": "v1.31.0", "gitCommit": "9edcffcde5595e8a5b1a35f88c421764e575afce", "gitTreeState": "clean", "buildDate": "2024-08-13T07:37:34Z", "goVersion": "go1.22.5", "compiler": "gc", "platform": "linux/amd64" }, "kustomizeVersion": "v5.4.2", "serverVersion": { "major": "1", "minor": "31", "gitVersion": "v1.31.0", "gitCommit": "9edcffcde5595e8a5b1a35f88c421764e575afce", "gitTreeState": "clean", "buildDate": "2024-08-13T07:28:49Z", "goVersion": "go1.22.5", "compiler": "gc", "platform": "linux/amd64" } } ```

OS version

```console # On Linux: $ cat /etc/os-release NAME="Fedora Linux" VERSION="40.20240825.3.0 (CoreOS)" ID=fedora VERSION_ID=40 VERSION_CODENAME="" PLATFORM_ID="platform:f40" PRETTY_NAME="Fedora CoreOS 40.20240825.3.0" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:40" HOME_URL="https://getfedora.org/coreos/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/" SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/" BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=40 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=40 SUPPORT_END=2025-05-13 VARIANT="CoreOS" VARIANT_ID=coreos OSTREE_VERSION='40.20240825.3.0' $ uname -a Linux Pierre 6.10.6-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 19 14:09:30 UTC 2024 x86_64 GNU/Linux ```

Additional environment details (AWS, VirtualBox, physical, etc.)

Physical K8S cluster

[kwilczynski]

This one is either a regression if we take the following into account:

• https://github.com/cri-o/cri-o/issues/2351

Or a legitimate expected behaviour per:

• https://github.com/containers/image/issues/1736

@mtrmac, your thoughts?

[kwilczynski]

List of changes for reference:

• https://github.com/cri-o/cri-o/compare/v1.30.4...v1.31.0

[mtrmac]

From the c/image side, such inputs are intentionally not supported, and never were.

CRI-O did impose the Docker-like semantics, but it’s a bit hidden inside CandidatesForPotentiallyShortImageName.

8212 has no equivalent of that on userSpecifiedImageName, so that is probably now failing. Sure, imposing the same “Strip the tag from ambiguous image references” policy on the container-creation path would be consistent. (Please factor it out into a single shared implementation.)

[Nemric]

On a testing node I went back to Cri-o v1.30.5 and I was able to run the the job ... What should be the risks for now if I run a K8S 1.31 with cri-o 1.30 ?

[haircommander]

1.30 cri-o and 1.31 kubelet should be fine. you may not be able to use the newest alpha features but everything should work.

@mtrmac for my understanding: the docker behavior is just ignoring the tag if it's both tagged and digested?

[mtrmac]

CRI-O’s current behavior is https://github.com/cri-o/cri-o/blob/548e0706d55fdc3956c3c7a94a26d530f16d5b14/internal/storage/image.go#L969-L983 .

[Foritus]

Specifically for ingress-nginx, you can kubectl edit the deployment for the controller and admission pods and remove the tags. e.g.

image: registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce

becomes

image: registry.k8s.io/ingress-nginx/controller@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce

and then things work for now :)

[Nemric]

I did try removing digest without success and gave up modifying the file ! thanks for the right thing to remove ;)

[Jeansen]

For me, it does not work, yet. Installing a k8s cluster from scratch with kubernetes and cri-o 1.30 works like a charm. That is, I create my k8s cluster AND install Cilium overlay network. No Problem there. Doing the same with 1.31 does not work. I can create a cluster and until then, all seems fine. But as soon as I try to install Cilium, all pods have the OPs problem.

@haircommander When will there be a new patch release?

[Ivan-Tamayo]

@Jeansen Same problem with cilium and cri-o 1.31.

I had to install cilium via helm and a lot of config parameters: "--set image.useDigest=false"

But when I run the cilium connectivity test, same issue, all container creation fail because of the digest

[Nemric]

Specifically for ingress-nginx, you can kubectl edit the deployment for the controller and admission pods and remove the tags. e.g.

image: registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce

becomes

image: registry.k8s.io/ingress-nginx/controller@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce

and then things work for now :)

You have to edit the image link in your manifests to only provide digest as said in this quoted comment

[Jeansen]

@Nemric, @Ivan-Tamayo Thanks for your feedback. As far as I can see, there already is a merged PR, but no release yet. I'll stick with 1.30 until there is a new release that hopefully remedies this problem.

[kariya-mitsuru]

The kubernetes documentation states that if both tag and digest are specified, tag is ignored and digest is used.

https://kubernetes.io/docs/concepts/containers/images/

Wouldn't cri-o be better off adapting to that behavior?

(Personally, I would prefer error or warning, so it would be best if there were options for error, warning, ignore tag, and ignore digest, but I doubt there is that much demand...)

[Jeansen]

@kariya-mitsuru Great finding! So, I agree cri-o should adapt this behavior (and optionally issue a warning).

[Jeansen]

Latest release 1.31.1 fixed the issue for me.