Prepopulating cache with kaniko warmer still causes problems with manifest mediaType

[hojo0590]

Actual behavior Thank you big time for fixing #1836.
We use kaniko in a gitlab CI job and I noticed that when building a Docker image (at least the one below) with pre populating the cache with kaniko's warmer the push to the gitlab registry will fail. But once the warmer is not used any more the push to gitlab's registry works.

Error with the warmer:

error pushing image: failed to push to destination gitlab....com:4567/user/repo/warmertest:latest: PUT https://gitlab.....com:4567/v2/user/repo/warmertest/manifests/new2: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

Expected behavior Using kaniko warmer should not change the build result (when only warming with the base image).

To Reproduce Steps to reproduce the behavior: Everything of the following in done inside Docker

docker run -it --rm --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:v1.15.0-debug

Failing behaviour:

# add login credential for gitlab registry beforehand
mkdir /kcache
/kaniko/warmer --cache-dir=/kcache --image=condaforge/mambaforge:23.3.1-0
/kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --cache=true --cache-copy-layers=false --cache-dir=/kcache --destination gitlab.....com:4567/user/repo/warmertest:latest --verbosity=info --cleanup

Working behaviour:

# add login credential for gitlab registry beforehand
mkdir /kcache
# do not warm with base image
/kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --cache=true --cache-copy-layers=false --cache-dir=/kcache --destination gitlab.....com:4567/user/repo/warmertest:latest --verbosity=info --cleanup

Note: Corporate URL and username are removed

Additional Information

• Dockerfile

  FROM condaforge/mambaforge:23.3.1-0
  RUN mamba update --all -q -y

• Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor@sha256:9a798741047d3761221100f586ea9fa8a3f09957176630faeb77923894b7355a gcr.io/kaniko-project/executor:v1.15.0-debug

  Triage Notes for the Maintainers

  Description	Yes/No
  Please check if this a new feature you are proposing	- [ ]
  Please check if the build works in docker but not in kaniko	- [ ]
  Please check if this error is seen when you use --cache flag	- [x]
  Please check if your dockerfile is a multistage dockerfile	- [ ]

[hojo0590]

Thanks @aaron-prindle and @loganprice - looking forward to that fix being released!

[hojo0590]

Whilst we had no further issues in kaniko version 1.16 it does not work in 1.20.1, where it fails with the initially reported error

error pushing image: failed to push to destination gitlab....com:4567/user/repo/warmertest:latest: PUT https://gitlab.....com:4567/v2/user/repo/warmertest/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

While upgrading our kaniko version we stumbled upon this regression - (haven't tested the versions along the way 1.17 and so on). Can you affirm and reopen the issue?

[laurivosandi]

I can confirm this is still issue with recent warmer, executor (1.23.1) and GitLab 17.0

 MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

[guimorg]

Same here. I've already opened an issue about this here: https://github.com/GoogleContainerTools/kaniko/issues/2883

This impacts the usage of container images in some AWS Services (like SageMaker) since OCI mediaType is not supported.