search

GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.85k stars 1.44k forks source link

push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error #2423

Open luxurine opened 1 year ago

luxurine commented 1 year ago

Actual behavior When push builded image, with base image(ubuntu:20.04) cached by warmer, will get error:

error pushing image: failed to push to destination luxurine/counter:latest: PUT https://index.docker.io/v2/luxurine/counter/manifests/latest: 
MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' 
not 'application/vnd.oci.image.manifest.v1+json'

(If there is no cache for base image, image push is ok)

Expected behavior Image should be pushed successfully.

Preliminary analysis

It looks like ubuntu:20.04 use "mediaType": "application/vnd.oci.image.manifest.v1+json", but dockerhub expect application/vnd.docker.distribution.manifest.v2+json.

The strange thing is, if build without base image cache, pushing image is just ok, I guess reasons could be:

  1. if build without base image cache, kaniko will do automatic transformation
  2. if build with base image cache, as cache dir is readonly in kaniko executor's view, it will build on top of the cache

    kaniko can cache images in a local directory that can be volume mounted into the kaniko pod. To do so, the cache must first be populated, as it is read-only. We provide a kaniko cache warming image at gcr.io/kaniko-project/warmer

Some proposal：

To Reproduce Steps to reproduce the behavior:

  1. prepare manifests 
    
λ tree .
.
├── Dockerfile
├── bin
│   └── counter
└── config.json

2 directories, 5 files


2. warm base image
```text
$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:v1.9.1 --cache-dir=/workspace/cache --image="ubuntu:20.04"
INFO[0000] Retrieving image manifest ubuntu:20.04
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io
  1. build and push image 
    $ docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.9.1-debug --cache=true --cache-dir=/workspace/cache --dockerfile=Dockerfile --destination=luxurine/counter:latest
INFO[0002] Retrieving image manifest ubuntu:20.04
INFO[0002] Retrieving image ubuntu:20.04 from registry index.docker.io
INFO[0005] Found sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d in local cache
INFO[0005] Found manifest at /workspace/cache/sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d.json
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest ubuntu:20.04
INFO[0005] Returning cached image manifest
INFO[0005] Found sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d in local cache
INFO[0005] Found manifest at /workspace/cache/sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d.json
INFO[0005] Executing 0 build triggers
INFO[0005] Building stage 'ubuntu:20.04' [idx: '0', base-idx: '-1']
INFO[0005] Unpacking rootfs as cmd COPY bin/ /app requires it.
INFO[0007] COPY bin/ /app
INFO[0007] Taking snapshot of files...
INFO[0007] WORKDIR /app
INFO[0007] Cmd: workdir
INFO[0007] Changed working directory to /app
INFO[0007] No files changed in this command, skipping snapshotting.
INFO[0007] ENTRYPOINT ["/app/counter"]
INFO[0007] No files changed in this command, skipping snapshotting.
INFO[0007] Pushing image to luxurine/counter:latest
error pushing image: failed to push to destination luxurine/counter:latest: PUT https://index.docker.io/v2/luxurine/counter/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

Additional Information

COPY bin/ /app

WORKDIR /app

ENTRYPOINT ["/app/counter"]


 - Kaniko Image: gcr.io/kaniko-project/executor:v1.9.1-debug

ref:
- https://docs.docker.com/registry/spec/manifest-v2-2/
- https://github.com/opencontainers/image-spec/blob/main/manifest.md
- https://github.com/moby/moby/issues/44898
jamgregory commented 1 year ago

I've just experienced a similar issue building a local development image and pushing it to our own GitLab Registry instance (albeit without the useful error message - I just got MANIFEST_INVALID: manifest invalid; map[]

I'm not sure how you worked around this @luxurine, but my solution was to build the Ubuntu base image myself:

# Based on: https://git.launchpad.net/cloud-images/+oci/ubuntu-base/tree/?h=focal-20.04
FROM scratch
ADD source/ubuntu-focal-oci-amd64-root.tar.gz /
SeWieland commented 1 year ago

I think this is related to https://github.com/GoogleContainerTools/kaniko/issues/1836

aaron-prindle commented 1 year ago

I believe this is a dupe of #1836 which was recently fixed and in the latest kaniko release (v1.15.0). Closing, feel free to re-open if there is more to address here

luxurine commented 1 year ago

I've just tried kaniko-project/executor:v1.17.0 again, still the same problem. Could you please reopen this issue? @aaron-prindle

manifest

# Dockfile
FROM ubuntu:20.04 # with OCI format
WORKDIR /app

step 1) warmup image

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:v1.17.0 --cache-dir=/workspace/cache --image=ubuntu:20.04

# output
INFO[0000] Retrieving image manifest ubuntu:20.04
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io

2) build and push

$ docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.17.0-debug --cache=true --cache-dir=/workspace/cache --dockerfile=Dockerfile --destination=luxurine/demo:latest --verbosity=debug

# output
INFO[0002] Retrieving image manifest ubuntu:20.04
INFO[0002] Retrieving image ubuntu:20.04 from registry index.docker.io
INFO[0006] Found sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d in local cache
INFO[0006] Found manifest at /workspace/cache/sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d.json
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest ubuntu:20.04
INFO[0006] Returning cached image manifest
INFO[0006] Found sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d in local cache
INFO[0006] Found manifest at /workspace/cache/sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d.json
INFO[0006] Executing 0 build triggers
...
INFO[0006] Pushing image to luxurine/demo:latest
error pushing image: failed to push to destination luxurine/demo:latest: PUT https://index.docker.io/v2/luxurine/demo/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'
tanadeau commented 11 months ago

I've also seen this with 1.18.0 and 1.19.0. Seems possibly related to newer issue #2883. We're using GitLab Container Registry and not AWS ECR.

hojo0590 commented 8 months ago

maybe related/a regression of #2713?

mhio commented 3 months ago

Ran into this issue on executor-v1.23.2 and recent warmer-latest (doesn't report a version) to a gitlab v16.11.5 container registry.

luxurine commented 3 months ago

Ran into this issue on executor-v1.23.2 and recent warmer-latest (doesn't report a version) to a gitlab v16.11.5 container registry.

@mhio here is a temporary fix you can try:

  1. fork kaniko repo
  2. update func func (c *cachedImage) Manifest() (*v1.Manifest, error) in pkg/cache/cache.go and return tabball's manifest by default.
  3. build image

[!WARNING]
target image is always in docker format

func (c *cachedImage) Manifest() (*v1.Manifest, error) {
    if c.mfst == nil {
        return c.Image.Manifest()
    }
    return c.mfst, nil
}

->

func (c *cachedImage) Manifest() (*v1.Manifest, error) {
    mfst, err := c.Image.Manifest()
    if mfst != nil {
        return mfst, err
    }
    return c.mfst, nil
}
mhio commented 3 months ago

Thanks @luxurine we'll give that a try, disabling the warmed cache was the quick workaround.