GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.95k stars 1.45k forks source link

kaniko cant push to quay ent on prem #400

Closed MansM closed 5 years ago

MansM commented 6 years ago

Actual behavior Kaniko cant push to quay

Expected behavior kaniko able to push to quay To Reproduce Steps to reproduce the behavior: as gitlab runner

Additional Information

INFO[0014] Taking snapshot of full filesystem...        
2018/10/17 13:27:37 pushed blob sha256:b8d3e972c2e2a5ce50d644aac7c6c6a9ea301e9992ea4678a89cf49ab29a4c22
2018/10/17 13:27:39 pushed blob sha256:bfde11b989fdb1e010026dc77bbd13641c7f654a3183855d927f198d2bcc0dd9
2018/10/17 13:27:43 pushed blob sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
error pushing image: failed to push to destination privatequay/kaniko/testimage:latest: MANIFEST_INVALID: "manifest invalid"
ERROR: Job failed: command terminated with exit code 1

Looks like its related to: https://github.com/bazelbuild/rules_docker/issues/102

priyawadhwa commented 6 years ago

Hey @MansM yah it looks like quay is working on supporting schema v2-2 based on this comment, so this probably won't work until that happens. kaniko depends on a library which assumes this format for pulling and pushing images.

xoen commented 6 years ago

Hello,

I see the following error while trying to build a docker image when the base image is on quay.io:

error building image: getting stage builder for stage 0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

EDIT: To be clear, this is a public repository.

@priyawadhwa Is this caused by the lack of support for schema v2-2 as well or should I open another issue for this?

Thanks

xoen commented 6 years ago

Also, it may be worth adding a note regarding the fact kaniko doesn't play well with quay.io in the documentation/limitations section. IMHO of course.

priyawadhwa commented 6 years ago

I'm not sure if the issue is related to schema, but definitely agree that we should add some documentation around this. Are you able to push with kaniko to a registry that supports the schema?

kameshsampath commented 5 years ago

@priyawadhwa I got a v2_2 enabled on my quay repo and I still get the same issue. When I was able to push via docker cli it was able to push the v2_2 image.

vdemeester commented 5 years ago

The detailed error is a bit different though :angel:

docker run -it -v /home/vincent/.docker/config.json:/root/.docker/config.json:ro -v /home/vincent/src/github.com/vdemeester/break-all-the-thing/foo/:/workspace/foo -e DOCKER_CONFIG=/root/.docker gcr.io/kaniko-project/executor:debug --cont
ext=/workspace/foo --dockerfile=/workspace/foo/Dockerfile --destination=quay.io/rhdevelopers/small-kaniko:0.0.1
INFO[0000] Downloading base image golang:alpine
INFO[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:d0b6fa6923af1fa27cd324325de44261e7bb801d5bba39cbbf3589ffe5a59293: no such file or directory
INFO[0002] Downloading base image golang:alpine
INFO[0003] Unpacking rootfs as cmd RUN go install -v ./... requires it.
INFO[0260] Taking snapshot of full filesystem...
INFO[0261] RUN go install -v ./...
INFO[0261] cmd: /bin/sh
INFO[0261] args: [-c go install -v ./...]
go: warning: "./..." matched no packages
INFO[0262] Taking snapshot of full filesystem...
INFO[0262] Adding whiteout for /root/.cache/go-build/5e/5eddca62332aaa6cb767cf6e8b4ce6b00d97ee63b3f549ac9586becdaea0f26c-d
INFO[0262] Adding whiteout for /root/.cache/go-build/e0/e0d9637a42516f6aa3defff56cc9a4d3deaf96b2ade546448835c41bd63d82bc-a
INFO[0262] Adding whiteout for /root/.cache/go-build/98/9816bb371bccde8cb7bf5a44febdb32bbc327a0a5c2e940d9b8b3a402f62e4f0-d
INFO[0262] Adding whiteout for /root/.cache/go-build/77/77a522d1008cfd60284c5bd5cc6fc283abe7ef012d61d06d0fd6b1ff03becc68-a
INFO[0262] Adding whiteout for /root/.cache/go-build/b5/b5cb5076dbba489139e99e802f57979f0f7f4f96806267201ca4e383513a312b-a
INFO[0262] Adding whiteout for /root/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d
INFO[0262] Adding whiteout for /root/.cache/go-build/c4/c483d510e0608431902a9e3aa9e162e4b88f745b790a07a1a4b45f22ef9013f4-a
INFO[0262] Adding whiteout for /root/.cache/go-build/31/311571a11a5934d4f63ba6daaa1b0a1e578ee13a1a373547131af1d9d157dbb4-d
error pushing image: failed to push to destination quay.io/rhdevelopers/small-kaniko:0.0.1: INVALID_REQUEST: "Invalid request"
kameshsampath commented 5 years ago

@vdemeester - if you make the docker-0 to be like "https://quay.io/v2" you will get a 401 error

HerrmannHinz commented 5 years ago

running into a similar issue here: --- jenkins file:


 * This pipeline will build and deploy a Docker image with Kaniko
 * https://github.com/GoogleContainerTools/kaniko
 * without needing a Docker host
 *
 * You need to create a jenkins-docker-cfg secret with your docker config
 * as described in
 * https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-in-the-cluster-that-holds-your-authorization-token
 */

def label = "kaniko-${UUID.randomUUID().toString()}"

podTemplate(name: 'kaniko', label: label, yaml: """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
      - name: jenkins-docker-cfg
        mountPath: /root
  volumes:
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: regcred
          items:
            - key: .dockerconfigjson
              path: .docker/config.json
"""
  ) {

  node(label) {
    stage('Build with Kaniko') {
      git 'https://github.com/jenkinsci/docker-jnlp-slave.git'
      container(name: 'kaniko', shell: '/busybox/sh') {
        withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
          sh '''#!/busybox/sh
          /kaniko/executor -f `pwd`/Dockerfile -c `pwd` --insecure --skip-tls-verify --cache=true --destination=quay.cicd.dev.intra.domain.io/admin/jenkins
          '''
        }
      }
    }
  }
}```

--- the log
```[36mINFO[0000] Resolved base name jenkins/slave:latest to jenkins/slave:latest 
INFO[0000] Resolved base name jenkins/slave:latest to jenkins/slave:latest 
INFO[0000] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:00 No matching credentials were found, falling back on anonymous
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196: no such file or directory 
INFO[0001] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:01 No matching credentials were found, falling back on anonymous
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:02 No matching credentials were found, falling back on anonymous
INFO[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196: no such file or directory 
INFO[0002] Downloading base image jenkins/slave:latest  
2019/03/13 22:41:03 No matching credentials were found, falling back on anonymous
INFO[0003] Executing 0 build triggers                   
WARN[0003] maintainer is deprecated, skipping           
INFO[0003] Applying label Description=This is a base image, which allows connecting Jenkins agents via JNLP protocols 
INFO[0003] Applying label Vendor=Jenkins project        
INFO[0003] Applying label Version=3.27                  
INFO[0003] Using files from context: [/home/jenkins/workspace/docker-image-build-and-push-kaniko-quay/jenkins-slave] 
INFO[0003] Skipping unpacking as no commands require it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] LABEL Description="This is a base image, which allows connecting Jenkins agents via JNLP protocols" Vendor="Jenkins project" Version="3.27" 
INFO[0003] Applying label Description=This is a base image, which allows connecting Jenkins agents via JNLP protocols 
INFO[0003] Applying label Vendor=Jenkins project        
INFO[0003] Applying label Version=3.27                  
INFO[0003] No files changed in this command, skipping snapshotting. 
INFO[0003] Using files from context: [/home/jenkins/workspace/docker-image-build-and-push-kaniko-quay/jenkins-slave] 
INFO[0003] COPY jenkins-slave /usr/local/bin/jenkins-slave 
INFO[0003] Taking snapshot of files...                  
INFO[0003] ENTRYPOINT ["jenkins-slave"]                 
INFO[0003] No files changed in this command, skipping snapshotting. 
2019/03/13 22:41:04 existing blob: sha256:9da6b28682cfe6db721c143309125728417dca93643b3d8d4c22ae7fbb4eb940
2019/03/13 22:41:04 existing blob: sha256:173a06ff64cce302b24e870f0b9d5758161a5bb6f1ab8ba330305552baf530d1
2019/03/13 22:41:04 existing blob: sha256:54f7e8ac135a5f502a6ee9537ef3d64b1cd2fa570dc0a40b4d3b6f7ac81e7486
2019/03/13 22:41:04 existing blob: sha256:eaa976dc543cb2e46a89970e2d079b99ccc3ca4b2c8e6c31adf9511ce8933950
2019/03/13 22:41:04 existing blob: sha256:e02013eddffd972a7b96f084ab9c5eb3683ca9e5c886e56db8f258b24ececedd
2019/03/13 22:41:04 existing blob: sha256:28fc185aee236ed4e066945827b8d3e1e7e63bea22a2173e54f05105a62faf3b
2019/03/13 22:41:04 existing blob: sha256:ee38d9f85cf610794355dc0458445408ded4d648cbc45984cd259611a8a72cc3
2019/03/13 22:41:04 existing blob: sha256:203f1094a1e2165da6f6ec505e8cffb8853d3c72a8088f41c25218121f883b0a
2019/03/13 22:41:04 existing blob: sha256:7f692fae02b67cf2beabf3ef9ce647697740ba979ab16eaec12425ead1c1ddfd
2019/03/13 22:41:04 existing blob: sha256:cc49fe331e2ecfab35824a84df31f7857284c5c52fcf559f6451a8d923d9435f
2019/03/13 22:41:04 existing blob: sha256:087a57faf9491b1b82a83e26bc8cc90c90c30e4a4d858b57ddd5b4c2c90095f6
2019/03/13 22:41:04 existing blob: sha256:5d71636fb824265e30ff34bf20737c9cdc4f5af28b6bce86f08215c55b89bfab
2019/03/13 22:41:04 existing blob: sha256:d6341e30912f12f56e18564a3b582853f65376766f5f9d641a68a724ed6db88f
2019/03/13 22:41:04 pushed blob: sha256:8307581c6ee32b3cd86cc97109f5e46d0c321e6dd8b00fda0f93715c0f143561
2019/03/13 22:41:04 pushed blob: sha256:5099fe4311c0246345cb4ea2a742234b79d20d5ac76b6ca950fc18d0ec66adec
error pushing image: failed to push to destination quay.cicd.dev.intra.domain.io/admin/jenkins:latest: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]```
samm-git commented 5 years ago

Same for me:

DEBU[0000] Adding /var/lib to layer, because it was changed.
DEBU[0000] Adding file /var/lib to tar
2019/05/12 15:03:58 existing blob: sha256:688a776db95ffbd66dd4696263d34ca00bd330f30f39a9d39d818a07b086ed17
2019/05/12 15:03:58 existing blob: sha256:6bfc4ec4420a10145bd40caf0499a57618342f27c0ad95f1785b8a1e31090058
2019/05/12 15:03:58 existing blob: sha256:743f2d6c1f65c793009f30acb07845ba2ef968192732afdab2ecf9a475515393
2019/05/12 15:04:00 pushed blob sha256:e927e04a911a46c6b85fec28ba094ca117aa91f6eda342e2c8251c172eb80aa0
error pushing image: failed to push to destination docker.private/oleksii_samorukov/testme:latest: MANIFEST_INVALID: "manifest invalid"
vdemeester commented 5 years ago

@samm-git @HerrmannHinz the MANIFEST_INVALID error means this repository is not a schema v2 enabled repository — and kaniko/go-containerregistry does not support schema v1. You may need to update your quay instance to get support for schema v2 (Red Hat Quay 3 supports v2 — not sure if Quay 2.x does)

samm-git commented 5 years ago

@vdemeester thank you for reply. With quay io i am getting INVALID_REQUEST: "Invalid request"

Anything i can do here?

error pushing image: failed to push to destination quay.io/samm_git/kaniko-test:latest: INVALID_REQUEST: "Invalid request"
NicolasRouquette commented 5 years ago

I ran into the same error as @HerrmannHinz pushing to a Quay Enterprise v2.9.2 repository:

...
INFO[0056] CMD "/app/zeppelin/bin/zeppelin.sh"          
2019/07/28 15:17:12 existing blob: sha256:b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639
2019/07/28 15:17:12 existing blob: sha256:f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2
2019/07/28 15:17:12 existing blob: sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10
2019/07/28 15:17:13 pushed blob: sha256:310aee8eca5b3e62f6d7bff786e99e24a5cf933cd00c44197514e0b7affe8faf
2019/07/28 15:17:13 pushed blob: sha256:19c179ca1c56c1071175d0657a16630f234b3170612914b956f5de1b3e1011b2
2019/07/28 15:17:18 pushed blob: sha256:9ddf6b971be8df6d43dcc11156e9fce1c2d826c657e2e804fa6c2f2f9fe2613c
2019/07/28 15:18:34 pushed blob: sha256:ecba3643a9b9be57afffaacffed8ef369a3fae0c52d3fa193ab2fb4d5825128c
error pushing image: failed to push to destination registry.jpl.nasa.gov/caesar/zeppelin:latest: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]

Based on @vdemeester, it seems that the sysadmins would need to deploy Quay 3.

NicolasRouquette commented 5 years ago

I was curious what the manifest looks like. So, I changed my k8s job like this:

apiVersion: batch/v1
kind: Job
...
spec:
  ...
  template:
    ...
    spec:
      ...
      initContainers:
          ...
        - name: build-and-publish
          image: gcr.io/kaniko-project/executor:latest
          args:
            - "--verbosity=debug"
            - "--context=/data"
            - "--dockerfile=/data/Dockerfile"
            - "--destination=registry.jpl.nasa.gov/caesar/zeppelin:latest"
            # https://github.com/GoogleContainerTools/kaniko/issues/400#issuecomment-515772279
            - "--no-push"
            - "--tarPath=/data/image.tar"
          volumeMounts:
            - mountPath: /kaniko/.docker/config.json
              name: secrets
              subPath: config.json
            - mountPath: /data
              name: data
          resources:
            requests:
              cpu: "1"
              memory: 4Gi
      containers:
        - name: inspect-image
          image: opencaesar/docker-git-utilities:latest
          command:
            - /bin/bash
          args:
            - "-c"
            - "tar xf /data/image.tar manifest.json; cat manifest.json"
          volumeMounts:
            - mountPath: /data
              name: data
          resources:
            requests:
              cpu: "1"
              memory: 128Mi

And I got this:

kubectl logs -l job-name==build-and-publish-zeppelin-image | jq .
[
  {
    "Config": "sha256:5c676888e40a41bed73936071aeabf5150fda254f3a54bcb270d966f4b6437ad",
    "RepoTags": [
      "registry.jpl.nasa.gov/caesar/zeppelin:latest"
    ],
    "Layers": [
      "e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10.tar.gz",
      "f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2.tar.gz",
      "b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639.tar.gz",
      "6d1eceff905aba4af4996da3dfee3ad66f9ee07535d3c4d13f5e9f50dfad3afe.tar.gz",
      "ca893d739e75743cebef3abc0af7386faafea8f5b630cc17814ac4e3a3af434e.tar.gz",
      "34b5a0a5f004db98ba6dada48442d75d72cd9d4c9f0ac104ae89e2e1eb57452d.tar.gz"
    ]
  }
]

Can anyone suggest solutions for a workaround like this:

HerrmannHinz commented 5 years ago

Afaik quay 3 on prem is not available yet? For the saas version One can request to get updated to 3.x

There is an issue under moby/buildkit...

HerrmannHinz commented 5 years ago

https://github.com/moby/buildkit/issues/409

cvgw commented 5 years ago

Kaniko only support the v2 schema. There are no plans to support the v1 schema. Closing this as won't fix. Please re-open if this was closed in error.