Closed MansM closed 5 years ago
Hey @MansM yah it looks like quay is working on supporting schema v2-2 based on this comment, so this probably won't work until that happens. kaniko depends on a library which assumes this format for pulling and pushing images.
Hello,
I see the following error while trying to build a docker image when the base image is on quay.io:
error building image: getting stage builder for stage 0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
EDIT: To be clear, this is a public repository.
@priyawadhwa Is this caused by the lack of support for schema v2-2 as well or should I open another issue for this?
Thanks
Also, it may be worth adding a note regarding the fact kaniko doesn't play well with quay.io in the documentation/limitations section. IMHO of course.
I'm not sure if the issue is related to schema, but definitely agree that we should add some documentation around this. Are you able to push with kaniko to a registry that supports the schema?
@priyawadhwa I got a v2_2 enabled on my quay repo and I still get the same issue. When I was able to push via docker cli it was able to push the v2_2 image.
The detailed error is a bit different though :angel:
docker run -it -v /home/vincent/.docker/config.json:/root/.docker/config.json:ro -v /home/vincent/src/github.com/vdemeester/break-all-the-thing/foo/:/workspace/foo -e DOCKER_CONFIG=/root/.docker gcr.io/kaniko-project/executor:debug --cont
ext=/workspace/foo --dockerfile=/workspace/foo/Dockerfile --destination=quay.io/rhdevelopers/small-kaniko:0.0.1
INFO[0000] Downloading base image golang:alpine
INFO[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:d0b6fa6923af1fa27cd324325de44261e7bb801d5bba39cbbf3589ffe5a59293: no such file or directory
INFO[0002] Downloading base image golang:alpine
INFO[0003] Unpacking rootfs as cmd RUN go install -v ./... requires it.
INFO[0260] Taking snapshot of full filesystem...
INFO[0261] RUN go install -v ./...
INFO[0261] cmd: /bin/sh
INFO[0261] args: [-c go install -v ./...]
go: warning: "./..." matched no packages
INFO[0262] Taking snapshot of full filesystem...
INFO[0262] Adding whiteout for /root/.cache/go-build/5e/5eddca62332aaa6cb767cf6e8b4ce6b00d97ee63b3f549ac9586becdaea0f26c-d
INFO[0262] Adding whiteout for /root/.cache/go-build/e0/e0d9637a42516f6aa3defff56cc9a4d3deaf96b2ade546448835c41bd63d82bc-a
INFO[0262] Adding whiteout for /root/.cache/go-build/98/9816bb371bccde8cb7bf5a44febdb32bbc327a0a5c2e940d9b8b3a402f62e4f0-d
INFO[0262] Adding whiteout for /root/.cache/go-build/77/77a522d1008cfd60284c5bd5cc6fc283abe7ef012d61d06d0fd6b1ff03becc68-a
INFO[0262] Adding whiteout for /root/.cache/go-build/b5/b5cb5076dbba489139e99e802f57979f0f7f4f96806267201ca4e383513a312b-a
INFO[0262] Adding whiteout for /root/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d
INFO[0262] Adding whiteout for /root/.cache/go-build/c4/c483d510e0608431902a9e3aa9e162e4b88f745b790a07a1a4b45f22ef9013f4-a
INFO[0262] Adding whiteout for /root/.cache/go-build/31/311571a11a5934d4f63ba6daaa1b0a1e578ee13a1a373547131af1d9d157dbb4-d
error pushing image: failed to push to destination quay.io/rhdevelopers/small-kaniko:0.0.1: INVALID_REQUEST: "Invalid request"
@vdemeester - if you make the docker-0 to be like "https://quay.io/v2" you will get a 401 error
running into a similar issue here: --- jenkins file:
* This pipeline will build and deploy a Docker image with Kaniko
* https://github.com/GoogleContainerTools/kaniko
* without needing a Docker host
*
* You need to create a jenkins-docker-cfg secret with your docker config
* as described in
* https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-in-the-cluster-that-holds-your-authorization-token
*/
def label = "kaniko-${UUID.randomUUID().toString()}"
podTemplate(name: 'kaniko', label: label, yaml: """
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
volumeMounts:
- name: jenkins-docker-cfg
mountPath: /root
volumes:
- name: jenkins-docker-cfg
projected:
sources:
- secret:
name: regcred
items:
- key: .dockerconfigjson
path: .docker/config.json
"""
) {
node(label) {
stage('Build with Kaniko') {
git 'https://github.com/jenkinsci/docker-jnlp-slave.git'
container(name: 'kaniko', shell: '/busybox/sh') {
withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
sh '''#!/busybox/sh
/kaniko/executor -f `pwd`/Dockerfile -c `pwd` --insecure --skip-tls-verify --cache=true --destination=quay.cicd.dev.intra.domain.io/admin/jenkins
'''
}
}
}
}
}```
--- the log
```[36mINFO[0m[0000] Resolved base name jenkins/slave:latest to jenkins/slave:latest
[36mINFO[0m[0000] Resolved base name jenkins/slave:latest to jenkins/slave:latest
[36mINFO[0m[0000] Downloading base image jenkins/slave:latest
2019/03/13 22:41:00 No matching credentials were found, falling back on anonymous
[36mINFO[0m[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196: no such file or directory
[36mINFO[0m[0001] Downloading base image jenkins/slave:latest
2019/03/13 22:41:01 No matching credentials were found, falling back on anonymous
[36mINFO[0m[0002] Built cross stage deps: map[]
[36mINFO[0m[0002] Downloading base image jenkins/slave:latest
2019/03/13 22:41:02 No matching credentials were found, falling back on anonymous
[36mINFO[0m[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:5683f906bab2e28abe332619bcec340a4da8553f10c34bbcbe6b555d32f76196: no such file or directory
[36mINFO[0m[0002] Downloading base image jenkins/slave:latest
2019/03/13 22:41:03 No matching credentials were found, falling back on anonymous
[36mINFO[0m[0003] Executing 0 build triggers
[33mWARN[0m[0003] maintainer is deprecated, skipping
[36mINFO[0m[0003] Applying label Description=This is a base image, which allows connecting Jenkins agents via JNLP protocols
[36mINFO[0m[0003] Applying label Vendor=Jenkins project
[36mINFO[0m[0003] Applying label Version=3.27
[36mINFO[0m[0003] Using files from context: [/home/jenkins/workspace/docker-image-build-and-push-kaniko-quay/jenkins-slave]
[36mINFO[0m[0003] Skipping unpacking as no commands require it.
[36mINFO[0m[0003] Taking snapshot of full filesystem...
[36mINFO[0m[0003] LABEL Description="This is a base image, which allows connecting Jenkins agents via JNLP protocols" Vendor="Jenkins project" Version="3.27"
[36mINFO[0m[0003] Applying label Description=This is a base image, which allows connecting Jenkins agents via JNLP protocols
[36mINFO[0m[0003] Applying label Vendor=Jenkins project
[36mINFO[0m[0003] Applying label Version=3.27
[36mINFO[0m[0003] No files changed in this command, skipping snapshotting.
[36mINFO[0m[0003] Using files from context: [/home/jenkins/workspace/docker-image-build-and-push-kaniko-quay/jenkins-slave]
[36mINFO[0m[0003] COPY jenkins-slave /usr/local/bin/jenkins-slave
[36mINFO[0m[0003] Taking snapshot of files...
[36mINFO[0m[0003] ENTRYPOINT ["jenkins-slave"]
[36mINFO[0m[0003] No files changed in this command, skipping snapshotting.
2019/03/13 22:41:04 existing blob: sha256:9da6b28682cfe6db721c143309125728417dca93643b3d8d4c22ae7fbb4eb940
2019/03/13 22:41:04 existing blob: sha256:173a06ff64cce302b24e870f0b9d5758161a5bb6f1ab8ba330305552baf530d1
2019/03/13 22:41:04 existing blob: sha256:54f7e8ac135a5f502a6ee9537ef3d64b1cd2fa570dc0a40b4d3b6f7ac81e7486
2019/03/13 22:41:04 existing blob: sha256:eaa976dc543cb2e46a89970e2d079b99ccc3ca4b2c8e6c31adf9511ce8933950
2019/03/13 22:41:04 existing blob: sha256:e02013eddffd972a7b96f084ab9c5eb3683ca9e5c886e56db8f258b24ececedd
2019/03/13 22:41:04 existing blob: sha256:28fc185aee236ed4e066945827b8d3e1e7e63bea22a2173e54f05105a62faf3b
2019/03/13 22:41:04 existing blob: sha256:ee38d9f85cf610794355dc0458445408ded4d648cbc45984cd259611a8a72cc3
2019/03/13 22:41:04 existing blob: sha256:203f1094a1e2165da6f6ec505e8cffb8853d3c72a8088f41c25218121f883b0a
2019/03/13 22:41:04 existing blob: sha256:7f692fae02b67cf2beabf3ef9ce647697740ba979ab16eaec12425ead1c1ddfd
2019/03/13 22:41:04 existing blob: sha256:cc49fe331e2ecfab35824a84df31f7857284c5c52fcf559f6451a8d923d9435f
2019/03/13 22:41:04 existing blob: sha256:087a57faf9491b1b82a83e26bc8cc90c90c30e4a4d858b57ddd5b4c2c90095f6
2019/03/13 22:41:04 existing blob: sha256:5d71636fb824265e30ff34bf20737c9cdc4f5af28b6bce86f08215c55b89bfab
2019/03/13 22:41:04 existing blob: sha256:d6341e30912f12f56e18564a3b582853f65376766f5f9d641a68a724ed6db88f
2019/03/13 22:41:04 pushed blob: sha256:8307581c6ee32b3cd86cc97109f5e46d0c321e6dd8b00fda0f93715c0f143561
2019/03/13 22:41:04 pushed blob: sha256:5099fe4311c0246345cb4ea2a742234b79d20d5ac76b6ca950fc18d0ec66adec
error pushing image: failed to push to destination quay.cicd.dev.intra.domain.io/admin/jenkins:latest: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]```
Same for me:
DEBU[0000] Adding /var/lib to layer, because it was changed.
DEBU[0000] Adding file /var/lib to tar
2019/05/12 15:03:58 existing blob: sha256:688a776db95ffbd66dd4696263d34ca00bd330f30f39a9d39d818a07b086ed17
2019/05/12 15:03:58 existing blob: sha256:6bfc4ec4420a10145bd40caf0499a57618342f27c0ad95f1785b8a1e31090058
2019/05/12 15:03:58 existing blob: sha256:743f2d6c1f65c793009f30acb07845ba2ef968192732afdab2ecf9a475515393
2019/05/12 15:04:00 pushed blob sha256:e927e04a911a46c6b85fec28ba094ca117aa91f6eda342e2c8251c172eb80aa0
error pushing image: failed to push to destination docker.private/oleksii_samorukov/testme:latest: MANIFEST_INVALID: "manifest invalid"
@samm-git @HerrmannHinz the MANIFEST_INVALID
error means this repository is not a schema v2 enabled repository — and kaniko
/go-containerregistry
does not support schema v1. You may need to update your quay instance to get support for schema v2 (Red Hat Quay 3 supports v2 — not sure if Quay 2.x does)
@vdemeester thank you for reply. With quay io i am getting INVALID_REQUEST: "Invalid request"
Anything i can do here?
error pushing image: failed to push to destination quay.io/samm_git/kaniko-test:latest: INVALID_REQUEST: "Invalid request"
I ran into the same error as @HerrmannHinz pushing to a Quay Enterprise v2.9.2 repository:
...
INFO[0056] CMD "/app/zeppelin/bin/zeppelin.sh"
2019/07/28 15:17:12 existing blob: sha256:b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639
2019/07/28 15:17:12 existing blob: sha256:f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2
2019/07/28 15:17:12 existing blob: sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10
2019/07/28 15:17:13 pushed blob: sha256:310aee8eca5b3e62f6d7bff786e99e24a5cf933cd00c44197514e0b7affe8faf
2019/07/28 15:17:13 pushed blob: sha256:19c179ca1c56c1071175d0657a16630f234b3170612914b956f5de1b3e1011b2
2019/07/28 15:17:18 pushed blob: sha256:9ddf6b971be8df6d43dcc11156e9fce1c2d826c657e2e804fa6c2f2f9fe2613c
2019/07/28 15:18:34 pushed blob: sha256:ecba3643a9b9be57afffaacffed8ef369a3fae0c52d3fa193ab2fb4d5825128c
error pushing image: failed to push to destination registry.jpl.nasa.gov/caesar/zeppelin:latest: MANIFEST_INVALID: manifest invalid; map[message:manifest schema version not supported]
Based on @vdemeester, it seems that the sysadmins would need to deploy Quay 3.
I was curious what the manifest looks like. So, I changed my k8s job like this:
apiVersion: batch/v1
kind: Job
...
spec:
...
template:
...
spec:
...
initContainers:
...
- name: build-and-publish
image: gcr.io/kaniko-project/executor:latest
args:
- "--verbosity=debug"
- "--context=/data"
- "--dockerfile=/data/Dockerfile"
- "--destination=registry.jpl.nasa.gov/caesar/zeppelin:latest"
# https://github.com/GoogleContainerTools/kaniko/issues/400#issuecomment-515772279
- "--no-push"
- "--tarPath=/data/image.tar"
volumeMounts:
- mountPath: /kaniko/.docker/config.json
name: secrets
subPath: config.json
- mountPath: /data
name: data
resources:
requests:
cpu: "1"
memory: 4Gi
containers:
- name: inspect-image
image: opencaesar/docker-git-utilities:latest
command:
- /bin/bash
args:
- "-c"
- "tar xf /data/image.tar manifest.json; cat manifest.json"
volumeMounts:
- mountPath: /data
name: data
resources:
requests:
cpu: "1"
memory: 128Mi
And I got this:
kubectl logs -l job-name==build-and-publish-zeppelin-image | jq .
[
{
"Config": "sha256:5c676888e40a41bed73936071aeabf5150fda254f3a54bcb270d966f4b6437ad",
"RepoTags": [
"registry.jpl.nasa.gov/caesar/zeppelin:latest"
],
"Layers": [
"e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10.tar.gz",
"f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2.tar.gz",
"b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639.tar.gz",
"6d1eceff905aba4af4996da3dfee3ad66f9ee07535d3c4d13f5e9f50dfad3afe.tar.gz",
"ca893d739e75743cebef3abc0af7386faafea8f5b630cc17814ac4e3a3af434e.tar.gz",
"34b5a0a5f004db98ba6dada48442d75d72cd9d4c9f0ac104ae89e2e1eb57452d.tar.gz"
]
}
]
Can anyone suggest solutions for a workaround like this:
manifest.json
to be compatible with the v1 format that Quay v2 acceptsAfaik quay 3 on prem is not available yet? For the saas version One can request to get updated to 3.x
There is an issue under moby/buildkit...
Kaniko only support the v2 schema. There are no plans to support the v1 schema. Closing this as won't fix. Please re-open if this was closed in error.
Actual behavior Kaniko cant push to quay
Expected behavior kaniko able to push to quay To Reproduce Steps to reproduce the behavior: as gitlab runner
Additional Information
Dockerfile FROM centos:7 curl -O http://somewhere/internalcacerts.rpm && rpm -ivh internalcacerts.rpm && rm -rf internalcacerts.rpm
Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:debug Digest: sha256:24c1995868cff80f67a682449c5fe61f2016f6f6c7bbbdd908b4f8b28648fe75
Looks like its related to: https://github.com/bazelbuild/rules_docker/issues/102