Open deedubs opened 2 years ago
There have been some bugs with v1.7.0
related to auth, specifically against GCR, that caused us to roll back :latest
to point at v1.6.0
.
I believe these issues are fixed at head. Until v1.8.0 is out (#1871), could you try your build with the latest commit-tagged image, built from a7425d1fd0442b58dc24698285102176365a28d9, and let me know if that works for you?
gcr.io/kaniko-project/executor:a7425d1fd0442b58dc24698285102176365a28d9@sha256:939b0a1a0aaad97a06db665291ac2270a9abe538af4198000046f743d1e61cd4
If it does, then when v1.8.0 is released you should get the fix (and until then you can use the commit-tagged image)
If not, please let me know so we can find and fix the issue.
Confirmed that our pipelines can build against artifact registry using a7425d1
We'll continue to use the commit tagged image, thanks so much for the quick response!
o/t while bisecting my way from 1.6 to 1.7, I noticed the GCR helpers, is it even necessary to call docker-credential-gcr
manually as a pre-step?
I don't think it should be necessary* -- in v1.6.0 and v1.7.0 it was initialized in the Dockerfile (setting up /kaniko/.config/gcloud/docker_credential_gcr_config.json
which the helper uses), and at head the cred helpers aren't technically needed in the image since the same logic is embedded in kaniko itself -- it looks for creds available in the environment and will use those even if the cred helpers aren't available or initialized.
So in all cases it should be okay to omit any cred helper initialization pre-step, as far as I know.
*if you test and find out that it is necessary, please let me know!
When I dropped the call, and installation of docker-credential-gcr
, I get
time="2022-01-25T20:57:16Z" level=error msg="No matching credentials were found for \"us-east4-docker.pkg.dev\""
time="2022-01-25T20:57:16Z" level=error msg="No matching credentials were found for \"us-east4-docker.pkg.dev\""
time="2022-01-25T20:57:16Z" level=error msg="No matching credentials were found for \"us-east4-docker.pkg.dev\""
time="2022-01-25T20:57:16Z" level=fatal msg="deleting pod: context canceled" subtask=tools/skaffold task=Buil
Note this is being invoked via tekton
steps:
- name: skaffold-build
image: gcr.io/k8s-skaffold/skaffold:v1.35.1@sha256:edd5fefb172bb60396fed6b83868cfec38be8083e81b3c1aa8d3ec5cac66c09f
workingDir: $(workspaces.source.path)
script: |
skaffold build \
--default-repo=us-east4-docker.pkg.dev/$(params.DEFAULT_REPO) \
--output="{{range \$index, \$artifact := .Builds}}{{if \$index}},{{end}}{{\$artifact.Tag}}{{end}}" \
--file-output=/tekton/results/IMAGES
Well that's a little surprising to me. 🤔
It works with a step to initialize the cred helper? What's that look like?
And this is with the kaniko executor @main? Or v1.6.0 or 1.7.0?
When we run
docker-credential-gcr configure-docker --registries=us-east4-docker.pkg.dev
And then the skaffold build
invocation it works.
This is with the commit tagged version
I thought dockerhub had the issue but apparently, I had authentication issues with :latest tag, I didn't realize that you rollback to 1.6 so I thought dockerhub had the issue but now with the new release of 1.8 we still get the authentication error.
I thought dockerhub had the issue but apparently, I had authentication issues with :latest tag, I didn't realize that you rollback to 1.6 so I thought dockerhub had the issue but now with the new release of 1.8 we still get the authentication error.
The original issue seemed to be reporting issues authenticating with GCR/AR, not Dockerhub. Are you saying you also have issues with Dockerhub now?
In any case, especially where auth is involved, it's useful to tell whether you can successfully authorize a push to your registry using docker push
or another similar tool. If that works and Kaniko doesn't, it's a bug in Kaniko.
Yes we do use dockerhub sorry for not saying this clearly.
I cannot say with confidence that this is a bug with kaniko. It happened randomly about the time 1.7 was released then fixed it self which I now think was tagging 1.6 with latest again. We thought it was something with dockerhub. We had it again when 1.8 was released yesterday, most of our pushes are failing but not all and yes we can push with docker push
. I know I am vague, If there is more information I could give to help please let me know.
I've hit the same issue here.
GOOGLE_APPLICATION_CREDENTIALS=/gcp-builder-sa/token.json
echo "{\"credHelpers\":{\"europe-west1-docker.pkg.dev\":\"gcr\"}}" > /kaniko/.docker/config.json
using executor:v1.8.0-debug :
error checking push permissions --
make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for
"europe-west1-docker.pkg.dev/foo/bar/mayapp:f64ca23c": creating push check transport for europe-west1-docker.pkg.dev failed:
GET https://europe-west1-docker.pkg.dev/v2/token?scope=repository%3Afoo%2Fbar%2Fmyapp%3Apush%2Cpull&service=europe-west1-docker.pkg.dev:
UNAUTHORIZED: authentication failed
using executor:v1.7.0-debug :
WARN[0000] Skip running docker-credential-gcr as user provided docker configuration exists at /kaniko/.docker/config.json
E0310 17:12:37.509233
18 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for
"europe-west1-docker.pkg.dev/foo/bar/myapp:39038142": creating push check transport for europe-west1-docker.pkg.dev failed:
GET https://europe-west1-docker.pkg.dev/v2/token?scope=repository%3Afoo%2Fbar%2Fmyapp%3Apush%2Cpull&service=europe-west1-docker.pkg.dev:
UNAUTHORIZED: authentication failed
using executor:v1.6.0-debug :
WARN[0000]
Skip running docker-credential-gcr as user provided docker configuration exists at /kaniko/.docker/config.json
E0310 17:14:13.558756
18 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
INFO[0000] Using dockerignore file: /builds/foo/bar/myapp/.dockerignore
[...]
INFO[0013] Pushing image to europe-west1-docker.pkg.dev/foo/bar/myapp:1db4ec45
INFO[0014] Pushing image to europe-west1-docker.pkg.dev/foo/bar/myapp:latest
INFO[0014] Pushed image to 2 destinations
Is this somehow related ?
@BarthV could you do me a favor, and try this build without your ~/.docker/config.json
file mounted in? There were changes since v1.7.0 to compile in the cred helper logic into the Kaniko binary that should pick up your token.json
creds when pushing to *-docker.pkg.dev
, but they're only checked after the Docker config JSON.
If removing that causes your push to work again, that would be great signal that the cred helper fallback is working, and would give us an option for others facing similar auth issues.
@imjasonh
GOOGLE_APPLICATION_CREDENTIALS ENV set with token.json file path
no ~/.docker/config.json at all |
Version | Working? |
---|---|---|
v1.6.0 | :heavy_check_mark: | |
v1.7.0 | :x: | |
v1.8.0 | :heavy_check_mark: |
GOOGLE_APPLICATION_CREDENTIALS ENV set with token.json file path
~/.docker/config.json loaded with unused third-party external credentials (non-gcr, non-credHelper) |
Version | Working? |
---|---|---|
v1.6.0 | :x: | |
v1.7.0 | :x: | |
v1.8.0 | :heavy_check_mark: |
GOOGLE_APPLICATION_CREDENTIALS ENV set with token.json file path
~/.docker/config.json loaded with gcr credHelpers for target registry |
Version | Working? |
---|---|---|
v1.6.0 | :heavy_check_mark: | |
v1.7.0 | :x: | |
v1.8.0 | :x: |
So far, it looks good by removing config.json file. It even works when using a file with unused credentials :+1:
I can't get authentication to work with GAR using version 1.8 and any of the methods in the previous post.
I also can not get authentication to work with GAR using any of the above.
@beehivewarrior are you using a GKE cluster? You need to ensure your cluster has the oauth scope
@deedubs Ah, that makes sense. Thanks!
1.12.1 is doing the same now. I had no issues with GCP GSA and private artifact registry yesterday. First time i am seeing these errors.
Actual behavior While building several containers against GCP Artifact Registry via skaffold we are getting intermittent authentication failures.
Prior to invoking skaffold we issue:
Expected behavior We expect pushes continue to work throughout the whole build.
Additional Information