Closed adambkaplan closed 1 year ago
What if you leave off the "docker.io"? cc @jonjohnsonjr any ideas here?
I was able to verify that my ~/.docker/config.json contains valid auth credentials for docker.io
Can you share this file (with anything sensitive redacted)?
Looks like a pretty standard dockerconfigjson:
{
"auths": {
"cloud.openshift.com": {
"auth": "<TOKEN-REDACTED>"
},
"docker.io": {
"auth": "<TOKEN-REDACTED>"
},
"quay.io": {
"auth": "<TOKEN-REDACTED>"
},
"registry.connect.redhat.com": {
"auth": "<TOKEN-REDACTED>"
},
"registry.redhat.io": {
"auth": "<TOKEN-REDACTED>"
},
"registry.svc.ci.openshift.org": {
"auth": "<TOKEN-REDACTED>"
}
}
What if you leave off the "docker.io"? cc @jonjohnsonjr any ideas here?
Same result.
Note for quay.io authentication isn't the issue - cosign is able to log in just fine, however uploading signatures fails because quay.io doesn't accept the cosign layer type.
See log: https://gist.github.com/adambkaplan/0b8b4bfdaf8771d57730eface258d5a8
It's stupid but can you try changing the "docker.io"
key to "https://index.docker.io/v1/"
?
Some version of docker would set this as the key, and I haven't found a reasonable way to work around this without breaking someone: https://github.com/docker/cli/blob/25eee83d6b8c475548254b2decc9c8e0490d229c/cli/config/configfile/file.go#L23
It's stupid but can you try changing the
"docker.io"
key to"https://index.docker.io/v1/"
?
Sadly same result :disappointed:
Hm. You're right, it's not sending creds in the initial auth handshake. I would have guessed that https://index.docker.io/v1/
was the issue, as well.
Looks like ggcr does recognize docker.io
and does translate it to index.docker.io
under the hood, which means that it uses https://index.docker.io/v1/
as its auth key: https://github.com/google/go-containerregistry/blob/7e0ed51a7bb1930bfb6859f319975b578ed04de2/pkg/authn/keychain.go#L73-L76
Sadly same result
π
You're certain you don't have a credential store or credential helper configured?
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
$ cat config.json
{
"auths": {
"docker.io": {
"auth": "QXp1cmVEaWFtb25kOmh1bnRlcjI="
}
}
}
$ echo 'docker.io' | DOCKER_CONFIG=$PWD crane auth get
credentials not found in native keychain
$ cat config.json
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "QXp1cmVEaWFtb25kOmh1bnRlcjI="
}
}
}
$ echo 'docker.io' | DOCKER_CONFIG=$PWD crane auth get
{"Username":"AzureDiamond","Secret":"hunter2"}
This would match the failure mode I expect from your config file :/
I am running Fedora 34 Silverblue - no credential store as far as I am aware of, and assuming the presence of anything Docker is not a safe assumption!
I don't want to say I don't believe you, but I'm having a hard time imagining what could be going wrong here π
Can you verify that cosign is reading the right file (via strace
or similar)?
Are you executing cosign as a different user, maybe?
Is DOCKER_CONFIG
set to point to a different file?
Can you verify that cosign is reading the right file (via
strace
or similar)?
I can verify it is reading what I think is the right file. From strace, it's clear the config.json file is read:
read(6, "{\n\t\"auths\": {\n\t\t\"cloud.openshift"..., 512) = 512
read(6, "<redacted>=\"\n\t\t"..., 1024) = 1024
read(6, "<redacted>"..., 2048) = 1378
Are you executing cosign as a different user, maybe?
I'm running cosign in a Silverblue toolbox. It's a container-like environment where the user looks and feels the same as the user on the host system. So in the toolbox I'm my normal username with my normal permissions.
Is
DOCKER_CONFIG
set to point to a different file?
No - this is empty.
and that I can pull images using those credentials with my container engine (podman).
So I know that podman tries to read a different file by default from here:
The path of the authentication file can be specified by the user by setting the authfile flag. The default path for reading and writing credentials is ${XDG_RUNTIME_DIR}/containers/auth.json. Podman will use existing credentials if the user does not pass in a username. Podman will first search for the username and password in the ${XDG_RUNTIME_DIR}/containers/auth.json, if they are not valid, Podman will then use any existing credentials found in $HOME/.docker/config.json.
Are you certain that podman is reading the same credentials file as cosign
?
It's possible that fixing this issue gets you some credentials, but not valid credentials?
In your original debug logs, we see that cosign isn't sending any credentials to Docker Hub:
2021/08/27 10:46:56 --> GET https://auth.docker.io/token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io [body redacted: basic token response contains credentials]
2021/08/27 10:46:56 GET /token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io HTTP/1.1
Host: auth.docker.io
User-Agent: go-containerregistry/v0.6.0
Accept-Encoding: gzip
If we're sending invalid credentials, we would expect to see this line as well:
Authorization: <redacted>
@jonjohnsonjr it seems the root issue here is that it is very hard to identify which credentials are being used to push/pull from the container registry. Next chance I have some free "hacking" time I can try submitting a PR to surface this in debug logs.
Any initial guidance/pointers would be appreciated - thanks!
I'm hitting this same issue when trying to integrate cosign into our teams CI flow. Any use of buildah, podman, skopeo based tools where the credentials are stored in a different location than ~/.docker, seems to have cosign not recognise the use of a different container build tool and it's already authenticated information creds.
As these tools are commonly used in automated flows, it would be great if anyone has any ideas to work round the current situation.
I skimmed the issue but had issues (403) POST
'ing to GHCR.
I'm running Docker under a Snap (I assume I'll have the same issue w/ e.g. Podman) and the Snap maintains ./docker/config.json
in a Snap subdir (${HOME}/snap/docker/current/.docker/config.json
).
The solution was to copy (I guess I could have ln -s
too?) the file to Docker's default config location (${HOME}.docker/config.json
).
I assume the issue is that cosign
expects a Docker daemon with a default config (or env vars set).
I think it would be preferable either to be able to configure cosign sign
with an auth token for the registry referenced by the image (i.e. replacing the value from .docker/config.json
) or to implement the registry API and support e.g. cosign registry login
directly.
I assume the issue is that cosign expects a Docker daemon with a default config (or env vars set).
Not a daemon, just the normal docker config file behavior; i.e. $HOME/.docker/config.json
but it can be overridden by DOCKER_CONFIG
. If Snap is moving this around but not setting that environment variable, I'd argue it's a bug in Snap?
or to implement the registry API and support e.g. cosign registry login directly.
There's not really a registry API for logging in. We could support a login
command, but it just manipulates a local configuration file and wouldn't support configuring credential helpers...
I think the cosign tool and docs could probably do a better job describing how auth works, regardless.
We could support a login command, but it just manipulates a local configuration file and wouldn't support configuring credential helpers...
Veto :p
We should endeavor to figure out @adambkaplan's issue and identify how we've differed from Docker's credential resolution behavior. I might have volunteered myself for that π’
I think the cosign tool and docs could probably do a better job describing how auth works, regardless.
Agreed.
Any use of buildah, podman, skopeo based tools where the credentials are stored in a different location than ~/.docker, seems to have cosign not recognise the use of a different container build tool and it's already authenticated information creds.
@tommyreilly you're right, it only supports docker
's behavior right now.
If we end up supporting some alternative to Docker's configuration and authentication scheme, it should be one we maintain. I think documenting our docker
mimicry and prescribing a manual way of setting things up for automation should suffice for now
Hi, had auth issues (I use podman instead of docker), the following was a workaround for me:
cat ${XDG_RUNTIME_DIR}/containers/auth.json > ~/.docker/config.json
My two cents: auth.json
(podman) or config.json
(docker) must contain an entry with authentications for https://index.docker.io/v1/
otherwise the authentication seems to fail. I have solved it by adding an extra entry like so (trailing / needed):
{
"auths": {
"docker.io": {
"auth": "redacted"
},
"https://index.docker.io/v1/": {
"auth": "redacted"
}
}
}
I found out that if the registry URL is index.docker.io
go-containerregistry uses the full url when looking for creds.
Sadly, I'm expirencing the same issue as others. cosign
fails to sign an image on my local Quay installation, apparently due to lacking credentials:
# cosign -d sign --key cosign.key quay.skynet/ibazulic/quay:v2.9.5
an error occurred: no provider found for that key reference, will try to load key from disk...
Enter password for private key: 2022/02/21 14:04:19 --> GET https://quay.skynet/v2/
...
2022/02/21 14:04:19 --> GET https://quay.skynet/v2/auth?scope=repository%3Aibazulic%2Fquay%3Apull&service=quay.skynet [body redacted: basic token response contains credentials]
2022/02/21 14:04:19 GET /v2/auth?scope=repository%3Aibazulic%2Fquay%3Apull&service=quay.skynet HTTP/1.1
Host: quay.skynet
User-Agent: cosign/v1.5.2 (linux; amd64) go-containerregistry/v0.8.1-0.20220125170349-50dfc2733d10
Authorization: <redacted>
Accept-Encoding: gzip
2022/02/21 14:04:20 <-- 200 https://quay.skynet/v2/auth?scope=repository%3Aibazulic%2Fquay%3Apull&service=quay.skynet (193.918782ms) [body redacted: basic token response contains credentials]
2022/02/21 14:04:20 HTTP/1.1 200 OK
Content-Length: 977
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: application/json
Date: Mon, 21 Feb 2022 14:04:20 GMT
Server: nginx/1.14.1
X-Frame-Options: DENY
2022/02/21 14:04:20 --> GET https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5
2022/02/21 14:04:20 GET /v2/ibazulic/quay/manifests/v2.9.5 HTTP/1.1
Host: quay.skynet
User-Agent: cosign/v1.5.2 (linux; amd64) go-containerregistry/v0.8.1-0.20220125170349-50dfc2733d10
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip
2022/02/21 14:04:20 <-- 401 https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5 (4.850918ms)
2022/02/21 14:04:20 HTTP/1.1 401 UNAUTHORIZED
Content-Length: 112
Content-Type: application/json
Date: Mon, 21 Feb 2022 14:04:20 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: nginx/1.14.1
Www-Authenticate: Bearer realm="https://quay.skynet/v2/auth",service="quay.skynet",scope="repository:ibazulic/quay:pull"
{"errors":[{"code":"UNAUTHORIZED","detail":{},"message":"access to the requested resource is not authorized"}]}
Error: signing [quay.skynet/ibazulic/quay:v2.9.5]: accessing entity: GET https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5: UNAUTHORIZED: access to the requested resource is not authorized; map[]
main.go:46: error during command execution: signing [quay.skynet/ibazulic/quay:v2.9.5]: accessing entity: GET https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5: UNAUTHORIZED: access to the requested resource is not authorized; map[]
Output of Docker's config.json
file:
# cat ~/.docker/config.json
{
"auths": {
"quay.skynet": {
"auth": "SOMETHING"
}
}
}
Podman's auth.json
also exists:
# cat /run/containers/0/auth.json
{
"auths": {
"quay.skynet": {
"auth": "SOMETHING"
}
}
}
Any ideas?
Any ideas?
You're certain the credentials are valid?
Since https://github.com/google/go-containerregistry/pull/1185 which should be in cosign afaik, cosign will check for podman's auth config as well. Does this help at all?
Any ideas?
You're certain the credentials are valid?
Didn't see your response till now, sorry. Yes, credentials are fine, push/pulls against the registry function normally. This is my own local instance. I'll try again with a new release.
In my case, I discovered that this .docker/config.json results in authentication failure:
{"auths":{"https://index.docker.io":{"auth":"<redacted>"}}}
But this one results in success:
{"auths":{"https://index.docker.io/v1/":{"auth":"<redacted>"}}}
Interestingly the authentication failure when /v1
is not appended clearly shows that /v2/
is attempted by default:
main.go:46: error during command execution: GET https://index.docker.io/v2/myrepo/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myrepo/curl Type:repository]]
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue was closed because it has been stalled for 5 days with no activity.
Hi, had auth issues (I use podman instead of docker), the following was a workaround for me:
cat ${XDG_RUNTIME_DIR}/containers/auth.json > ~/.docker/config.json
working as expected , great π
can't cosign look in the default credentials location for both podman and docker?
In some CI/CD, if you need run cosign in buildah container with docker or podman, you must exec cat /run/containers/0/auth.json > ~/.docker/config.json
after buildah login. Otherwise, you will got the same error when sign image
Question
I'm trying to sign a container image that I pushed to docker.io. This seems simple enough:
However, when trying to pull the image, cosign does not appear to use my local credentials. Instead cosign appears to do the following:
I was able to verify that my ~/.docker/config.json contains valid auth credentials for docker.io, and that I can pull images using those credentials with my container engine (podman).
This could be related to #337.
Here's the debug output: