failed to publish images to quay.io, status code 401 Unauthorized

[T0MASD]

I'm trying to build an image for quay,

[tomas@rpi4 cli]$ export KO_DOCKER_REPO=quay.io/todabasi_openshift/tkn
[tomas@rpi4 cli]$ podman login --get-login quay.io
todabasi_openshift+robot

however I get an auth error:

[tomas@rpi4 cli]$ ko build ./cmd/tkn --platform=linux/arm64/v8
2022/09/28 20:56:25 Using base debian@sha256:3e82b1af33607aebaeb3641b75d6e80fd28d36e17993ef13708e9493e30e8ff9 for github.com/tektoncd/cli/cmd/tkn
2022/09/28 20:56:26 Building github.com/tektoncd/cli/cmd/tkn for linux/arm64/v8
2022/09/28 20:56:50 Publishing quay.io/todabasi_openshift/tkn/tkn-255681d3e39d18f1a8b30ef37f60c17d:latest
Error: failed to publish images: error publishing ko://github.com/tektoncd/cli/cmd/tkn: writing sbom: HEAD https://quay.io/v2/todabasi_openshift/tkn/tkn-255681d3e39d18f1a8b30ef37f60c17d/blobs/sha256:b742617b43b8ab254eee1f6de38fd1b5692522975a0e84edd48e6d84332e69a2: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
2022/09/28 20:56:51 error during command execution:failed to publish images: error publishing ko://github.com/tektoncd/cli/cmd/tkn: writing sbom: HEAD https://quay.io/v2/todabasi_openshift/tkn/tkn-255681d3e39d18f1a8b30ef37f60c17d/blobs/sha256:b742617b43b8ab254eee1f6de38fd1b5692522975a0e84edd48e6d84332e69a2: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

It seems ko is not reusing my auth

[imjasonh]

Can you docker push or push an image with Buildah or podman?

Ideally ko is supposed to reuse whatever auth you configure through docker login, which is supposed to be similar to podman login, but historically there have been inconsistencies between the two that ko etc have to detect and support both.

[T0MASD]

building and pushing to local docker registry works:

[tomas@rpi4 cli]$ export KO_DOCKER_REPO=registry.apps.local
[tomas@rpi4 cli]$ ko build ./cmd/tkn --platform=linux/arm64/v8
2022/09/29 12:28:50 No matching credentials were found, falling back on anonymous
2022/09/29 12:28:53 Using base debian@sha256:3e82b1af33607aebaeb3641b75d6e80fd28d36e17993ef13708e9493e30e8ff9 for github.com/tektoncd/cli/cmd/tkn
2022/09/29 12:28:54 Building github.com/tektoncd/cli/cmd/tkn for linux/arm64/v8
2022/09/29 12:37:54 Publishing registry.apps.local/tkn-255681d3e39d18f1a8b30ef37f60c17d:latest
2022/09/29 12:37:54 pushed blob: sha256:48f6fedda04cca86e71d4f598880c4fa42f1c4f09ac31754cafa5d701737e24a
2022/09/29 12:37:54 pushed blob: sha256:b742617b43b8ab254eee1f6de38fd1b5692522975a0e84edd48e6d84332e69a2
2022/09/29 12:37:54 registry.apps.local/tkn-255681d3e39d18f1a8b30ef37f60c17d:sha256-ce0524ae7d38cfaf04ecc7e4c0302514e31149d16746e996472e44afde81d584.sbom: digest: sha256:74c51ea6c500dca02074c6edd30143ad45c973b1220d00432c1c2875fa268ebb size: 370
2022/09/29 12:37:54 Published SBOM registry.apps.local/tkn-255681d3e39d18f1a8b30ef37f60c17d:sha256-ce0524ae7d38cfaf04ecc7e4c0302514e31149d16746e996472e44afde81d584.sbom
2022/09/29 12:37:55 pushed blob: sha256:250c06f7c38e52dc77e5c7586c3e40280dc7ff9bb9007c396e06d96736cf8542
2022/09/29 12:37:55 pushed blob: sha256:5ca9d30a68a8217a39e25315da9117df26b99584b0ac38229a7ae34a5f5235b5
2022/09/29 12:37:56 pushed blob: sha256:88d40a4f8b4f7e7ead295063aa56b013f2c93b36fa2b663d62cd9856ba081262
2022/09/29 12:38:09 pushed blob: sha256:10cff8997b4d4f243419e6bede830f1ac33f3d18c5200e5fb80e19333883ec2b
2022/09/29 12:38:09 registry.apps.local/tkn-255681d3e39d18f1a8b30ef37f60c17d:latest: digest: sha256:ce0524ae7d38cfaf04ecc7e4c0302514e31149d16746e996472e44afde81d584 size: 960
2022/09/29 12:38:09 Published registry.apps.local/tkn-255681d3e39d18f1a8b30ef37f60c17d@sha256:ce0524ae7d38cfaf04ecc7e4c0302514e31149d16746e996472e44afde81d584
registry.apps.local/tkn-255681d3e39d18f1a8b30ef37f60c17d@sha256:ce0524ae7d38cfaf04ecc7e4c0302514e31149d16746e996472e44afde81d584

however pushing to quay via ko not

[tomas@rpi4 cli]$ podman login -u="todabasi_openshift+robot" -p="***************************" quay.io
Login Succeeded!
[tomas@rpi4 cli]$ export KO_DOCKER_REPO=quay.io/todabasi_openshift/tkn
[tomas@rpi4 cli]$ podman login --get-login quay.io
todabasi_openshift+robot
[tomas@rpi4 cli]$ ko build ./cmd/tkn --platform=linux/arm64/v8
2022/09/29 12:58:11 Using base debian@sha256:3e82b1af33607aebaeb3641b75d6e80fd28d36e17993ef13708e9493e30e8ff9 for github.com/tektoncd/cli/cmd/tkn
2022/09/29 12:58:12 Building github.com/tektoncd/cli/cmd/tkn for linux/arm64/v8
2022/09/29 12:58:39 Publishing quay.io/todabasi_openshift/tkn/tkn-255681d3e39d18f1a8b30ef37f60c17d:latest
Error: failed to publish images: error publishing ko://github.com/tektoncd/cli/cmd/tkn: writing sbom: HEAD https://quay.io/v2/todabasi_openshift/tkn/tkn-255681d3e39d18f1a8b30ef37f60c17d/blobs/sha256:3231bf2a4bc72dcd7c7fc7b61b69b2ab88e5441e65073f72373dc3fe3b41bd2b: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
2022/09/29 12:58:40 error during command execution:failed to publish images: error publishing ko://github.com/tektoncd/cli/cmd/tkn: writing sbom: HEAD https://quay.io/v2/todabasi_openshift/tkn/tkn-255681d3e39d18f1a8b30ef37f60c17d/blobs/sha256:3231bf2a4bc72dcd7c7fc7b61b69b2ab88e5441e65073f72373dc3fe3b41bd2b: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

I suspect an issue with quay

[imjasonh]

Can you docker push quay.io/todabasi_openshift/tkn ? Or podman push?

If both of those are "yes" and ko still doesn't work, that's probably a ko bug about discovering podman-configured auth correctly.

[T0MASD]

I've spun up a new environment and gave it a go..

building and pushing tkn image to quay:

[tomas@rpi4-qa tektoncd-plumbing]$ podman images
REPOSITORY                TAG                IMAGE ID      CREATED         SIZE
localhost/tkn             latest             c2988362f284  35 seconds ago  96 MB
<none>                    <none>             75f6d31ce9e3  57 seconds ago  425 MB
docker.io/library/alpine  3.15               02e0d6fdf486  7 weeks ago     5.62 MB
docker.io/library/golang  1.18.3-alpine3.15  d0dfc3912327  4 months ago    335 MB
[tomas@rpi4-qa tektoncd-plumbing]$ podman tag localhost/tkn quay.io/todabasi_openshift/tkn
[tomas@rpi4-qa tektoncd-plumbing]$ podman images
REPOSITORY                      TAG                IMAGE ID      CREATED             SIZE
localhost/tkn                   latest             c2988362f284  About a minute ago  96 MB
quay.io/todabasi_openshift/tkn  latest             c2988362f284  About a minute ago  96 MB
<none>                          <none>             75f6d31ce9e3  About a minute ago  425 MB
docker.io/library/alpine        3.15               02e0d6fdf486  7 weeks ago         5.62 MB
docker.io/library/golang        1.18.3-alpine3.15  d0dfc3912327  4 months ago        335 MB
[tomas@rpi4-qa tektoncd-plumbing]$ podman login -u="todabasi_openshift+robot" -p="***" quay.io
Login Succeeded!
[tomas@rpi4-qa tektoncd-plumbing]$ podman push quay.io/todabasi_openshift/tkn
Getting image source signatures
Copying blob 49c0bf81a962 done  
Copying blob 9833f26c6bda done  
Copying config c2988362f2 done  
Writing manifest to image destination
Storing signatures

it's available in https://quay.io/repository/todabasi_openshift/tkn?tab=tags

looks like a problem with ko and quay

[imjasonh]

If you docker login I'd expect this issue to go away. I don't believe it's an issue with quay, so much as it's an issue with ko detecting credentials configured using podman auth.

edit: I'd also expect this issue to happen with podman auth used to configure creds for any other registry.

Previously: https://github.com/google/go-containerregistry/pull/1185

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Keep fresh with the 'lifecycle/frozen' label.

[mzychaco]

I have encountered the same issue. Is there a solution available?

here is my log: Error: error processing import paths in "config/core/999-cache.yaml": error resolving image references: HEAD https://index.docker.io/v2/chacomo/queue-39be6f1d08a095bd076a71d288d295b6/blobs/sha256:c70f6b4a3c7bf06318af4b410af93194047834d9dbb519602987a3785737b8b7: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

[eloo-abi]

Hi there,

it looks like this is still not solved.

Can this issue please be reopened? It looks like ko can still not be used with podman

update:

just found that there is now a ko login command

with this command you can use ko with podman as well :)