Can't push to GCR using username / password

[aholyoake-bc]

We upgraded to concourse 7.1.0 yesterday (registry-image-resource 1.2.0) and we are now having problems with pipelines using the registry-image resource to push to a private google container registry.

The credentials are still working fine for any pipelines still using the docker-image resource.

Pipeline resource definition:

- icon: docker
  name: service-image
  source:
    password: ((docker-credentials))
    repository: eu.gcr.io/XXX-project-XXX/service-image
    username: _json_key
  type: registry-image

Error message:

ERRO[0003] pushing image failed: pushing tag(s): PATCH https://eu.gcr.io/v2/XXX-project-XXX/service-image/blobs/uploads/AJAD5v0TaF8_kTQx2bOCfvnSIcxKmqfK5U2dQ88-om9B-uIlbF3aGYvJVzQ30qEcf86vftNWMpGEQFQ6bJGVGpQ: DENIED: Access denied.

I've manually added the rootfs / resource_manifest to our concourse workers for v1.0.0 and everything works as expected

[aholyoake-bc]

It looks like the change here is to blame: https://github.com/concourse/registry-image-resource/commit/b28e33ac454ab1351c6104338ec4f2e31a304fd6?branch=b28e33ac454ab1351c6104338ec4f2e31a304fd6&diff=split#diff-7122235fbfe9d72d2c145a8af13eca5541d36648e4e629b0e3fe4cd80aafd187R197

[xFJA]

I got the same error. I thought it was related to permissions issues on Google Cloud but after a research of the error, I've found this ticket. It would be great to fix it soon.

[tiredpixel]

I get this also on 7.1.0 as of today, authenticating against a private self-hosted Nexus Repository Manager. I also got it previously on 7.0.0. However 6.7.5 works fine.

[muntac]

Can confirm the maintainers team has seen this on our end as well. It's likely as noted above due to the remote.MultiWrite change. We've put this on our backlog but you're welcome to investigate further.

[aoldershaw]

a-ha - think I've pinpointed it. created this simple Go program to reproduce what the registry-image resource is doing on put:

package main

import (
    "net/http"
    "os"

    "github.com/google/go-containerregistry/pkg/authn"
    "github.com/google/go-containerregistry/pkg/name"
    "github.com/google/go-containerregistry/pkg/v1/random"
    "github.com/google/go-containerregistry/pkg/v1/remote"
    "github.com/google/go-containerregistry/pkg/v1/remote/transport"
)

func main() {
    repository, ok := os.LookupEnv("REPOSITORY")
    if !ok {
        panic("must set REPOSITORY")
    }

    serviceAccountKey, ok := os.LookupEnv("GCP_SERVICE_ACCOUNT_KEY")
    if !ok {
        panic("must set GCP_SERVICE_ACCOUNT_KEY")
    }

    repo, err := name.NewRepository(repository)
    if err != nil {
        panic(err)
    }

    img, err := random.Image(1024, 1)
    if err != nil {
        panic(err)
    }

    auth := &authn.Basic{
        Username: "_json_key",
        Password: serviceAccountKey,
    }

    tr := http.DefaultTransport.(*http.Transport)

    rt, err := transport.New(repo.Registry, auth, tr, []string{repo.Scope(transport.PullScope)})
    if err != nil {
        panic(err)
    }

    images := map[name.Reference]remote.Taggable{
        repo.Tag("latest"): img,
    }

    err = remote.MultiWrite(images, remote.WithAuth(auth), remote.WithTransport(tr), remote.WithTransport(rt))
    if err != nil {
        panic(err)
    }
}

What seems to be causing the error is the remote.WithTransport(rt), which was introduced in #264 - without that, we're able to push fine. I'm guessing it has something to do with requesting PullScope only