Google Artifact Registry Support

[hideto0710]

According to the official documentation, Google Artifact Registry supports OCI format. I tried to use ORAS with it. But I got Error: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized . docker push was success. So I think I have a proper authority. Is there anything I can do ?

# https://cloud.google.com/artifact-registry/docs/docker/quickstart#auth
gcloud auth configure-docker ${REGION}-docker.pkg.dev

oras push ${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/hello-artifact:v1 \
    --manifest-config /dev/null:application/vnd.acme.rocket.config.v1+json \
    --verbose \
    ./artifact.txt

Preparing artifact.txt
Uploading a948904f2f0f artifact.txt
WARN[0000] encountered unknown type application/vnd.acme.rocket.config.v1+json; children may not be fetched
WARN[0000] reference for unknown type: application/vnd.acme.rocket.config.v1+json  digest="sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" mediatype=application/vnd.acme.rocket.config.v1+json size=0
Error: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized

docker push

docker push ${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/quickstart-image:tag1
The push refers to repository [${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/quickstart-image]
3e207b409db3: Layer already exists
tag1: digest: sha256:xxxxx size: 528

[jzelinskie]

I'm not familiar with Google Cloud. Is this the same as GCR or a new product entirely? If it's new, it should probably get added to the OCI Distribution conformance test suite to measure how well it does actually support the standards.

[jdolitsky]

Very interesting. Will have to look a bit more.

You may want to use oras login using a generated access token. Here is an example of how to do that (for google container registry anyway):

      - name: Generate Google access token
        run: |
          echo ${GCLOUD_SERVICE_KEY} | base64 -d > ./credentials.json
          trap "rm -f ./credentials.json" EXIT
          gcloud auth activate-service-account REPLACEME@REPLACEME.iam.gserviceaccount.com --key-file="./credentials.json" --project=REPLACEME
          export GCLOUD_ACCESS_TOKEN="$(gcloud auth print-access-token)"
          echo "::add-mask::${GCLOUD_ACCESS_TOKEN}"
          echo "::set-env name=GCLOUD_ACCESS_TOKEN::${GCLOUD_ACCESS_TOKEN}"
        env:
          GCLOUD_SERVICE_KEY: ${{ secrets.GCLOUD_SERVICE_KEY }}
      - name: Run OCI Distribution Spec conformance tests
        uses: opencontainers/distribution-spec@master
        env:
          OCI_ROOT_URL: https://gcr.io
          OCI_USERNAME: oauth2accesstoken
          OCI_PASSWORD: ${{ env.GCLOUD_ACCESS_TOKEN }}
          OCI_NAMESPACE: REPLACEME/REPLACEME
          OCI_TEST_PUSH: 1
          OCI_HIDE_SKIPPED_WORKFLOWS: 1

[hideto0710]

Thanks for your quick reply!

Is this the same as GCR or a new product entirely?

Artifact Registry is the evolution of GCR. Now, it is in a pre-release state.

You may want to use oras login using a generated access token.

Thanks, I'll try.

For your information:

According to GCR documentation, we have four options. I tried first and second option. I set breakpoint at pkg/auth/docker/resolver.go#L44. Then I got AuthConfig values as below.

gcloud as a Docker credential helper

Username: "_dcgcloud_token"
Password: "xx00.yyy--zzz"

Standalone Docker credential helper

Username: "_dcgcr_2_0_0_token"
Password: "xx00.yyy--zzz"

[jdolitsky]

@hideto0710 see if you can use hardcoded "oauth2accesstoken" as the username. This is what we needed to do for GCR. I'd have to assume the auth works all the same given the service account used is granted access to this new artifact service.

[hideto0710]

I tried as below. But I got a same error.

$ export REGION=asia-northeast1

$ export PROJECT=xxx

$ gcloud auth activate-service-account oras-test@${PROJECT}.iam.gserviceaccount.com \
  --key-file="./credentials.json" \
  --project=${PROJECT}

$ export GCLOUD_ACCESS_TOKEN="$(gcloud auth print-access-token)"

$ docker login -u oauth2accesstoken -p ${GCLOUD_ACCESS_TOKEN} ${REGION}-docker.pkg.dev
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /Users/hideto0710/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

$ docker push ${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/quickstart-image:tag1
The push refers to repository [${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/quickstart-image]
3e207b409db3: Layer already exists
tag1: digest: sha256:xxx size: 528

$ oras login -u oauth2accesstoken -p ${GCLOUD_ACCESS_TOKEN} ${REGION}-docker.pkg.dev
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

$ oras push ${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/hello-artifact:v1 \
    --manifest-config /dev/null:application/vnd.acme.rocket.config.v1+json \
    --verbose \
    ./artifact.txt
WARN[0000] encountered unknown type application/vnd.acme.rocket.config.v1+json; children may not be fetched
WARN[0000] reference for unknown type: application/vnd.acme.rocket.config.v1+json  digest="sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" mediatype=application/vnd.acme.rocket.config.v1+json size=0
Error: failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized

Should I create GitHub workflow first refer to this? https://github.com/opencontainers/oci-conformance/blob/master/.github/workflows/gcr.yml

[DazWilkin]

You can use the oauth2accesstoken approach per @jdolitsky above to login:

REGISTRY="gcr.io"

gcloud auth print-access-token \
| oras login -u oauth2accesstoken --password-stdin https://${REGISTRY}

and then:

more ./artifact.txt 
Hello Freddie!

oras push gcr.io/${PROJECT}/hello-artifact:v1 ./artifact.txt
Uploading 0060e7e892de artifact.txt
Pushed gcr.io/${PROJECT}/hello-artifact:v1
Digest: sha256:c330be5cf8bbc34f87db380aaba72e774d7ee1064085b3e2ec2b071be292e2a2

rm artifact.txt 

oras pull gcr.io/${PROJECT}/hello-artifact:v1
Downloaded 0060e7e892de artifact.txt
Pulled gcr.io/${PROJECT}/hello-artifact:v1
Digest: sha256:c330be5cf8bbc34f87db380aaba72e774d7ee1064085b3e2ec2b071be292e2a2

cat ./artifact.txt 
Hello Freddie!

[bacongobbler]

@DazWilkin I think you're referring to Google Container Registry, right? @hideto0710 is trying to use a new product called Google Artifact Registry. Have you tried the same steps with Google Artifact Registry yet?

[DazWilkin]

That's correct but -- IIUC -- Artifact Registry is a veneer over Container Registry and so either should work.

[hideto0710]

Maybe I found difference between Container Registry and Artifact Registry! First, we call HEAD request. And we check Www-Authenticate header.

Container Registry

Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:PROJECT/IMAGE:pull"

Artifact Registry

Bearer realm="https://us-docker.pkg.dev/v2/token",service="us-docker.pkg.dev",scope=repository:PROJECT_ID/REPO/IMAGE:pull

In header from Artifact Registry, scope is not quated. After that, we call GET request with scope=repository, and got 401.

https://us-docker.pkg.dev/v2/token?account=oauth2accesstoken&scope=repository&scope=repository%3APROJECT%2FREPO%2FIMAGE%3Apush%2Cpull

https://github.com/containerd/containerd/blob/v1.3.4/remotes/docker/auth.go#L166-L168

So, I think this is Artifact Registry issue. ( or we have to work around in containerd/containerd.)

[hideto0710]

Container Registry

$ curl --head "https://gcr.io/v2/project/image/blobs/sha256:xxxx"
HTTP/2 401
docker-distribution-api-version: registry/2.0
content-type: application/json
www-authenticate: Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:project/image:pull"
date: Tue, 05 May 2020 04:52:25 GMT
server: Docker Registry
cache-control: private
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
accept-ranges: none
vary: Accept-Encoding

Artifact Registry

$ curl --head "https://us-docker.pkg.dev/v2/project/repo/image/blobs/sha256:xxxx"
HTTP/2 401
content-type: application/json
www-authenticate: Bearer realm="https://us-docker.pkg.dev/v2/token",service="us-docker.pkg.dev",scope=repository:project/repo/image:pull
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0
date: Tue, 05 May 2020 04:52:05 GMT
content-length: 102
alt-svc: h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

[DazWilkin]

@hideto0710 @bacongobbler let me know if there's anything you'd like me to test.

I'm a former Googler and very familiar Cloud Platform (though Artifact Registry is after my time!).

If there is a discrepancy (as appears the case), please file an issue with Google's Issue Tracker.

I know many of the Cloud PMs and can ping them too.

[DazWilkin]

Plus, I'd like to ensure Google's services are fairly represented by this (very good) tool, by wasm-to-oci and Krustlet which transitively depend upon ORAS and which brought me here :-)

[hideto0710]

@DazWilkin Thank you for your kindness! Yesterday, I sent a message to GCP Customer Support. If there are any update, I will comment t here. Thank you telling me about issue tracker. I'll use next time!

[hideto0710]

It has been fixed.

$ curl --head "https://us-docker.pkg.dev/v2/project/repo/image/blobs/sha256:xxxx"
HTTP/2 401
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://us-docker.pkg.dev/v2/token",service="us-docker.pkg.dev",scope="repository:project/repo/image:pull"

And, I got a different error.

$ oras push ${REGION}-docker.pkg.dev/${PROJECT}/quickstart-docker-repo/hello-artifact:v1 \
>     --verbose \
>     ./artifact.txt
Preparing artifact.txt
Uploading a948904f2f0f artifact.txt
WARN[0000] encountered unknown type application/vnd.unknown.config.v1+json; children may not be fetched
WARN[0000] reference for unknown type: application/vnd.unknown.config.v1+json  digest="sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a" mediatype=application/vnd.unknown.config.v1+json size=2
Error: failed commit on ref "unknown-sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a": invalid content digest in response: invalid checksum digest format

[jonjohnsonjr]

I'm looking into this now.

[SteveLasker]

Once this is fixed, we'd certainly welcome an addition to implementations.md with a PR similar to the recent AWS PR

[jonjohnsonjr]

It hasn't fully rolled out yet (should be next week, barring any complications), but I've landed a fix that should be live now in several regions, e.g. us-west2 and asia-east1:

$ oras push us-west2-docker.pkg.dev/<project>/test/oras README.md
Uploading d65088b448d9 README.md
Pushed us-west2-docker.pkg.dev/<project>/test/oras
Digest: sha256:20fbd468380799b6396082262eba839d5f57aad6c09985ef3c3dc7148d4a8a28

@hideto0710 please try it out and let me know if you're still seeing issues.

[hideto0710]

@jonjohnsonjr Thanks ! I tried it in asia-northeast1 and It worked! This issue should be closed!

[jacek-jablonski]

@jonjohnsonjr should I expect any problems yet in europe-west4?

 => ERROR exporting to image                                                                                      11.1s
 => => exporting layers                                                                                            0.0s
 => => exporting manifest sha256:d44d6d8774bf7da207749c50ce5b3acc751a0c11b745d6b06e2a8a54f839e201                  0.0s
 => => exporting config sha256:cb81191ed15e2b86550dbf03b123fda6d8b0d685404dd35e14ae88dc30cb7004                    0.0s
 => => pushing layers                                                                                             10.2s
 => => pushing manifest for europe-west4-docker.pkg.dev/xxx/appconfig:0.1.3                              0.8s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:d44d6d8774bf7da207749c50ce5b3acc751a0c11b745d6b06e2a8a54f839e201": unexpected status: 400 Bad Request

[jonjohnsonjr]

should I expect any problems yet in europe-west4?

No, this has been rolled out everywhere, so I would guess this is a separate issue. Do you have the full error response? Looking at https://github.com/docker/build-push-action/issues/171, it seems like you might just not have enough components in your image name.