Using ORAS from snap can't authenticate with gcloud credential helper

[jteichroeb-oanda]

What happened in your environment?

I'm using oras like this: oras push us-central1-docker.pkg.dev/... file.tar.gz

I have docker configured to use gcloud auth using this command: gcloud auth configure-docker us-central1-docker.pkg.dev

which sets up the ~/.docker/config.json like so:

{
  "credHelpers": {
    "us-central1-docker.pkg.dev": "gcloud"
  }
}

If I use oras from snap, authentication fails, but if I use the downloaded oras binary it works fine

What did you expect to happen?

Authentication doesn't depend on

How can we reproduce it?

Since this requires authenticating with google cloud platform, replicating my setup isn't the easiest. I can help someone setup a similar environment if necessary

What is the version of your ORAS CLI?

1.0.0, which is the latest version in snap

What is your OS environment?

Ubuntu 20.04

Are you willing to submit PRs to fix it?

• [ ] Yes, I am willing to fix it.

[qweeah]

Can you attach the debug log of failed oras push? You can get that with --debug flag set.

[jteichroeb-oanda]

Here's the debug log, overall not the most helpful. I've redacted some URLs with [...]

DEBU[0000] Request #0
> Request URL: "https://us-central1-docker.pkg.dev/v2/[...]/manifests/sha256:18aace200d0ad2bf20e9beed69026b8a82a0db94324080ec66731fd2e270e67e"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.0.0" 
DEBU[0000] Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "X-Xss-Protection": "0"
   "Date": "Fri, 25 Aug 2023 12:59:53 GMT"
   "Content-Length": "102"
   "Content-Type": "application/json; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Www-Authenticate": "Bearer realm=\"https://us-central1-docker.pkg.dev/v2/token\",service=\"us-central1-docker.pkg.dev\",scope=\"repository:[...]:pull\""
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN" 
DEBU[0000] Request #1
> Request URL: "https://us-central1-docker.pkg.dev/v2/token?scope=repository%3A[...]%3Apull&service=us-central1-docker.pkg.dev"
> Request method: "GET"
> Request headers:
   "User-Agent": "oras/1.0.0" 
DEBU[0000] Response #1
< Response Status: "403 Forbidden"
< Response headers:
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Date": "Fri, 25 Aug 2023 12:59:53 GMT"
   "Content-Length": "226"
   "Content-Type": "application/json; charset=utf-8" 
Error: HEAD "https://us-central1-docker.pkg.dev/v2/[...]/manifests/sha256:18aace200d0ad2bf20e9beed69026b8a82a0db94324080ec66731fd2e270e67e": GET "https://us-central1-docker.pkg.dev/v2/token?scope=repository%3A[...]%3Apull&service=us-central1-docker.pkg.dev": response status code 403: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/[...]/locations/us-central1/repositories/[...]" (or it may not exist)

The only difference on a successful debug log is that the second request has an additional header

"Authorization": "*****"

The debug logs don't show how it gets this header

[qweeah]

Thank @jteichroeb-oanda The logs at least show snap build doesn't pick up the stored creds from credHelper.

[qweeah]

@jteichroeb-oanda Just to confirm when you using downloaded oras binary and it succeeds, is it 1.0.0 or v1.0.1?

[jteichroeb-oanda]

@qweeah both 1.0.0 and 1.0.1 work. I can't try 1.0.1 with snap since the lastest I see is 1.0.0.

[qweeah]

Got it, taking a look. Thanks for the input. @jteichroeb-oanda

[FeynmanZhou]

Upgrading the ORAS installer to v1.0.1 on Snap is still in WIP and tracked in #965 .

[qweeah]

@jteichroeb-oanda snap apps uses separated config file. You should add cred helper to the docker config in snap's virtual environment ~/snap/oras/current/.docker/config.json and it should work.

[jteichroeb-oanda]

@qweeah that doesn't quite work either, it doesn't have a way to call the credential helper:

Error: failed to resolve latest: GET "https://us-central1-docker.pkg.dev/v2/[...]/manifests/latest": error getting credentials - err: docker-credential-gcloud resolves to executable in current directory (./docker-credential-gcloud), out: ``

[qweeah]

Looks like the credential helper needss to be accessible within the snap sandbox. I tried --classic installation and get the same error

Error: error storing credentials - err: docker-credential-pass resolves to executable in current directory (./docker-credential-pass), out: ...

@FeynmanZhou Maybe we should suggest user not to use snap release if they are using credstore or credhelpers, or even remove snap from the installation guide.

[FeynmanZhou]

@qweeah If there is no workaround for the issue caused by the Snap sandbox environment, we can consider removing the Snap installation from the installation guide and dropping its maintenance for future releases.

@SamirPS Do you have any suggestions on this issue?

[shizhMSFT]

Snap or Snappy is a package management system in Ubuntu with applications containerized and sandboxed.

Since a snap application runs in a sandbox environment, snap version of oras is not able to read the docker config and credential helpers directly and thus causing user experience problems.

[!NOTE] The home folder of the snap version of oras is ~/snap/oras/current.

[jteichroeb-oanda]

One thing that confuses me, I have the gcloud command installed from snap as well, if it can access files outside of the sandbox environment, can't oras do the same thing?

[shizhMSFT]

oras can access the files outside of the sandbox. For example, oras push <ref> file1 file2 where file1 and file2 can be at any place. However, oras cannot find docker-credential-gcloud in $PATH as $PATH has been modified by snap.

BTW, the error message docker-credential-gcloud resolves to executable in current directory (./docker-credential-gcloud) of oras v1.0.0 might be confusing. It is fixed in v1.0.1 as exec: "docker-credential-gcloud": executable file not found in $PATH.

[shizhMSFT]

Currently, oras is configured to strict confinement level.

https://github.com/oras-project/oras/blob/999ac385ed704b1cb0dfa97c7b88e2ce7d290c3c/snapcraft.yaml#L20

I'm not sure if setting confinement level to classic and install using --classic works or not. Noto to @qweeah, --classic flag does not work as it is not declared as classic.

Another thing we can try is the snap interfaces, mounting :home to oras:home.

[shizhMSFT]

Oh. The :home is already connected.

$ snap connections oras
Interface  Plug          Slot      Notes
home       oras:home     :home     -
network    oras:network  :network  -

[SamirPS]

can it's been link to this ? https://forum.snapcraft.io/t/kubectl-wrong-gcloud-path-when-doing-any-operations/18848

i'm checking with other people to know the issue

[SamirPS]

Normally, with this interface, the snap will work: https://snapcraft.io/docs/personal-files-interface , I will update the snapcraft.yaml file and let's you test it.

[qweeah]

@SamirPS I don't think using the personal file interface will be a good solution since user might use any config file via --config and specify any cred helper in the configuration. During build time, we cannot list out every possible path.

[qweeah]

I'm not sure if setting confinement level to classic and install using --classic works or not. Noto to @qweeah, --classic flag does not work as it is not declared as classic.

Tried building a snapcraft version locally with confinement level set to classic and it worked: both config file and cred binaries can be successfully invoked. I think this is the most ideal solution we currently have.

[SamirPS]

thanks for the head up, i let you do the PR and publish it then :)

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[qweeah]

Putting it to future milestone to avoid being flagged as stale. To resolve issue here, we tried to convert oras a classic snap app and it's pending snapcraft's approval, see https://forum.snapcraft.io/t/request-for-classic-confinement-oras-and-oras-test/36826