Oracle registry: failed to authorize: failed to fetch anonymous token

diegosiao commented 1 year ago

Issue Description

Running any docker-compose command that requires image pulling causes the error below:

failed to solve: iad.ocir.io/compartment/repo/image:tag: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://iad.ocir.io/20180419/docker/token?scope=%3A%3A&scope=repository%3Acompartment%2Frepo%2Fimage%3Apull&service=iad.ocir.io: 400 Bad Request

IMPORTANT: I noticed that the URL parameter scope is duplicated, taking the URL output on error and removing the first scope=%3A%3A parameter from it gives me a SUCCESSFUL 200 response like:

curl "https://iad.ocir.io/20180419/docker/token?scope=repository%3Acompartment%2Frepo%2Fimage%3Apull&service=iad.ocir.io"

Response (the token is redacted):

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.ey(...)0aW9uc1wiOltcInB1bGxcIl19XSIsImF1ZCI6ImlhZC5vY2lyLmlvIn0.hSj1OP_OIM-WGJjiYVVH8nj3W7AMPgMlL11QDN6PQSA","scope":"repository:compartment/repo/image:pull","expires_in":3600}

Steps to reproduce the issue

Add an Oracle registry on podman (iad.ocir.io)
Run any docker-compose command that requires image pulling (eg.: build, up)

Describe the results you received

failed to solve: iad.ocir.io/compartment/repo/image:tag: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://iad.ocir.io/20180419/docker/token?scope=%3A%3A&scope=repository%3Acompartment%2Frepo%2Fimage%3Apull&service=iad.ocir.io: 400 Bad Request

Describe the results you expected

A successful docker-compose build execution.

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.83
    systemPercent: 0.06
    userPercent: 0.11
  cpus: 24
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "38"
  eventLogger: journald
  freeLocks: 2046
  hostname: diegodesktop
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.90.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 31137153024
  memTotal: 33535012864
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.9.2-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589934592
  swapTotal: 8589934592
  uptime: 0h 30m 1.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - iad.ocir.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 12234330112
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 20
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695839078
  BuiltTime: Wed Sep 27 15:24:38 2023
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Luap99 commented 1 year ago

@vrothberg @mtrmac PTAL

mtrmac commented 1 year ago

failed to fetch anonymous token is not a string that exists in Podman; it does exist in Docker, and maybe in docker-compose itself (I didn’t check that one in detail, but it seems to call the same dependencies.)

So, to the extent that this is a registry client using the HTTP API incorrectly/incompatibly, my first guess is that Podman is not involved at all / can’t affect that implementation. (I didn’t try actually reproducing this, though.)

diegosiao commented 1 year ago

The registries information are on podman side. The token URL is not built by podman and forwarded to docker-compose?

mtrmac commented 1 year ago

No.

diegosiao commented 1 year ago

It is hard to make sense of this information. Docker compose works fine with docker daemon, but not with podman :(

Any workaround/suggestion?

vrothberg commented 1 year ago

Can you share an exact reproducer? I am sure we'll get to the bottom of it.

diegosiao commented 1 year ago

Thank you @vrothberg, appreciate your help! I would be happy to share a reproducer, but doing that will require sharing an image on Oracle Cloud private registry, so I will try to setup a personal account there and everything required to reproduce this error.

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

Luap99 commented 7 months ago

Given there was no reproducer shared and the error message does not seem to be from podman, closing

containers / podman