Registries not hosted on / seem unsupported by `podman search`

Issue Description

We host a registry on an internal gitlab instance that runs on a custom port, resulting in a base URL that looks like gitlab.domain.tld:1234/group/repo The port is irrelevant, but I include it here for completeness' sake.

To ensure I'm logged in, I run podman login gitlab.domain.tld:1234 as well as podman login gitlab.domain.tld:1234/group/repo (the first one is usually enough, second one is to make sure)

running podman search gitlab.domain.tld:1234/group/repo/image results in :

ERRO[0000] error getting search results from v2 endpoint "gitlab.domain.tld:1234": requested access to the resource is denied 
Error: 1 error occurred:
        * couldn't search registry "gitlab.domain.tld:1234": requested access to the resource is denied

Adding gitlab.domain.tld:1234/group/repo to /etc/containers.registries.conf in unqualified-search-registries breaks podman completely to the point where even podman info results in this error:

Error: getting registries: loading registries configuration "/etc/containers/registries.conf": Invalid unqualified-search-registries entry "gitlab.domain.tld:1234/group/repo"

Steps to reproduce the issue

We'll use registry.gitlab.com/thelabnyc/dind:16.04 (found on https://gitlab.com/thelabnyc/dind/container_registry/95894) to try. It is an image that is publicly available.

Steps to reproduce the issue, scenario 1:

Add registry.gitlab.com/thelabnyc to your unqualified-search-registries
Run podman info

Steps to reproduce the issue, scenario 2:

Alternatively, without the unqualified-search-registries:

Run podman search registry.gitlab.com/thelabnyc/dind:16.04

Describe the results you received

For scenario 1:

> podman info
Error: getting registries: loading registries configuration "/etc/containers/registries.conf": Invalid unqualified-search-registries entry "registry.gitlab.com/thelabnyc"

For scenario 2:

> podman search registry.gitlab.com/thelabnyc/dind:16.04
ERRO[0001] error getting search results from v2 endpoint "registry.gitlab.com": requested access to the resource is denied 
Error: 1 error occurred:
        * couldn't search registry "registry.gitlab.com": requested access to the resource is denied

In this case, you can see that podman tries to search registry.gitlab.com instead of registry.gitlab.com/thelabnyc

Describe the results you expected

For scenario 1: podman info output For scenario 2: a search result from searching registry.gitlab.com/thelabnyc instead of registry.gitlab.com

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.7-0ubuntu22.04+obs15.24_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 98.82
    systemPercent: 0.39
    userPercent: 0.79
  cpus: 20
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: bbd-fsczyl3
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 10067
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 266585
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-71-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 27494141952
  memTotal: 33386106880
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun_101:1.8.4-0ubuntu22.04+obs55.10_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/266585/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/266585/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-0ubuntu22.04+obs10.63_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 1h 18m 19.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/username/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/username/.local/share/containers/storage
  graphRootAllocated: 1005393256448
  graphRootUsed: 188279689216
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/266585/containers
  transientStore: false
  volumePath: /home/username/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

Privileged Or Rootless

Rootless

Upstream Latest Release

Additional environment details

No response

Additional information

It is still perfectly possible to pull and run the images.

@vrothberg @mtrmac PTAL

I see no indication that “registries not hosted on /” are related… (I’d even say that such a thing cannot exist in the protocol at all; the API uses HTTP paths like https://host[:port]/v2, with no provision for nesting that API in deeper paths.)

Please provide a podman --log-level=debug output for the failing task (and don’t tinker with unqualified-search-registries, it’s entirely unrelated to podman search).

If you have a separate set of credentials for registry.example vs. registry.example/some/namespace, search is only going to use the registry.example credentials, because the rest of the “term” parameter to podman search (IIRC after the first /), is passed unmodified to the registry, has registry-specific semantics (possibly including metacharacters!) and in general just isn’t the same thing as a namespace.

E.g. it is expected for a podman search docker.io/busybox to find docker.io/library/busybox or for podman search registry.example/busybox to find registry.example/some-public-namespace/busybox; that clearly contradicts the idea of using credentials for the $registry/busybox namespace , if any.

and don’t tinker with unqualified-search-registries, it’s entirely unrelated to podman search)

I’m sorry, that’s not actually the case; it is used when the input term does not specify a registry. Either way, the reproducer commands do specify a registry, so that option doesn’t apply.

After podman login -u <user> -p <token> registry.gitlab.com/thelabnyc:

podman --log-level=debug search registry.gitlab.com/thelabnyc/dind:16.04
INFO[0000] podman filtering at log level debug
DEBU[0000] Called search.PersistentPreRunE(podman --log-level=debug search registry.gitlab.com/thelabnyc/dind:16.04)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/username/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/username/.local/share/containers/storage
DEBU[0000] Using run root /run/user/266585/containers
DEBU[0000] Using static dir /home/username/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/266585/libpod/tmp
DEBU[0000] Using volume path /home/username/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 61
INFO[0000] podman filtering at log level debug
DEBU[0000] Called search.PersistentPreRunE(podman --log-level=debug search registry.gitlab.com/thelabnyc/dind:16.04)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/username/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/username/.local/share/containers/storage
DEBU[0000] Using run root /run/user/266585/containers
DEBU[0000] Using static dir /home/username/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/266585/libpod/tmp
DEBU[0000] Using volume path /home/username/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 61
DEBU[0000] Searching images matching term thelabnyc/dind:16.04 at the following registries [registry.gitlab.com]
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf"
DEBU[0000] Found credentials for registry.gitlab.com in credential helper containers-auth.json in file /run/user/266585/containers/auth.json
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.gitlab.com
DEBU[0000] trying to talk to v1 search endpoint
DEBU[0000] GET https://registry.gitlab.com/v2/
DEBU[0000] Ping https://registry.gitlab.com/v2/ status 401
DEBU[0000] GET https://registry.gitlab.com/v1/search?n=25&q=thelabnyc%2Fdind%3A16.04
DEBU[0000] error getting search results from v1 endpoint "registry.gitlab.com": invalid status code from registry 404 (Not Found)
DEBU[0000] trying to talk to v2 search endpoint
DEBU[0000] GET https://gitlab.com/jwt/auth?account=username&service=container_registry
DEBU[0000] Increasing token expiration to: 60 seconds
DEBU[0000] GET https://registry.gitlab.com/v2/_catalog
DEBU[0001] Detected insufficient_scope error, will retry request with updated scope
DEBU[0001] GET https://gitlab.com/jwt/auth?account=username&scope=registry%3Acatalog%3A%2A&service=container_registry
DEBU[0001] Increasing token expiration to: 60 seconds
DEBU[0001] GET https://registry.gitlab.com/v2/_catalog
DEBU[0002] Discarding non-primary errors:
DEBU[0002]   unauthorized: authentication required
ERRO[0002] error getting search results from v2 endpoint "registry.gitlab.com": requested access to the resource is denied
Error: 1 error occurred:
        * couldn't search registry "registry.gitlab.com": requested access to the resource is denied

I'm not even sure whether gitlab offers any registry on registry.gitlab.com - the general idea is that each source code repository can have a container registry associated with it that lives on the same base path as the code repository.

For completeness' sake: adding a registry like registry.gitlab.com/thelabnyc to unqualified-search-registries and then trying podman search dind also results in the same error that podman info throws. The reason I experimented with adding it in there was the hope that podman would interpret the whole path as a registry.

(I think there’s some terminology confusion: A “registry” is the whole server on the host name. quay.io/mycompany is not a registry, it’s a namespace on the quay.io registry.)

Anyway, the conversation seems to confirm that you expect a nested-namespace set of credentials to be used, but as described above, and proven by

DEBU[0000] Found credentials for registry.gitlab.com in credential helper containers-auth.json in file /run/user/266585/containers/auth.json

(not … for registry.gitlab.com/thelabnyc), search is only using a whole-registry set of credentials. And that’s how it must be for search to be useful (… if search can ever be a recommended workflow at all, considering “supply chain” risks of using a malicious or or just an unmaintained image).

Agreed on the terminology confusion, thanks for clearing that up!

I do only have one set of credentials - my gitlab.com login and the access token I generated there with read_registry permissions (described as "Grants read-only access to container registry images on private projects.")

So I guess the problem lies with my use of search? If so, how would I go about searching for this particular image in this namespace in this registry? Or is this a gitlab (configuration or other) issue?

As a side note: my end goal here is getting the labels on a container in a registry hosted on our internal gitlab without having to download the whole thing. Being able to restrict search as much as possible (to at least registry, better to namespace) would hopefully limit the risks.

I would expect a podman login … registry.gitlab.com (not podman login … registry.gitlab.com/thelabnyc) to work here.

my end goal here is getting the labels on a container in a registry hosted on our internal gitlab without having to download the whole thing

Aren’t you looking for skopeo list-tags, then?

It seems like I was and completely missed the existence of skopeo - combined with skopeo inspect I can get all the information I need. Thank you for your patience with me. I guess the issue can be closed as a user error?

Thanks for the confirmation.

containers / podman