`podman pull` on macOS produces `credentials not found in native keychain` error even when logged in

[spkane]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

On macOS, registry credentials do not appear to work properly.

Steps to reproduce the issue:

1. Use podman to login to docker.io

$ podman login docker.io
Authenticating with existing credentials for docker.io
Existing credentials are valid. Already logged in to docker.io

2. Try to pull and image with podman

$ podman pull docker.io/spkane/outyet:latest
Error: 1 error occurred:
    * credentials not found in native keychain

3. Try to pull the same image with docker

$ docker pull docker.io/spkane/outyet:latest
latest: Pulling from spkane/outyet
2fdfe1cd78c2: Pull complete
e4b6be3c5603: Pull complete
fbdcddda74d2: Pull complete
Digest: sha256:111273c4d33ff75b8e6566c40e4178fcc579e88704a186ed17f4b23ea892de30
Status: Downloaded newer image for spkane/outyet:latest
docker.io/spkane/outyet:latest

Describe the results you received:

Error: 1 error occurred:
    * credentials not found in native keychain

Describe the results you expected:

• The image should have pulled fine.

Additional information you deem important (e.g. issue happens only occasionally):

• I can get a podman pull to work if I pass the credentials in on the command line.
• However even after doing this, I can still not run that container, as I get the exact same error message, even though the container has already been pulled.

$ podman run -it --rm docker.io/spkane/outyet:latest
Error: 1 error occurred:
    * credentials not found in native keychain

Output of podman version:

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 12:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 13:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.13-200.fc34.x86_64
  linkmode: dynamic
  memFree: 1595199488
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 40m 14.68s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 20:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew list
==> Formulae
adns                libssh
aom             libssh2
argo                libtasn1
argocd              libtiff
autoconf            libtool
automake            libunistring
aws-iam-authenticator       libusb
aws-sam-cli         libusb-compat
awscli              libuv
bash                libvidstab
bash-completion         libvorbis
bat             libvpx
bdw-gc              libx11
bfg             libxau
brotli              libxcb
c-ares              libxdmcp
cairo               libxext
cmake               libxrender
coreutils           libyaml
curl                linkerd
dav1d               little-cms2
direnv              lzo
docker-credential-helper-ecr    m4
dockerize           mas
dyff                minikube
eksctl              moreutils
etcd                mpdecimal
fasd                mpfr
ffmpeg              ncurses
flac                nettle
flatbuffers         nghttp2
fontconfig          node
freetype            npth
frei0r              oniguruma
fribidi             opencore-amr
fzf             openjdk
gawk                openjpeg
gdbm                openldap
gettext             openssl@1.1
giflib              opus
gist                p11-kit
git-crypt           pcre
git-flow-avh            pcre2
git-lfs             pinentry
git-toolbelt            pipenv
glib                pipx
gmp             pixman
gnu-getopt          pkg-config
gnu-sed             podman
gnupg               popeye
gnutls              protobuf
go              python@3.8
gobject-introspection       python@3.9
golangci-lint           qemu
grafana             rav1e
graphite2           rbenv
grep                readline
grpcurl             ripgrep
guile               rtmpdump
harfbuzz            rubberband
helm                ruby
hidapi              ruby-build
hub             rustup-init
hugo                sdl2
icu4c               six
jansson             skaffold
jemalloc            snappy
jid             sonobuoy
jpeg                sops
jq              speex
k3d             sqlite
k9s             srt
kind                starship
krew                stern
kubebuilder         tcl-tk
kubectl-argo-rollouts       tesseract
kubectx             tfenv
kubernetes-cli          thefuck
kubie               theora
kustomize           tinygo
lame                tmux
lazydocker          torsocks
lazygit             unbound
leptonica           utf8proc
libass              vde
libassuan           velero
libbluray           watch
libev               webp
libevent            wget
libffi              whalebrew
libgcrypt           wtfutil
libgpg-error            x264
libidn2             x265
libksba             xorgproto
libmetalink         xvid
libogg              xz
libpng              yarn
libpthread-stubs        youtube-dl
libsamplerate           yq
libslirp            zeromq
libsndfile          zimg
libsodium           zstd
libsoxr

==> Casks
1password-cli   flutter     keybase     stats
aws-vault   font-fira-code  macfuse

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

• macOS BigSur
  • Homebrew
  • qemu

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ashley-cui PTAL

[vrothberg]

@mtrmac may be able to help track the bug down on Mac OS.

[mtrmac]

• Are there any credential helpers configured in ~/.docker/config.json or ~/.config/containers/auth.json? In registries.conf?
• Logs of podman --log-level=debug login and podman --log-level=debug pull could help reveal what is being read/written.

[spkane]

Thank you for that feedback @mtrmac. It helped me narrow down the issue.

• Something very close to this was in my ~/.docker/config.json:

...  
"credHelpers" : {
    "000000000000.dkr.ecr.us-west-2.amazonaws.com" : "ecr-login"
  },
...

Removing it works around the reported problem, but is obviously not the desired solution.

After looking at the debugging below, I saw this:

DEBU[0000] Error looking up credentials for 000000000000.dkr.ecr.us-west-2.amazonaws.com in credential helper containers-auth.json: credentials not found in native keychain

So, it appears to be trying to lookup the credentials for that registry even though I am not trying to pull from registry. And to be fair, that is an old AWS ECR registry so I can pull it out, but it feels like there might still be a bug here, since it is looking for credentials for a registry that I am not trying to interact with.

Debug Output

$ podman login --log-level=debug docker.io

INFO[0000] podman filtering at log level debug
DEBU[0000] Called login.PersistentPreRunE(podman login --log-level=debug docker.io)
DEBU[0000] SSH Ident Key "/Users/spkane/.ssh/podman-machine-default" SHA256:... ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.ru7ZJV89WE/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:... ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.1/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for docker.io in credential helper containers-auth.json
Authenticating with existing credentials for docker.io
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0000] GET https://auth.docker.io/token?account=spkane&service=registry.docker.io
DEBU[0001] GET https://registry-1.docker.io/v2/
Existing credentials are valid. Already logged in to docker.io
DEBU[0001] Called login.PersistentPostRunE(podman login --log-level=debug docker.io)

$ podman --log-level=debug pull docker.io/spkane/outyet:latest

INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level=debug pull docker.io/spkane/outyet:latest)
DEBU[0000] SSH Ident Key "/Users/spkane/.ssh/podman-machine-default" SHA256:... ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.ru7ZJV89WE/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:... ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.1/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for docker.io in credential helper containers-auth.json
DEBU[0000] Error looking up credentials for 000000000000.dkr.ecr.us-west-2.amazonaws.com in credential helper containers-auth.json: credentials not found in native keychain
Error: 1 error occurred:
    * credentials not found in native keychain

[mtrmac]

Is this a “remote” Podman connecting to a remote server?

  serviceIsRemote: true

In that case, IIRC podman pull on the client collects all credentials and sends them to the server, even for registries not specifically named on the command line (@vrothberg please confirm or correct me); that’s necessary to support mirrors (… without excessive implementation complexity, at least).

[spkane]

I am running podman machine start on my Mac. I suppose that could be considered remote, since it is a VM by necessity.

[spkane]

If what you suspect is true, maybe simply printing a warning would suffice if it can't find credentials for one of the registries, and leave it up to the remote system to throw an error if it actually fails to auth against the registry...

[mtrmac]

I have filed https://github.com/containers/image/issues/1406 to add more context to the reported error.

As for whether failures to get credentials should be non-fatal, I’ll let Podman maintainers decide.

[rhatdan]

It should not be fatal, all we care about is pulling the image, This should be a warning.

[rhatdan]

@vrothberg PTAL, this seems to be more podman-remote then podman machine.

[oofnikj]

Thank you for that feedback @mtrmac. It helped me narrow down the issue.

• Something very close to this was in my ~/.docker/config.json:

...  
"credHelpers" : {
    "000000000000.dkr.ecr.us-west-2.amazonaws.com" : "ecr-login"
  },
...

I ran into this issue today. To work around it I ended up removing the Docker-style configuration from ~/.docker/config.json and instead adding the equivalent podman configuration to ~/.config/containers/registries.conf:

credential-helpers = [ "ecr-login" ]

Now I can pull both images that do not require auth as well as ECR images.

[vrothberg]

There's actually a bug in c/image. It's totally 0K when a credential helper doesn't have credentials but the error should be checked for and we do, but just not in this specific code path.