Unable to authenticate with podman-remote over ssh to drive remote podman.sock

[FreedomBen]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to run Podman on a remote machine and use the podman-remote client to drive it.

Following instructions here: https://github.com/containers/podman/blob/master/docs/tutorials/mac_win_client.md

Podman on the remote machine seems to be working fine, but it cannot be driven by the local podman because the local Podman fails to authenticate properly over SSH.

Steps to reproduce the issue:

1. Setup podman on remote machine per: https://github.com/containers/podman/blob/master/docs/tutorials/mac_win_client.md
2. Ensure correct SSH key is in ssh agent
3. Add remote connection to podman: podman system connection add test ssh://192.168.122.1/run/user/1000/podman/podman.sock
4. Attempt to drive remote podman from local machine: podman-remote ps

Describe the results you received:

Authentication error:

Error: Failed to create sshClient: Connection to bastion host (ssh://ben@192.168.2.186:22/run/user/1000/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Describe the results you expected:

I expected podman-remote ps to behave normally, outputting something like this:

[ben@benssystem76 ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

Additional information you deem important (e.g. issue happens only occasionally):

I also tried adding the connection with an explicit identity file instead of relying on the SSH agent:

podman system connection add test2 --identity /home/ben/.ssh/id_rsa ssh://192.168.122.1/run/user/1000/podman/podman.sock

When running podman-remote ps with that connection, I am prompted for the passphrase for the SSH key (as I would expect) but I get the same error message indicating that authentication failed, I think because podman didn't do the SSH handshake properly or something:

Error: Failed to create sshClient: Connection to bastion host (ssh://ben@192.168.2.186:22/run/user/1000/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I also checked to make sure that /run/user/1000/podman/podman.sock existed on the remote machine, and it did. Remote user id is 1000 as expected.

Output of podman version:

[ben@benssystem76 config-files]$ podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14.9
Built:        Wed Sep 30 13:31:11 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: benssystem76
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.18-200.fc32.x86_64
  linkmode: dynamic
  memFree: 1297784832
  memTotal: 33637113856
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 16869486592
  swapTotal: 16869486592
  uptime: 30h 0m 31.03s (Approximately 1.25 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/ben/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/ben/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 601
  runRoot: /run/user/1000/containers
  volumePath: /home/ben/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601494271
  BuiltTime: Wed Sep 30 13:31:11 2020
  GitCommit: ""
  GoVersion: go1.14.9
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-7.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Remote podman is the same latest version (2.1.1), running on an F33 Server that is in a KVM VM on a Dell R620 host. Local podman version 2.1.1 is on an F32 Workstation with Gnome 3, and a nice photo of my family as the wallpaper background and screensaver.

[Luap99]

Does podman system connection add ... fail and did you run this on the client or server?

[Luap99]

Can you provide the output of podman system connection ls?

[FreedomBen]

@Luap99 thanks for looking into this. podman system connection add does not fail. I ran it on the client. Output of podman system connection ls is:

[ben@benssystem76 ~]$ podman system connection ls
Name    Identity               URI
test3*  /home/ben/.ssh/id_rsa  ssh://ben@192.168.2.186:22/run/user/1000/podman/podman.sock

That is the correct username and IP for the remote machine, and the SSH key is in the client ssh agent. SSHing normally works fine.

[Luap99]

I followed the steps and it worked for me. Not sure what's going on here if normal ssh works.

@baude @jwhonce any ideas? This issue has five thumbs up so I guess more people are running into this.

[FreedomBen]

@Luap99 I don't know if it helps, but I dug into it a bit and the error message comes from a dependency, line 77 of vendor/golang.org/x/crypto/ssh/client_auth.go

For convenience again this is the error message:

Error: Failed to create sshClient: Connection to bastion host (ssh://ben@192.168.2.186:22/run/user/1000/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I wondered if the SSH connection doesn't even try to authenticate with the key. If the key isn't in the ssh-agent then it will ask for the passphrase but then fails the same way.

I thought too maybe a possible bug in crypto/ssh or if we aren't doing it right in pkg/bindings/connection.go but I didn't see anything obvious, and it appears that crypto/ssh is used extensively by large projects, so a bug like this feels unlikely.

[jwhonce]

That error is coming from pkg/bindings/connection.go:252. The go ssh client is failing to connect to the remote sshd. It currently uses either public key (directly or via ssh-agent) or password authentication methods. Given your connection string it should be using public key. If you add --log-level=debug option, you should see debugging events printed to the screen showing you additional information on the identity parsing. This debugging will also inform you if an ssh-agent has been found. But I doubt that with the error message including [none publickey] methods.

[ssbarnea]

$ podman --log-level=debug ps                                                                                                    [9:59:16]
INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps)
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.VBspeEuRIU/Listeners", ssh-agent signer enabled
Error: Failed to create sshClient: Connection to bastion host (ssh://root@leno/run/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
FAIL: 125

And ssh has no problem connecting to the host. Yes, I am using ssh-agent as they keys are encrypted at rest.

$ ssh-add -l                                                                                                                    [10:03:13]
4096 SHA256:IIgRCJ84QIlEIoYJ1RiFYRwPlbxXVr3z/7jo+FTM6zg /Users/ssbarnea/.ssh/id_rsa (RSA)
2048 SHA256:oAGaCUURqEYWeDlI5OfD+lGUfTb8IYy0e79jLQojXM0 /Users/ssbarnea/.ssh/id_rsa.uploader (RSA)
4096 SHA256:ICr5NPxWA0AyFifvQt/n/N4fFiy/Y9dezPQl2FiklD0 sorin.sbarnea+bot@gmail.com (RSA)

The correct key is the first one but I really doubt that the macos podman cli is really trying to use the agent key.

[jwhonce]

@FreedomBen I added https://github.com/containers/podman/issues/8499#issuecomment-736639134 Could that be related here as well?

[FreedomBen]

@jwhonce Interesting, it could be related. When I tried podman-remote without the key in my SSH agent I was prompted me for my passphrase, but it then after entering the passphrase it failed to authenticate in the same way. I can try some of these things a bit later, tomorrow for sure.

[Talbot3]

hi, did you run his command.I have same error on my macOS big sure I did it with "https://github.com/containers/podman/blob/master/docs/tutorials/mac_win_client.md", then I found error

$ podman ps
Error: failed to create sshClient: dial unix /private/tmp/com.apple.launchd.zFwQB0vrnx/Listeners: connect: no such file or directory
FAIL: 125

I think it is ssh-client or system config question, after I run that command , it was solved.

eval "$(ssh-agent -s)"

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@jwhonce @baude PTAL

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ssbarnea @Talbot3 @FreedomBen Is this still an issue? A lot of work has been happening in podman-remote for Mac.

[ashley-cui]

Ah! Fedora by default rejects rsa keys, if you use an ed25519 key, this works properly. Closing now, If this is still an issue, please re-open.

[tonykay]

I would like to confirm the above, saw the same issue (Fedora 34 host, Big Sur 11.3.1 Mac, podman 3.1.2 at both ends). By generating an ed25519 key this worked perfectly after a frustrating 30 minutes with my old rsa key.

[runlevel5]

Thanks @tonykay, I've run into the same issue on my macOS Big Sur.

I think this article https://www.redhat.com/sysadmin/podman-clients-macos-windows should be updated accordingly

[jwboyer]

@rhatdan @ashley-cui I think forcing people to regenerate keys here is a pretty poor user experience.

I'm also concerned that the actual issue is that podman/go are using SHA1 for the key exchange protocol for RSA keys and that's actually what is causing issues in some cases. When I tried using an RSA key on a CentOS Stream 9 machine to copy an image to a different CentOS 9 Stream machine, it keeps failing even though normal ssh between the machines with the same key works fine. If that's actually the case, then having podman use a different encryption for the key exchange should allow RSA keys to work.

[rhatdan]

@baude @jwhonce @mtrmac @vrothberg PTAL

[mtrmac]

Cc: @lsm5

[GregHanson]

I am still hitting this problem on an M1 Mac even after regenerating the ssh key.

1. generate the ssh key in line with the GitHub docs here

   $ ssh-keygen -t ed25519 -C "your_email@example.com"
   $ eval "$(ssh-agent -s)"
   # modify ~/.ssh/config
   $ ssh-add -K ~/.ssh/id_ed25519

2. Follow the added steps from #12445

   $ podman-remote system connection add myuser --identity ~/.ssh/id_ed25519 ssh://192.168.122.1/run/user/1000/podman/podman.sock
   $ podman-remote system connection default myuser
   $ podman-remote system connection list
   Name                         Identity                                     URI
   myuser*                    /Users/myuser/.ssh/id_ed25519              ssh://myuser@192.168.122.1:22/run/user/1000/podman/podman.sock

3. Connection still fails for podman commands

   $ podman machine init
   $ podman machine start
   $ podman info --log-level=debug                              
   INFO[0000] podman filtering at log level debug          
   DEBU[0000] Called info.PersistentPreRunE(podman info --log-level=debug) 
   DEBU[0000] SSH Ident Key "/Users/myuser/.ssh/id_ed25519" SHA256:ubDEOIjZO+n01TviiUX8+4gICWwAlbO/6l/6SCNy/NY ssh-ed25519 
   DEBU[0000] Found SSH_AUTH_SOCK "/var/folders/n1/_q7tx11j5cl8pv3m3n2q38p40000gn/T//ssh-VlRRohsdSvuz/agent.24841", ssh-agent signer(s) enabled 
   Error: failed to create sshClient: Connection to bastion host (ssh://myuser@192.168.122.1:22/run/user/1000/podman/podman.sock) failed.: dial tcp 192.168.122.1:22: i/o timeout

MacOS doesn't have the systemctl command which most of the docs use, are there any relevant launchctl commands Mac users need to run?

[ssbarnea]

TBH, I find very annoying that I need to run podman machine start every time after a reboot. I wish it was a way to either configure podman to start the machine on demand or automatically at login. Probably on demand would a better approach as it would not drain the battery or hog the cpu when not really needed.

[lsm5]

@rhatdan @ashley-cui I think forcing people to regenerate keys here is a pretty poor user experience.

I'm also concerned that the actual issue is that podman/go are using SHA1 for the key exchange protocol for RSA keys and that's actually what is causing issues in some cases. When I tried using an RSA key on a CentOS Stream 9 machine to copy an image to a different CentOS 9 Stream machine, it keeps failing even though normal ssh between the machines with the same key works fine. If that's actually the case, then having podman use a different encryption for the key exchange should allow RSA keys to work.

@jwboyer do you have a reproducer for this? I tried podman system connection add --identity $RSA_KEY $CONNECTION_NAME $REMOTE_HOST on both C9S and Fedora. Worked on C9S, didn't work on fedora, I suspect because of fedora getting rid of rsa. And of course, podman image scp $IMAGE $CONNECTION_NAME:: also worked fine on C9S. This was with podman-4.0.3-1.el9.x86_64

[jwboyer]

Yep, I tried this again today using a CentOS Stream 9 VM trying to podman image scp to a RHEL 9 Beta machine using a 2048 bit RSA key.

CentOS Stream 9 machine info

[jwboyer@localhost ~]$ cat /etc/os-release 
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[jwboyer@localhost ~]$ rpm -q podman
podman-4.0.3-1.el9.x86_64
[jwboyer@localhost ~]$

SSH connection with the key working:

[jwboyer@localhost ~]$ ssh -A 192.168.122.170
Warning: Permanently added '192.168.122.170' (ED25519) to the list of known hosts.
Web console: https://localhost:9090/ or https://192.168.122.170:9090/

Last login: Mon Apr 25 13:02:35 2022 from 192.168.122.1
[jwboyer@localhost ~]$ exit
[jwboyer@localhost ~]$ update-crypto-policies --show
DEFAULT
[jwboyer@localhost ~]$ 

Podman connection add

[jwboyer@localhost ~]$

podman image scp failing with handshake issue

[jwboyer@localhost ~]$ podman pull ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4eef1fa1f1c1 done  
Copying blob eb24191cef20 done  
Copying config c54243b588 done  
Writing manifest to image destination
Storing signatures
c54243b58814cd424740dfebb046f356ba3acc23f04e04ffba60004eb1e8b0ea
[jwboyer@localhost ~]$ podman image scp ubi8 CONNECTION::
Copying blob 30adffdbd388 done  
Copying blob 0804b3644b85 done  
Copying config c54243b588 done  
Writing manifest to image destination
Storing signatures
Key Passphrase: 
Error: failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
[jwboyer@localhost ~]$ 

RHEL 9 VM info

[jwboyer@localhost ~]$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.0 Beta (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0 Beta"
[jwboyer@localhost ~]$ rpm -q podman
podman-4.0.2-4.el9_0.x86_64
[jwboyer@localhost ~]$ update-crypto-policies --show
DEFAULT
[jwboyer@localhost ~]$ 

As you can see, an SSH works fine between the machines with the same key, but podman image scp fails with the handshake issue. RSA keys aren't deprecated in CS9/RHEL9 and the regular ssh connection works fine.

[LewisGaul]

I have also hit this issue, with RSA keys being rejected. I agree with @jwboyer that it would be far better for users to be able to use the same keys that are usable by the standard ssh client.

To be clear, I am able to ssh using the RSA key, but podman rejects it, so this is not simply the remote host rejecting the key. I have seen this sshing to Ubuntu22.04 and CentOS9. In general it would be preferable for podman to default to using the same ssh keys that the ssh client uses rather than having to specify CONTAINER_SSHKEY or --identity.

Could this be reopened as I think there's still an issue in podman here?

[mtrmac]

If the hypothesis is using SSH with RSA keys and SHA1 (where it should be using SHA2), please follow #14001 , it contains more recent investigation, and in particular a supposed fix.

[LewisGaul]

Am I right to assume there won't be a fix/workaround in v3.x then? :(

[rhatdan]

That is a safe assumption.

[dybxin]

hi, did you run his command.I have same error on my macOS big sure I did it with "https://github.com/containers/podman/blob/master/docs/tutorials/mac_win_client.md", then I found error

$ podman ps
Error: failed to create sshClient: dial unix /private/tmp/com.apple.launchd.zFwQB0vrnx/Listeners: connect: no such file or directory
FAIL: 125

I think it is ssh-client or system config question, after I run that command , it was solved.

eval "$(ssh-agent -s)"

i run elval "$(ssh-agnet -s)" command.

run command before:

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://core@localhost:59949/run/user/1000/podman/podman.sock) failed.: ssh: handshake failed: s

run command after:

$ podman ps
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: dial unix C:/Users/yabdong/AppData/Local/Temp/ssh-WE7zhJvY7rSa/agent.1145: connect: No connection could be made because the target machine actively refused it.