search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.16k stars 2.36k forks source link

Unable to connect to remote vm with podman: ssh: rejected: connect failed (open failed) #12432

Closed pratikmallya closed 2 years ago

pratikmallya commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm unable to connect to a linux vm from podman on a mac.

Steps to reproduce the issue:

  1. Get a VM on OCI. I got the free tier with the following "shape" and Oracle-Linux-8.4-aarch64-2021.10.25-0:
Type Value
Shape VM.Standard.A1.Flex
OCPU count 4
Network bandwidth (Gbps) 4
Memory (GB) 24
Local disk Block storage only

I specified the public ssh keys. In addition to using my existing rsa keys, I also created a ed25519 key to rule out issues with key algorithm, and added its public key to the machine. I am able to log into a machine thus provisioned:

❯ ssh -i ~/.ssh/oci/id_ed25519 opc@192.9.226.230
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Sun Nov 28 18:46:56 2021 from 52.119.114.22
[opc@instance-20211128-1024 ~]$

  1. Install podman on the vm:

    sudo yum module enable -y container-tools
sudo yum -y install podman
systemctl --user enable podman.socket
sudo loginctl enable-linger opc

  2. Try to add this connection to local podman

    ❯ podman system connection add oci --identity ~/.ssh/oci/id_ed25519 ssh://opc@192.9.226.230:22/run/user/1000/podman/podman.sock
❯ podman system connection list
Name        Identity                                 URI
oci*        /Users/pratikmallya/.ssh/oci/id_ed25519  ssh://opc@192.9.226.230:22/run/user/1000/podman/podman.sock

  3. Try to get info:

    ❯ podman --log-level=debug info --debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info --debug)
DEBU[0000] SSH Ident Key "/Users/pratikmallya/.ssh/oci/id_ed25519" SHA256:15QBOSVd+96gZ3p2cUCPFM4aRE45EksAGSgeEp8UV2s ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.NrigAZAzMU/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:8IK+xJRl6YCCCr4D4js9tgxHQTJCifwdKDUuPecRSSA ssh-rsa
DEBU[0000] SSH Agent Key SHA256:bJPexGaDqkxNtFQmSJysQKIwjhHC+vJ15d8kdLOnZW0 ssh-rsa
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.2/libpod/_ping
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v3.4.2/libpod/_ping": ssh: rejected: connect failed (open failed)

Describe the results you received: podman client failed to connect to the remote linux with podman installed

Describe the results you expected: Expected podman to connect successfully

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

server:
[opc@instance-20211128-1024 ~]$ podman version
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.7
Built:        Wed Nov 10 16:08:24 2021
OS/Arch:      linux/arm64

client:
❯ brew info podman
podman: stable 3.4.2 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/3.4.2 (170 files, 40MB) *
  Poured from bottle on 2021-11-15 at 09:13:31
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✔, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
    Install HEAD version
==> Caveats
fish completions have been installed to:
  /usr/local/share/fish/vendor_completions.d
==> Analytics
install: 11,680 (30 days), 40,436 (90 days), 69,182 (365 days)
install-on-request: 11,682 (30 days), 40,439 (90 days), 69,130 (365 days)
build-error: 0 (30 days)

Output of podman info --debug:

❯ podman --log-level=debug info --debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info --debug)
DEBU[0000] SSH Ident Key "/Users/pratikmallya/.ssh/oci/id_ed25519" SHA256:15QBOSVd+96gZ3p2cUCPFM4aRE45EksAGSgeEp8UV2s ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.NrigAZAzMU/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:8IK+xJRl6YCCCr4D4js9tgxHQTJCifwdKDUuPecRSSA ssh-rsa
DEBU[0000] SSH Agent Key SHA256:bJPexGaDqkxNtFQmSJysQKIwjhHC+vJ15d8kdLOnZW0 ssh-rsa
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.2/libpod/_ping
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v3.4.2/libpod/_ping": ssh: rejected: connect failed (open failed)

Package info (e.g. output of rpm -q podman or apt list podman):

[opc@instance-20211128-1024 ~]$ rpm -q podman
podman-3.3.1-9.0.1.module+el8.5.0+20416+d687fed7.aarch64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

rhatdan commented 2 years ago

@jwhonce PTAL

github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

baude commented 2 years ago

is it possible you forgot to start the podman service? i see the enable but nothing to start it ?

pratikmallya commented 2 years ago

@baude podman is definitely running on the instance:

[opc@instance-20211128-1024 ~]$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
[opc@instance-20211128-1024 ~]$ ps aux | grep podman
opc        42489  0.0  0.0  49216 19264 ?        S     2021   0:00 podman
opc      2173603  0.0  0.0   7168  1536 pts/0    S+   11:41   0:00 grep --color=auto podman
pratikmallya commented 2 years ago

Sorry, I don't think the comment above answered the question. Here is a more relevant output, showing that the podman.socket systemd unit is active:

[opc@instance-20211128-1024 ~]$ systemctl status podman.socket
● podman.socket - Podman API Socket
   Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Mon 2022-01-31 04:47:05 GMT; 40s ago
     Docs: man:podman-system-service(1)
   Listen: /run/podman/podman.sock (Stream)
    Tasks: 0 (limit: 36881)
   Memory: 0B
   CGroup: /system.slice/podman.socket

and

[opc@instance-20211128-1024 ~]$ sudo podman --remote info
host:
  arch: arm64
  buildahVersion: 1.22.3
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.5.0+20416+d687fed7.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 232488fca3145f0078ae02ab8d29baadf1809495'
  cpus: 4
  distribution:
    distribution: '"ol"'
    version: "8.4"
  eventLogger: file
  hostname: instance-20211128-1024
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.17-2102.206.1.el8uek.aarch64
  linkmode: dynamic
  memFree: 20477575168
  memTotal: 24237244416
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module+el8.5.0+20416+d687fed7.aarch64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+20416+d687fed7.aarch64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8589869056
  swapTotal: 8589869056
  uptime: 1522h 18m 44.99s (Approximately 63.42 days)
registries:
  search:
  - container-registry.oracle.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1636560504
  BuiltTime: Wed Nov 10 16:08:24 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/arm64
  Version: 3.3.1
pratikmallya commented 2 years ago

I noticed a version mismatch between client and server, could this cause an issue?

Server:

[opc@instance-20211128-1024 ~]$ podman --version
podman version 3.3.1

Client:

 podman --version
podman version 3.4.2
rhatdan commented 2 years ago

Can you update the machine?

github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 2 years ago

Since I got no more response, I am going to close. Reopen if you have more feedback.

pratikmallya commented 2 years ago

I could not update the version on the machine since Oracle doesn’t provide a newer version of podman in their package repo. I have to wait for them to release a newer version.

baude commented 2 years ago

seems like you should be writing the bug against oracle?

pratikmallya commented 2 years ago

The issue is with podman and the current hypothesis is that it might possibly be due to version mismatch but it’s not confirmed. I don’t think it’s an Oracle issue.

I would ask that this issue be kept open until either someone can confirm or reject that hypothesis. I can do this but I’m not familiar with the package manager used on this os so I’ve been hoping that either they release a newer version soon or I find a way to downgrade my local podman version.

baude commented 2 years ago

@pratikmallya I'm going to re-open the issue as you asked. But the problem is not upstream podman The problems is your distribution has not released a version of podman new enough for the vm. You actually state as such. One thing you could do is build podman yourself on the mac by cloning upstream 3.3.1. We very well could be in the same position as this if you were using and older version of Ubuntu or even Fedora.

pratikmallya commented 2 years ago

Ok let me try the rebuild podman suggestion. Will report back with results.

github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

pratikmallya commented 2 years ago

Looking at this again now (apologies, been busy with other stuff).

It looks like oci released a newer version of podman (3.4.2) but that too seems to not work:

server:

podman version

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.12
Built:        Sat Feb  5 01:20:30 2022
OS/Arch:      linux/arm64

client:

podman -v

podman version 3.4.2

I see that the latest version is 4.0.3, so its possible that this may not be an issue in the latest version.

Going to see if I can build podman from scratch on server next to see If I can repro with a newer version

pratikmallya commented 2 years ago

Additional datapoint: I looked at the sshd logs server side. It looks like its able to connect, but the session is closed:

sudo journalctl -u sshd.service -f

Apr 03 21:41:56 instance-20211128-1024 sshd[401278]: Accepted publickey for opc from 52.119.114.22 port 54346 ssh2: ED25519 SHA256:15QBOSVd+96gZ3p2cUCPFM4aRE45EksAGSgeEp8UV2s
Apr 03 21:41:56 instance-20211128-1024 sshd[401278]: pam_unix(sshd:session): session opened for user opc by (uid=0)
Apr 03 21:41:57 instance-20211128-1024 sshd[401278]: pam_unix(sshd:session): session closed for user opc
rhatdan commented 2 years ago

Did this work with podman 4.0?

pratikmallya commented 2 years ago

Did this work with podman 4.0?

It does not

Compiled from source on remote VM:

$ podman --version
podman version 4.0.0-dev

$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

I used homebrew to update local podman on my mac:

$ podman --version
podman version 4.0.3

The command to connect still fails:

podman -c oci --log-level=debug info --debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman -c oci --log-level=debug info --debug)
DEBU[0000] SSH Ident Key "/Users/pratikmallya/.ssh/id_rsa" SHA256:mlHbe5rLtBO7IhIN9H47/9wi7e8Qt1uj0nou1IJMniM ssh-rsa
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.b46v3qrjvS/Listeners", ssh-agent signer(s) enabled
DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.3/libpod/_ping
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.0.3/libpod/_ping": ssh: rejected: connect failed (open failed)
rhatdan commented 2 years ago

@baude @jwhonce @ashley-cui PTAL

github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

LKajan commented 2 years ago

I have the same issue. My podman host is on RHEL 8 (podman 3.4.2, newest that is provided in the official repo). Client is on Windows 11 (podman 3.4.4).

On host machine: I have setted up a rootless configuration. Socket is started and listening. podman --remote info works. podman --remote run hello-world works.

On client machine: podman system info gives the following error:

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v3.4.4/libpod/_ping": ssh: rejected: connect failed (open failed)

Just like @pratikmallya also noticed sshd accepts the connection on a host machine.

catniptwinz commented 2 years ago

I'm having the same issue.

Client and server are both Arch Linux, podman 4.1.0 on both. Server is a VM (libvirt).

This is technically a rootful setup, but I'm accessing the root socket (/run/podman/podman.sock) over ssh as a non-root user, using a systemd drop in to ensure that the socket is owned by the "podman" group, to which my user also belongs.

podman info works as expected on server. Client side error message and server sshd logs are the same ones that @pratikmallya and @LKajan have posted.

UPDATE: This was a permissions issue on my end regarding /run/podman -- resolved per instructions in https://github.com/containers/podman/discussions/12903#discussioncomment-1991193

kuwv commented 2 years ago

I ran into this issue when using CRC while having Podman also installed on Fedora 36. I mistakenly used the Podman4 client to attempt to connect to CRC (Podman 3.4 server). I realized my error and switched to podman-remote.

Now, when I shutdown CRC and spin up Podman the residual connection from the previous attempt prevents some commands from using the correct connection.

I can SSH into the correct instance 192.168.130.11, but when attempting a podman login it attempts it using 192.168.127.2.

LKajan commented 2 years ago

I solved the issue in my case! The problem was that tcp forwarding was not permitted in the sshd config. It is enabled by default but on this server it was somehow disabled. So I changed the AllowTcpForwarding to yes in the /etc/ssh/sshd_config There is also a AllowStreamLocalForwarding option that I'm not sure if it matters but I set to yes just in case.

umohnani8 commented 2 years ago

@cdoern can you PTAL to see if this is related to the other ssh issues you are working on.

github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 2 years ago

@cdoern any update?

cdoern commented 2 years ago

@pratikmallya please try this with podman's main branch (or the forthcoming 4.3 release), I believe this is fixed.

rhatdan commented 2 years ago

Reopen if @cdoern is mistaken.

stendler commented 1 year ago

I seem to have run into this issue with version 4.4.4 on host and client.

ssh myhost works without any additional identity file or user specification due to ssh config and agent, and I would expect podman to use those as well. This is also a rootless scenario, so permissions should not be the issue here.

Trying to connect from the client results in varying errors:

$ podman --url ssh://myhost ps -a
Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures
$ podman --url=ssh://myhost/run/user/1000/podman/podman.sock ps -a
Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures
$ podman --url=ssh://username@myhost/run/user/1000/podman/podman.sock ps -a
Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures
$ podman --url=ssh://username@myhost/run/user/1000/podman/podman.sock --identity ~/.ssh/id_myhost  ps -a
Key Passphrase:
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": strconv.Atoi: parsing "": invalid syntax
$ podman --url ssh://username@myhost:22/run/user/1000/podman/podman.sock --identity ~/.ssh/id_myhost ps -a
Key Passphrase: 
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": ssh: rejected: connect failed (open failed)
$ podman --url ssh://username@myhost:22 ps
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": ssh: rejected: connect failed (open failed)
Same iterations when setting the podman system connection similarly. (click to expand) ```sh $ podman system connection add myhost myhost Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures $ podman system connection add myhost myhost:22 Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures $ podman system connection add myhost ssh://myhost Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures $ podman system connection add myhost ssh://myhost:22 Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures $ podman system connection add myhost username@myhost $podman system connection list Name URI Identity Default myhost ssh://username@myhost:22/run/user/1000/podman/podman.sock true $ podman --connection myhost ps Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": ssh: rejected: connect failed (open failed) $ podman ---ssh native --connection myhost ps Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": ssh: rejected: connect failed (open failed) ```

On the host I verified, that the podman.socket exists and works:

$ ls -l /run/user/1000/podman/podman.sock
srw-rw----. 1 username username 0 Apr 15 13:03 /run/user/1000/podman/podman.sock
$ podman --url unix://run/user/1000/podman/podman.sock ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$ podman system connection add user unix://run/user/1000/podman/podman.sock
WARN[0000] "/user/1000/podman/podman.sock" does not exists
$ podman system connection list
Name        URI                                      Identity    Default
user        unix://run/user/1000/podman/podman.sock              true
$ podman --connection user ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
Interestingly, trying to use localhost ssh does not work with podman either: (click to expand) ```sh $ podman --url ssh://localhost ps $ podman --url ssh://localhost:22 ps $ podman --url ssh://username@localhost ps $ podman --url ssh://username@localhost:22 ps $ podman --url ssh://username@localhost:22 ps $ podman --url ssh://localhost/run/user/1000/podman/podman.sock ps $ podman --url ssh://localhost:22/run/user/1000/podman/podman.sock ps $ podman --url ssh://username@localhost:22/run/user/1000/podman/podman.sock ps # all resulting in Error: failed to connect: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures # or sometimes also in Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": ssh: rejected: connect failed (open failed) # with the exception of username and socket path but without port: $ podman --url ssh://username@localhost/run/user/1000/podman/podman.sock ps Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: Get "http://d/v4.4.4/libpod/_ping": strconv.Atoi: parsing "": invalid syntax ```

Although ssh localhost does work on the host via forwarded agent.

or am I just using it horribly wrong?

deltawi commented 1 year ago

Did someone fix this ? I am facing exactly the same issue with podman version 4.3.1 Error: unable to connect to Podman socket: Get "http://d/v4.3.1/libpod/_ping": ssh: rejected: connect failed (open failed) exit code: 125

podman machine ls: image

BlackHole1 commented 1 year ago

I encountered this issue after compiling podman based on the latest commit. All commands involving requests to libpod have failed, such as:

image

When I enter the virtual machine through podman machine ssh, it is very slow and almost always fails. So I manually executed the ssh command and found that the ssh command is also very slow, and ultimately stuck at:

Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Tue Jun 20 10:08:30 2023 from 192.168.127.1
[systemd]
Failed Units: 2
  envset-fwcfg.service
  systemd-homed.service

without moving. When I pressed ctrl + c, I unexpectedly entered the virtual machine.

ssh -vvv -i /Users/black-hole/.oomol-studio/.containers/.ssh/oomol-1.0.0 -p 49834 core@localhost -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= 
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/black-hole/.ssh/config
debug3: /Users/black-hole/.ssh/config line 1: Including file /Users/black-hole/.orbstack/ssh/config depth 0
debug1: Reading configuration data /Users/black-hole/.orbstack/ssh/config
debug1: /Users/black-hole/.ssh/config line 3: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/black-hole/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/black-hole/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to localhost port 49834.
debug1: Connection established.
debug1: identity file /Users/black-hole/.oomol-studio/.containers/.ssh/oomol-1.0.0 type 3
debug1: identity file /Users/black-hole/.oomol-studio/.containers/.ssh/oomol-1.0.0-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.0
debug1: compat_banner: match: OpenSSH_9.0 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to localhost:49834 as 'core'
debug3: put_host_port: [localhost]:49834
debug3: record_hostkey: found key type ED25519 in file /Users/black-hole/.ssh/known_hosts:93
debug3: record_hostkey: found key type RSA in file /Users/black-hole/.ssh/known_hosts:94
debug3: record_hostkey: found key type ECDSA in file /Users/black-hole/.ssh/known_hosts:95
debug3: load_hostkeys_file: loaded 3 keys from [localhost]:49834
debug1: load_hostkeys: fopen /Users/black-hole/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC:  compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC:  compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:JeG28MZy5I1KjDKTgVgCoEi+s8H/JG96GSTtb0MPd/I
debug3: put_host_port: [127.0.0.1]:49834
debug3: put_host_port: [localhost]:49834
debug3: record_hostkey: found key type ED25519 in file /Users/black-hole/.ssh/known_hosts:93
debug3: record_hostkey: found key type RSA in file /Users/black-hole/.ssh/known_hosts:94
debug3: record_hostkey: found key type ECDSA in file /Users/black-hole/.ssh/known_hosts:95
debug3: load_hostkeys_file: loaded 3 keys from [localhost]:49834
debug1: load_hostkeys: fopen /Users/black-hole/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[localhost]:49834' is known and matches the ED25519 host key.
debug1: Found key in /Users/black-hole/.ssh/known_hosts:93
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation
debug1: get_agent_identities: agent returned 4 keys
debug1: Will attempt key: Github Authentication Key ED25519 SHA256:AJebMzlWoSz73/vMbzLwPGRI10/BCpY1P83R7mO9ZG4 agent
debug1: Will attempt key: Github Signing Key ED25519 SHA256:q3J3WRHU39SXEc1R7w9a+RJB1wHQS76Tpt+5A9KhEd8 agent
debug1: Will attempt key: AWS:EC2:ssh-vpc ED25519 SHA256:27RYQSXYdRQSlqw0lKon8n2S8bM8IREmFWKhnr9EiT8 agent
debug1: Will attempt key: oomol-ci ED25519 SHA256:90W+brOs/9PitBityawmcMkrIRcozQuODp9feYGyPQ4 agent
debug1: Will attempt key: /Users/black-hole/.oomol-studio/.containers/.ssh/oomol-1.0.0 ED25519 SHA256:xgQDKglFGxbbBL2jMQJA2t7/b0DfEiQlQkaabhAb4jo explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-with-mic
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: Github Authentication Key ED25519 SHA256:AJebMzlWoSz73/vMbzLwPGRI10/BCpY1P83R7mO9ZG4 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-with-mic
debug1: Offering public key: Github Signing Key ED25519 SHA256:q3J3WRHU39SXEc1R7w9a+RJB1wHQS76Tpt+5A9KhEd8 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-with-mic
debug1: Offering public key: AWS:EC2:ssh-vpc ED25519 SHA256:27RYQSXYdRQSlqw0lKon8n2S8bM8IREmFWKhnr9EiT8 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-with-mic
debug1: Offering public key: oomol-ci ED25519 SHA256:90W+brOs/9PitBityawmcMkrIRcozQuODp9feYGyPQ4 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-with-mic
debug1: Offering public key: /Users/black-hole/.oomol-studio/.containers/.ssh/oomol-1.0.0 ED25519 SHA256:xgQDKglFGxbbBL2jMQJA2t7/b0DfEiQlQkaabhAb4jo explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/black-hole/.oomol-studio/.containers/.ssh/oomol-1.0.0 ED25519 SHA256:xgQDKglFGxbbBL2jMQJA2t7/b0DfEiQlQkaabhAb4jo explicit
debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ED25519 SHA256:xgQDKglFGxbbBL2jMQJA2t7/b0DfEiQlQkaabhAb4jo
debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:xgQDKglFGxbbBL2jMQJA2t7/b0DfEiQlQkaabhAb4jo
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to localhost ([127.0.0.1]:49834) using "publickey".
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: filesystem
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received RSA key SHA256:KrQo9LvRruDNvD21ZKeNYf8j8hI3Q64opgR2Wsd1Cc0
debug3: client_input_hostkeys: received ECDSA key SHA256:Xp7khC335e1jodD4h+tSEV47iGxH0KkAkUJDy4yVR9s
debug3: client_input_hostkeys: received ED25519 key SHA256:JeG28MZy5I1KjDKTgVgCoEi+s8H/JG96GSTtb0MPd/I
debug3: put_host_port: [localhost]:49834
debug1: client_input_hostkeys: searching /Users/black-hole/.ssh/known_hosts for [localhost]:49834 / (none)
debug3: hostkeys_foreach: reading file "/Users/black-hole/.ssh/known_hosts"
debug3: hostkeys_find: found ssh-ed25519 key at /Users/black-hole/.ssh/known_hosts:93
debug3: hostkeys_find: found ssh-rsa key at /Users/black-hole/.ssh/known_hosts:94
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key at /Users/black-hole/.ssh/known_hosts:95
debug1: client_input_hostkeys: searching /Users/black-hole/.ssh/known_hosts2 for [localhost]:49834 / (none)
debug1: client_input_hostkeys: hostkeys file /Users/black-hole/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 0 new, 3 retained, 0 incomplete match. 0 to remove
debug1: client_input_hostkeys: no new or deprecated keys from server
debug3: receive packet: type 4
debug1: Remote: /usr/libexec/ssh-key-dir %u:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /usr/libexec/ssh-key-dir %u:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 6 setting TCP_NODELAY
debug3: set_sock_tos: set socket 6 IP_TOS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env TERM_SESSION_ID
debug3: Ignored env SSH_AUTH_SOCK
debug1: channel 0: setting env LC_TERMINAL_VERSION = "3.4.19"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORFGBG
debug3: Ignored env ITERM_PROFILE
debug3: Ignored env XPC_FLAGS
debug1: channel 0: setting env LANG = "zh_CN.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env PWD
debug3: Ignored env SHELL
debug3: Ignored env __CFBundleIdentifier
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env PATH
debug1: channel 0: setting env LC_TERMINAL = "iTerm2"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env COLORTERM
debug3: Ignored env COMMAND_MODE
debug3: Ignored env TERM
debug3: Ignored env HOME
debug3: Ignored env TMPDIR
debug3: Ignored env USER
debug3: Ignored env XPC_SERVICE_NAME
debug3: Ignored env LOGNAME
debug3: Ignored env ITERM_SESSION_ID
debug3: Ignored env __CF_USER_TEXT_ENCODING
debug3: Ignored env SHLVL
debug3: Ignored env OLDPWD
debug3: Ignored env P9K_TTY
debug3: Ignored env _P9K_TTY
debug3: Ignored env ZSH
debug3: Ignored env PAGER
debug3: Ignored env LESS
debug3: Ignored env LSCOLORS
debug3: Ignored env LS_COLORS
debug3: Ignored env http_proxy
debug3: Ignored env HTTP_PROXY
debug3: Ignored env https_proxy
debug3: Ignored env HTTPS_PROXY
debug3: Ignored env ftp_proxy
debug3: Ignored env FTP_PROXY
debug3: Ignored env rsync_proxy
debug3: Ignored env RSYNC_PROXY
debug3: Ignored env ALL_PROXY
debug3: Ignored env all_proxy
debug3: Ignored env no_proxy
debug3: Ignored env P9K_SSH
debug3: Ignored env _P9K_SSH_TTY
debug3: Ignored env GVM_ROOT
debug3: Ignored env GVM_VERSION
debug3: Ignored env GVM_PATH_BACKUP
debug3: Ignored env gvm_go_name
debug3: Ignored env gvm_pkgset_name
debug3: Ignored env GOROOT
debug3: Ignored env GOPATH
debug3: Ignored env GVM_OVERLAY_PREFIX
debug3: Ignored env LD_LIBRARY_PATH
debug3: Ignored env PKG_CONFIG_PATH
debug3: Ignored env NVM_DIR
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env NVM_BIN
debug3: Ignored env NVM_INC
debug3: Ignored env TZ_LIST
debug3: Ignored env OP_PLUGIN_ALIASES_SOURCED
debug3: Ignored env NO_AUTH_BOTO_CONFIG
debug3: Ignored env GIT_CACHE_PATH
debug3: Ignored env CCACHE_DIR
debug3: Ignored env IDE_SHELL
debug3: Ignored env PYENV_SHIMS
debug3: Ignored env PNPM_HOME
debug3: Ignored env _
debug1: channel 0: setting env LC_ALL = ""
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Fedora CoreOS 38.20230609.2.1

Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Tue Jun 20 10:08:30 2023 from 192.168.127.1
[systemd]
Failed Units: 2
  envset-fwcfg.service
  systemd-homed.service

^C-bash-5.2$
-bash-5.2$
-bash-5.2$

I also tried adding the option -o IdentitiesOnly=yes, but the problem still existed. So it has nothing to do with my local SSH config.

When I entered the virtual machine, I executed journalctl -u sshd.service -f. And when I executed podman image again on the host machine, I got the following log:

Jun 20 10:16:17 localhost.localdomain sshd[1515]: Accepted publickey for core from 192.168.127.1 port 28812 ssh2: ED25519 SHA256:xgQDKglFGxbbBL2jMQJA2t7/b0DfEiQlQkaabhAb4jo
Jun 20 10:18:17 localhost.localdomain sshd[1515]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 20 10:18:17 localhost.localdomain sshd[1515]: pam_unix(sshd:session): session opened for user core(uid=502) by (uid=0)
Jun 20 10:18:17 localhost.localdomain sshd[1521]: error: connect to /run/user/502/podman/podman.sock port -2 failed: No such file or directory
Jun 20 10:18:17 localhost.localdomain sshd[1521]: error: connect to /run/user/502/podman/podman.sock port -2 failed: No such file or directory
Jun 20 10:18:17 localhost.localdomain sshd[1521]: error: connect to /run/user/502/podman/podman.sock port -2 failed: No such file or directory
Jun 20 10:18:17 localhost.localdomain sshd[1515]: pam_unix(sshd:session): session closed for user core

BTW. Podman was working fine last night, but after I closed my laptop (put it in sleep mode) and opened it the next day, this problem appeared.

I feel like simply executing podman machine stop/start would be enough to restore things, but rather than just solving the problem by restarting, I'd like to know the cause of this issue and how to fix it within Podman.

irusensei commented 1 year ago

Having same issues as described here. Tried all the solutions that solved for some but none worked. Podman on the linux server is 4.5.1 on Fedora 38 amd64. Client is MacOS installed from brew, also 4.5.1 but ARM64. SSH works, podman is running on Linux system.

> podman --log-level=debug ps
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps) 
DEBU[0000] Found SSH_AUTH_SOCK "/Users/iru/.gnupg/S.gpg-agent.ssh", ssh-agent signer enabled 
DEBU[0000] SSH Agent Key SHA256:[REDACTED] ssh-rsa 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.5.1/libpod/_ping 
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.5.1/libpod/_ping": strconv.Atoi: parsing "": invalid syntax
DEBU[0000] Shutting down engines

SSHD logs on host shows the login works. Setting selinux enforcing to 0 doesn't change the results. Where is this http request attempt is coming from?

rhatdan commented 1 year ago

Please open a new issue, and do not comment on a closed issue, even if you think you have the same issue as reported.

mil1i commented 1 year ago

Also running into this.

Trying to figure it out. No error messages in journalctl sshd.service. I can ssh to the machine without issue, I can run podman containers with my user, and podman -r info returns normal from the host machine.

I just installed openSUSE microOS to give it a try, and this is the last piece I'm currently stuck on figuring out tonight.