search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.41k stars 2.31k forks source link

Podman machine start error on MacOS M2 #20462

Open MiaoXiaoTao opened 8 months ago

MiaoXiaoTao commented 8 months ago

Issue Description

after I upgrade os、homebrew and podman ,podman cannot start now. i've tried reinit machine. but it doesn't work.

Steps to reproduce the issue

Steps to reproduce the issue

  1. podman machine start

Describe the results you received

Starting machine "podman-machine-default" Waiting for VM ... kex_exchange_identification: Connection closed by remote host kex_exchange_identification: Connection closed by remote host kex_exchange_identification: Connection closed by remote host kex_exchange_identification: Connection closed by remote host kex_exchange_identification: Connection closed by remote host kex_exchange_identification: Connection closed by remote host Error: machine did not transition into running state: ssh error: exit status 255

Describe the results you expected

can start podman success。

podman info output

podman info get error:
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:57416: connect: connection refused

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

podman version:4.7.1 qemu version: 8.1.2 macos version: Ventura 13.5 apple M2

when i use debug mode，the qemu start success，but ssh return 255

ashley-cui commented 8 months ago

Does clearing localhost ssh connections from ~/.ssh/known_hosts do anything?

MiaoXiaoTao commented 8 months ago

Does clearing localhost ssh connections from ~/.ssh/known_hosts do anything?

I've tried clean the known_hosts,it didn't work too.

rafaeltuelho commented 8 months ago

Same issue here. Podman 4.7.1 Podman Desktop 1.4.0

MacOS Sonoma 14.0 (Macbook Pro M2, 2023)

If I recreate the Podman machine it works for a while, but it fails again after a while

baude commented 8 months ago

did qemu also get upgraded as part of the installations ?

baude commented 8 months ago

can you also run podman --log-level debug machine start ... look for debug details in the terminal window as well in the qemu gui console.

rafaeltuelho commented 8 months ago

Here is Podman the machine startup output in debug mode as you suggested:

podman --log-level debug machine start                                                                                                                                                                     [10:56:12]
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `qemu` virtualization provider
Starting machine "podman-machine-default"
[/opt/homebrew/opt/podman/libexec/podman/gvproxy -listen-qemu unix:///var/folders/8x/2bh7r3s13xx2z4zz4f6_gt380000gn/T/podman/qmp_podman-machine-default.sock -pid-file /var/folders/8x/2bh7r3s13xx2z4zz4f6_gt380000gn/T/podman/podman-machine-default_proxy.pid -ssh-port 51039 -forward-sock /Users/rsoares/.local/share/containers/podman/machine/qemu/podman.sock -forward-dest /run/podman/podman.sock -forward-user root -forward-identity /Users/rsoares/.ssh/podman-machine-default --debug]
DEBU[0000] qemu cmd: [/opt/homebrew/bin/qemu-system-aarch64 -m 5720 -smp 4 -fw_cfg name=opt/com.coreos/config,file=/Users/rsoares/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix:/var/folders/8x/2bh7r3s13xx2z4zz4f6_gt380000gn/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/8x/2bh7r3s13xx2z4zz4f6_gt380000gn/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=apodman-machine-default_ready -device virtserialport,chardev=apodman-machine-default_ready,name=org.fedoraproject.port.0 -pidfile /var/folders/8x/2bh7r3s13xx2z4zz4f6_gt380000gn/T/podman/podman-machine-default_vm.pid -accel hvf -accel tcg -cpu host -M virt,highmem=on -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/rsoares/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -virtfs local,path=/Users,mount_tag=vol0,security_model=none -virtfs local,path=/private,mount_tag=vol1,security_model=none -virtfs local,path=/var/folders,mount_tag=vol2,security_model=none -drive if=virtio,file=/Users/rsoares/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-38.20231014.2.0-qemu.aarch64.qcow2]
Waiting for VM ...
DEBU[0014] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true]
Mounting volume... /Users:/Users
DEBU[0015] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= -q -- sudo chattr -i / ; sudo mkdir -p /Users ; sudo chattr +i / ;]
DEBU[0015] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= -q -- sudo mount -t 9p -o trans=virtio vol0 /Users -o version=9p2000.L,msize=131072]
Mounting volume... /private:/private
DEBU[0015] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= -q -- sudo chattr -i / ; sudo mkdir -p /private ; sudo chattr +i / ;]
DEBU[0015] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= -q -- sudo mount -t 9p -o trans=virtio vol1 /private -o version=9p2000.L,msize=131072]
Mounting volume... /var/folders:/var/folders
DEBU[0015] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= -q -- sudo chattr -i / ; sudo mkdir -p /var/folders ; sudo chattr +i / ;]
DEBU[0015] Executing: ssh [-i /Users/rsoares/.ssh/podman-machine-default -p 51039 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= -q -- sudo mount -t 9p -o trans=virtio vol2 /var/folders -o version=9p2000.L,msize=131072]
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully
DEBU[0016] Called machine start.PersistentPostRunE(podman --log-level debug machine start)
DEBU[0016] Shutting down engines

I'm wondering, in my case, Podman machine gets stuck after I do wake-up my Macbook Pro from sleep/standby mode (open the lid). This issue didn't use to happen on my previous Intel MacBook. 🤔

baude commented 7 months ago

I think Podman 4.8 will solve this problem ... RC1 was cut on Monday and we expect a release cut next week. If the problem is truly hibernation, then it is likely a different issue.

vazra commented 3 months ago

I'm using Apple M2 Max with MacOS Samona 14.4

ProductName:            macOS
ProductVersion:         14.4
BuildVersion:           23E214

in a fresh podman install, when I try starting podman machine, I get connection refused error. 

➜  podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
ssh: connect to host localhost port 50571: Connection refused
ssh: connect to host localhost port 50571: Connection refused
ssh: connect to host localhost port 50571: Connection refused
ssh: connect to host localhost port 50571: Connection refused
ssh: connect to host localhost port 50571: Connection refused
ssh: connect to host localhost port 50571: Connection refused
Error: machine did not transition into running state: ssh error: exit status 255

after that the machine seems to be running, but not able to connect with ssh

➜  podman machine ssh  
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
ssh: connect to host localhost port 50571: Connection refused

podman version 4.9.3 QEMU emulator version 8.2.1

Also, tried with log-level debug, here's the logs. 

➜ podman --log-level debug machine start
INFO[0000] podman filtering at log level debug          
DEBU[0000] Using Podman machine with `qemu` virtualization provider 
Starting machine "podman-machine-default"
DEBU[0000] {true 1500 map[forward-dest:[/run/user/501/podman/podman.sock] forward-identity:[/Users/ameen/.ssh/podman-machine-default] forward-sock:[/Users/ameen/.local/share/containers/podman/machine/qemu/podman.sock] forward-user:[core]] [] map[listen-qemu:unix:///var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/qmp_podman-machine-default.sock] /var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/podman-machine-default_proxy.pid 50571} 
DEBU[0000] gvproxy args: [/opt/homebrew/Cellar/podman/4.9.3/libexec/podman/gvproxy -debug -mtu 1500 -ssh-port 50571 -listen-qemu unix:///var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/qmp_podman-machine-default.sock -forward-user core -forward-identity /Users/ameen/.ssh/podman-machine-default -forward-sock /Users/ameen/.local/share/containers/podman/machine/qemu/podman.sock -forward-dest /run/user/501/podman/podman.sock -pid-file /var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/podman-machine-default_proxy.pid] 
DEBU[0000] qemu cmd: [/opt/homebrew/bin/qemu-system-aarch64 -accel hvf -accel tcg -cpu host -M virt,highmem=off -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/ameen/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -m 2048 -smp 6 -fw_cfg name=opt/com.coreos/config,file=/Users/ameen/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix:/var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=apodman-machine-default_ready -device virtserialport,chardev=apodman-machine-default_ready,name=org.fedoraproject.port.0 -pidfile /var/folders/03/0zklq9yd6t760gfsrpyd2xsr0000gn/T/podman/podman-machine-default_vm.pid -virtfs local,path=/Users,mount_tag=vol0,security_model=none -virtfs local,path=/private,mount_tag=vol1,security_model=none -virtfs local,path=/var/folders,mount_tag=vol2,security_model=none -drive if=virtio,file=/Users/ameen/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-39.20240309.2.0-qemu.aarch64.qcow2] 
Waiting for VM ...
DEBU[0014] Executing: ssh [-i /Users/ameen/.ssh/podman-machine-default -p 50571 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true] 
ssh: connect to host localhost port 50571: Connection refused
DEBU[0014] SSH readiness check for machine failed: exit status 255 
DEBU[0014] Executing: ssh [-i /Users/ameen/.ssh/podman-machine-default -p 50571 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true] 
ssh: connect to host localhost port 50571: Connection refused
DEBU[0014] SSH readiness check for machine failed: exit status 255 
DEBU[0016] Executing: ssh [-i /Users/ameen/.ssh/podman-machine-default -p 50571 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true] 
ssh: connect to host localhost port 50571: Connection refused
DEBU[0016] SSH readiness check for machine failed: exit status 255 
DEBU[0018] Executing: ssh [-i /Users/ameen/.ssh/podman-machine-default -p 50571 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true] 
ssh: connect to host localhost port 50571: Connection refused
DEBU[0018] SSH readiness check for machine failed: exit status 255 
DEBU[0022] Executing: ssh [-i /Users/ameen/.ssh/podman-machine-default -p 50571 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true] 
ssh: connect to host localhost port 50571: Connection refused
DEBU[0022] SSH readiness check for machine failed: exit status 255 
DEBU[0030] Executing: ssh [-i /Users/ameen/.ssh/podman-machine-default -p 50571 core@localhost -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=ERROR -o SetEnv=LC_ALL= true] 
ssh: connect to host localhost port 50571: Connection refused
DEBU[0030] SSH readiness check for machine failed: exit status 255 
Error: machine did not transition into running state: ssh error: exit status 255
DEBU[0030] Shutting down engines

@ashley-cui I also tried to cleaning up ~/.ssh/known_hosts , it didn't help.

@baude this issue seems to be similar, but the version I use is 4.9.3

any workarounds to fix this?

campbsb commented 3 months ago

I had the same issue. Running the ssh command manually with a -v flag showed that, in my case, the issue was caused by a default ProxyJump setting in my .ssh/config. Adding the following resolved it:

Host localhost
    ProxyJump none
ashley-cui commented 1 week ago

Is this still an issue with 5.0?