search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.79k stars 2.42k forks source link

unable to create/run a podman machine using libkrun provider with podman 5.3.0 #24559

Closed benoitf closed 5 hours ago

benoitf commented 10 hours ago

Issue Description

Creating/starting a podman machine using libkrun on macOS I have the following error Error: krunkit exited unexpectedly with exit code 1

Steps to reproduce the issue

Steps to reproduce the issue

download and install podman 5.3.0 from the releases page on macOS (sequoia 15.1)

  1. CONTAINERS_MACHINE_PROVIDER=libkrun podman machine init --now my-machine
Looking up Podman Machine image at quay.io/podman/machine-os:5.3 to create VM
Extracting compressed file: my-machine-arm64.raw: done
Machine init complete
Starting machine "my-machine"
Error: krunkit exited unexpectedly with exit code 1

Describe the results you received

Error: krunkit exited unexpectedly with exit code 1

Describe the results you expected

machine starts

podman info output

podman info                                                                                                                                  OS: darwin/arm64
provider: libkrun
version: 5.3.0

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:63626: connect: connection refused

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Tried with --log-leve=DEBUG and got this terminal window (and the command is hanging)

Image

CONTAINERS_MACHINE_PROVIDER=libkrun podman machine --log-level=DEBUG init --now my-machine3                                                
INFO[0000] podman filtering at log level debug
DEBU[0000] Using Podman machine with `libkrun` virtualization provider
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0000] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun.json is 66
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun2.json is 67
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun3.json is 67
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine.json is 72
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine2.json is 73
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0000] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0000] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json is 73
DEBU[0000] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw is 83
Looking up Podman Machine image at quay.io/podman/machine-os:5.3 to create VM
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying to access "quay.io/podman/machine-os:5.3"
DEBU[0000] Found credentials for quay.io/podman/machine-os in credential helper containers-auth.json in file /Users/benoitf/.config/containers/auth.json
DEBU[0000]  No signature storage configuration found for quay.io/podman/machine-os:5.3, using built-in default file:///Users/benoitf/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io
DEBU[0000] GET https://quay.io/v2/
DEBU[0000] Ping https://quay.io/v2/ status 401
DEBU[0000] GET https://quay.io/v2/auth?account=fbenoit&scope=repository%3Apodman%2Fmachine-os%3Apull&service=quay.io
DEBU[0001] Increasing token expiration to: 60 seconds
DEBU[0001] GET https://quay.io/v2/podman/machine-os/manifests/5.3
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0001] found image in digest: "sha256:414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7"
DEBU[0001] GET https://quay.io/v2/podman/machine-os/manifests/sha256:414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0004] original artifact file name: podman-machine.aarch64.applehv.raw.zst
DEBU[0004] GET https://quay.io/v2/podman/machine-os/manifests/sha256:414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0004] original artifact file name: podman-machine.aarch64.applehv.raw.zst
DEBU[0004] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache/414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7.raw.zst is 140
DEBU[0004] cached image exists and is latest: /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache/414a851157f853ca4fc6bf6755f1acffea33a59e1cbf86116e5c0c61e2eb38e7.raw.zst
DEBU[0004] Detected compression format zstd
Extracting compressed file: my-machine3-arm64.raw: done
DEBU[0012] --> imagePath is "/Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw"
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign is 72
DEBU[0012] resizing /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw to 107374182400 bytes
DEBU[0012] writing ignition file to "/Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign"
DEBU[0012] writing configuration file "/Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json"
Machine init complete
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json is 73
Starting machine "my-machine3"
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun.json is 66
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun2.json is 67
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/krun3.json is 67
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine.json is 72
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine2.json is 73
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json is 73
DEBU[0012] connection refused: http://localhost:63860/vm/state
DEBU[0012] connection refused: http://localhost:64052/vm/state
DEBU[0012] connection refused: http://localhost:64269/vm/state
DEBU[0012] connection refused: http://localhost:49277/vm/state
DEBU[0012] connection refused: http://localhost:65067/vm/state
DEBU[0012] connection refused: http://localhost:49345/vm/state
DEBU[0012] writing configuration file "/Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.json"
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.pid is 67
DEBU[0012] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun is 56
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun is 61
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/cache is 67
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman is 55
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-api.sock is 76
DEBU[0012] socket length for /Users/benoitf/.local/share/containers/podman/machine/podman.sock is 65
DEBU[0012] {true 1500 map[forward-dest:[/run/user/501/podman/podman.sock] forward-identity:[/Users/benoitf/.local/share/containers/podman/machine/machine] forward-sock:[/var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-api.sock] forward-user:[core]] [] map[] /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.log /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.pid 49327}
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock is 80
DEBU[0012] gvproxy command-line: /opt/podman/bin/gvproxy -debug -mtu 1500 -ssh-port 49327 -listen-vfkit unixgram:///var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock -forward-user core -forward-identity /Users/benoitf/.local/share/containers/podman/machine/machine -forward-sock /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-api.sock -forward-dest /run/user/501/podman/podman.sock -pid-file /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.pid -log-file /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/gvproxy.log
DEBU[0012] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock is 80
DEBU[0012] checking that "gvproxy" socket is ready
DEBU[0013] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.log is 71
DEBU[0013] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.sock is 72
DEBU[0013] helper binary path is: /opt/podman/bin/krunkit
DEBU[0013] socket length for /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock is 87
DEBU[0013] first boot detected
DEBU[0013] serving ignition file over /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock
DEBU[0013] listening for ready on: /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.sock
DEBU[0013] socket length for /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign is 72
DEBU[0013] reading ignition file: /Users/benoitf/.config/containers/podman/machine/libkrun/my-machine3.ign
DEBU[0013] waiting for ready notification
DEBU[0013] helper command-line: [/opt/podman/bin/krunkit --cpus 6 --memory 2048 --bootloader efi,variable-store=/Users/benoitf/.local/share/containers/podman/machine/libkrun/efi-bl-my-machine3,create --device virtio-blk,path=/Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-arm64.raw --device virtio-rng --device virtio-vsock,port=1025,socketURL=/var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3.sock,listen --device virtio-net,unixSocketPath=/var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/my-machine3-gvproxy.sock,mac=5a:94:ef:e4:0c:ee --device virtio-fs,sharedDir=/Users,mountTag=a2a0ee2c717462feb1de2f5afd59de5fd2d8 --device virtio-fs,sharedDir=/private,mountTag=71708eb255bc230cd7c91dd26f7667a7b938 --device virtio-fs,sharedDir=/var/folders,mountTag=a0bb3a2c8b0b02ba5958b0576f0d6530e104 --restful-uri tcp://localhost:49345 --device virtio-gpu,width=800,height=600 --device virtio-input,pointing --device virtio-input,keyboard --gui --device virtio-vsock,port=1024,socketURL=/Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock,listen]
DEBU[0013] socket length for /var/folders/_s/rlbw3qg15vx0j6vr4l7kd1hm0000gn/T/podman/krunkit-debug.sh is 72
DEBU[0013] ignition socket device: /Users/benoitf/.local/share/containers/podman/machine/libkrun/my-machine3-ignition.sock

If I switch to podman v5.2.5 I'm able to create and start podman machines

benoitf commented 9 hours ago

it seems related to https://github.com/containers/krunkit/issues/17 or any other fixes made in this repository as I'm able to start using a version compiled from the main branch

Luap99 commented 8 hours ago

Starts fine on my system but I don't have macos 15 installed.

cc @jakecorrenti @slp @baude

benoitf commented 8 hours ago

@Luap99 yes it does not seem to affect all the person using macOS

But with 5.2.5 it works on my computer and not with 5.3.0 using krunkit

jakecorrenti commented 8 hours ago

Taking a look

jakecorrenti commented 7 hours ago

I'm unable to reproduce by installing 5.3 from the installer on macOS 15.1. I'm still able to start a machine with krunkit v0.1.3 and podman 5.3

jakecorrenti commented 7 hours ago

@benoitf have you tried using the upstream main branch of krunkit to see if that solved your issue? There's been a few commits to deal with memory limits and it will provide a --krun-log-level flag to get some more information

benoitf commented 7 hours ago

it seems the issue is with machine with a lot of memory (so not all computers are affected)

and yes I'm able to start with the main branch krunkit

without the latest patch (fixing some memory issues), but the patch with krun-log-level

I have

[2024-11-14T13:25:58Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x102674000 guest_addr=0 len=8000000
[2024-11-14T13:25:58Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x300000000 guest_addr=40000000 len=80000000
[2024-11-14T13:25:58Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x7000000000 guest_addr=100000000 len=1800000000
[2024-11-14T13:25:58Z ERROR krun] Building the microVM failed: Internal(Vm(SetUserMemoryRegion(MemoryMap)))
Error: unable to begin running krun workload
benoitf commented 7 hours ago

@jakecorrenti this is what I was referencing there: https://github.com/containers/podman/issues/24559#issuecomment-2476121325

jakecorrenti commented 7 hours ago

I missed that comment. Seems like we might need to make a krunkit 0.1.4