search

abiosoft / colima

Container runtimes on macOS (and Linux) with minimal setup
MIT License
17.75k stars 365 forks source link

Creating aarch64 instance on intel macOS hangs on starting ... context=docker #963

Closed marc-ostrow closed 5 months ago

marc-ostrow commented 5 months ago

Description

From initial install of colima with brew, starting the initial, the default instance with aarch64 architecture never completes.

Version

marc@USLC02C30YTMD6T ~ % colima version && limactl --version && qemu-img --version
colima version 0.6.7
git commit: ba1be00e9aec47f2c1ffdacfb7e428e465f0b58a
limactl version 0.19.1
qemu-img version 8.2.0
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
marc@USLC02C30YTMD6T ~ %

Operating System

Output of colima status

colima status before creating the first instance

marc@USLC02C30YTMD6T ~ % colima status
FATA[0004] error retrieving current runtime: empty value
marc@USLC02C30YTMD6T ~ %

colima instances and status after being hung for awhile

marc@USLC02C30YTMD6T ~ % colima list
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Running    aarch64    2       2GiB      60GiB    docker
marc@USLC02C30YTMD6T ~ % colima status
FATA[0002] error retrieving current runtime: empty value
marc@USLC02C30YTMD6T ~ %

Reproduction Steps

After installing colima with brew, created first instance with colima start --arch aarch64 on an 2.3 GHz 8-Core Intel Core i9 MacBook Pro.

Expected behaviour

There would be a single colima instance in the default profile having default cpu, memory, disk and docker runtime with architecture aarch64, with a running status.

Additional context

While hung starting the instance, I'm able to ssh into the instance

marc@USLC02C30YTMD6T ~ % colima ssh
marc@colima:/Users/marc$ uname -a
Linux colima 6.5.0-10-generic #10-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 13 18:28:22 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
marc@colima:/Users/marc$ exit
logout
marc@USLC02C30YTMD6T ~ % uname -a
Darwin USLC02C30YTMD6T 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10 PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64 x86_64
marc@USLC02C30YTMD6T ~ %

Also while hung starting the instance, I'm able to list the docker context but cannot use it, for example, docker images which should return an empty list errors out:

marc@USLC02C30YTMD6T ~ % docker context list
NAME       DESCRIPTION                               DOCKER ENDPOINT                                  ERROR
colima *   colima                                    unix:///Users/marc/.colima/default/docker.sock
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
marc@USLC02C30YTMD6T ~ % docker images
error during connect: Get "http://%2FUsers%2Fmarc%2F.colima%2Fdefault%2Fdocker.sock/v1.24/images/json": EOF
marc@USLC02C30YTMD6T ~ %

Also while hung starting the instance, Docker context info

marc@USLC02C30YTMD6T ~/.docker % tree -p
[drwxr-xr-x]  .
├── [drwxr-xr-x]  cli-plugins
│   ├── [lrwxr-xr-x]  docker-buildx -> /usr/local/opt/docker-buildx/bin/docker-buildx
│   └── [lrwxr-xr-x]  docker-compose -> /usr/local/opt/docker-compose/bin/docker-compose
├── [-rw-------]  config.json
└── [drwxr-xr-x]  contexts
    └── [drwxr-xr-x]  meta
        └── [drwxr-xr-x]  f24fd3749c1368328e2b149bec149cb6795619f244c5b584e844961215dadd16
            └── [-rw-r--r--]  meta.json

5 directories, 4 files
marc@USLC02C30YTMD6T ~/.docker % cat contexts/meta/f24fd3749c1368328e2b149bec149cb6795619f244c5b584e844961215dadd16/meta.json
{"Name":"colima","Metadata":{"Description":"colima"},"Endpoints":{"docker":{"Host":"unix:///Users/marc/.colima/default/docker.sock","SkipTLSVerify":false}}}%
marc@USLC02C30YTMD6T ~/.docker %

Installing the first colima instance in default profile using foreign architecture aarch64 with verbose enabled

marc@USLC02C30YTMD6T ~ % colima start --arch aarch64 -v
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] creating and starting ...                     context=vm
> Terminal is not available, proceeding without opening an editor
> Starting the instance "colima" with VM driver "qemu"
> "Attempting to download the image" arch=aarch64 digest="sha512:bbfd97c7aa9dc0f240cbe09f75f98eb19812ce5da1536e4bca2044c0528fd409b3c8f2c3ba85e54707544e3c6619d585504b3eca7ba8b3f3fbaba141cec0181c" location="https://github.com/abiosoft/colima-core/releases/download/v0.6.7/ubuntu-23.10-minimal-cloudimg-arm64.qcow2"
> Using cache "/Users/marc/Library/Caches/lima/download/by-url-sha256/c882ac9804a77fb38b79073ee4e799de6c23c9f90f3a472388bfefea8549335e/data"
> [hostagent] hostagent socket created at /Users/marc/.colima/_lima/colima/ha.sock
> [hostagent] Attempting to download UEFI code https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2
> [hostagent] decompressing .bz2 with [bzip2 -d]
> [hostagent] Decompressing data

> [hostagent] 1.51 MiB / 1.51 MiB (100.00%) ? p/s
> [hostagent] Using cache "/Users/marc/Library/Caches/lima/download/by-url-sha256/9d6efeae76809fa9a7d0bd6097f40b30fea2b58f88dfe246d81a00cca4a6f1ad/data"
> [hostagent] Using firmware "/Users/marc/.colima/_lima/colima/qemu-efi-code.fd" (downloaded from "https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2")
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/marc/.colima/_lima/colima/serial*.log")
> SSH Local Port: 54070
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] The essential requirement 1 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 4: "user session is ready for ssh"
> [hostagent] Waiting for the essential requirement 2 of 4: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 3 of 4: "sshfs binary to be installed"
> [hostagent] The essential requirement 3 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 4 of 4: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
> [hostagent] Waiting for the essential requirement 4 of 4: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
> [hostagent] The essential requirement 4 of 4 is satisfied
> [hostagent] Mounting "/Users/marc" on "/Users/marc"
> [hostagent] Mounting "/tmp/colima" on "/tmp/colima"
> [hostagent] Waiting for the guest agent to be running
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/docker.sock" (host)
> [hostagent] Guest agent is running
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Not forwarding TCP 127.0.0.53:53
> [hostagent] Not forwarding TCP 127.0.0.54:53
> [hostagent] Not forwarding TCP [::]:22
> [hostagent] The final requirement 1 of 1 is satisfied
> READY. Run `limactl shell colima` to open the shell.
INFO[0376] provisioning ...                              context=docker
> colima
> Successfully created context "colima"
> colima
> Current context is now "colima"
INFO[0383] starting ...                                  context=docker
marc-ostrow commented 5 months ago

networks.yaml.md _lima_networks.md _lima-dir.md serialv.log serialp.log serial.log ha.stdout.log ha.stderr.log

marc-ostrow commented 5 months ago

macOS Version

marc@USLC02C30YTMD6T ~ % sw_vers
ProductName:        macOS
ProductVersion:     14.2.1
BuildVersion:       23C71
marc@USLC02C30YTMD6T ~ %
marc-ostrow commented 5 months ago

After a long time, the log updates with a SIGINT being sent to hostagent and tries again... attaching whole log:

marc@USLC02C30YTMD6T ~ % colima start --arch aarch64 -v
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] creating and starting ...                     context=vm
> Terminal is not available, proceeding without opening an editor
> Starting the instance "colima" with VM driver "qemu"
> "Attempting to download the image" arch=aarch64 digest="sha512:bbfd97c7aa9dc0f240cbe09f75f98eb19812ce5da1536e4bca2044c0528fd409b3c8f2c3ba85e54707544e3c6619d585504b3eca7ba8b3f3fbaba141cec0181c" location="https://github.com/abiosoft/colima-core/releases/download/v0.6.7/ubuntu-23.10-minimal-cloudimg-arm64.qcow2"
> Using cache "/Users/marc/Library/Caches/lima/download/by-url-sha256/c882ac9804a77fb38b79073ee4e799de6c23c9f90f3a472388bfefea8549335e/data"
> [hostagent] hostagent socket created at /Users/marc/.colima/_lima/colima/ha.sock
> [hostagent] Attempting to download UEFI code https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2
> [hostagent] decompressing .bz2 with [bzip2 -d]
> [hostagent] Decompressing data

> [hostagent] 1.51 MiB / 1.51 MiB (100.00%) ? p/s
> [hostagent] Using cache "/Users/marc/Library/Caches/lima/download/by-url-sha256/9d6efeae76809fa9a7d0bd6097f40b30fea2b58f88dfe246d81a00cca4a6f1ad/data"
> [hostagent] Using firmware "/Users/marc/.colima/_lima/colima/qemu-efi-code.fd" (downloaded from "https://gitlab.com/kraxel/qemu/-/raw/704f7cad5105246822686f65765ab92045f71a3b/pc-bios/edk2-aarch64-code.fd.bz2")
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/marc/.colima/_lima/colima/serial*.log")
> SSH Local Port: 54070
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] The essential requirement 1 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 4: "user session is ready for ssh"
> [hostagent] Waiting for the essential requirement 2 of 4: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 3 of 4: "sshfs binary to be installed"
> [hostagent] The essential requirement 3 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 4 of 4: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
> [hostagent] Waiting for the essential requirement 4 of 4: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
> [hostagent] The essential requirement 4 of 4 is satisfied
> [hostagent] Mounting "/Users/marc" on "/Users/marc"
> [hostagent] Mounting "/tmp/colima" on "/tmp/colima"
> [hostagent] Waiting for the guest agent to be running
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/docker.sock" (host)
> [hostagent] Guest agent is running
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Not forwarding TCP 127.0.0.53:53
> [hostagent] Not forwarding TCP 127.0.0.54:53
> [hostagent] Not forwarding TCP [::]:22
> [hostagent] The final requirement 1 of 1 is satisfied
> READY. Run `limactl shell colima` to open the shell.
INFO[0376] provisioning ...                              context=docker
> colima
> Successfully created context "colima"
> colima
> Current context is now "colima"
INFO[0383] starting ...                                  context=docker
> Sending SIGINT to hostagent process 85093
> Waiting for the host agent and the driver processes to shut down
> [hostagent] 1.51 MiB / 1.51 MiB (100.00%) ? p/s{"level":"debug","msg":"res.ValidatedDigest=true","time":"2024-01-16T01:26:01-05:00"}
> [hostagent] Received SIGINT, shutting down the host agent
> [hostagent] Shutting down the host agent
> [hostagent] Stopping forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/default/docker.sock" (host)
> [hostagent] Stopping forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/docker.sock" (host)
> [hostagent] Unmounting "/Users/marc"
> [hostagent] Unmounting "/tmp/colima"
> [hostagent] Shutting down QEMU with ACPI
> [hostagent] Sending QMP system_powerdown command
> [hostagent] QEMU has exited
> Using the existing instance "colima"
> Starting the instance "colima" with VM driver "qemu"
> [hostagent] hostagent socket created at /Users/marc/.colima/_lima/colima/ha.sock
> [hostagent] Using existing firmware ("/Users/marc/.colima/_lima/colima/qemu-efi-code.fd")
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/marc/.colima/_lima/colima/serial*.log")
> SSH Local Port: 54668
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
> [hostagent] The essential requirement 1 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 4: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 3 of 4: "sshfs binary to be installed"
> [hostagent] The essential requirement 3 of 4 is satisfied
> [hostagent] Waiting for the essential requirement 4 of 4: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
> [hostagent] The essential requirement 4 of 4 is satisfied
> [hostagent] Mounting "/Users/marc" on "/Users/marc"
> [hostagent] Mounting "/tmp/colima" on "/tmp/colima"
> [hostagent] Waiting for the guest agent to be running
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/default/docker.sock" (host)
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/marc/.colima/docker.sock" (host)
> [hostagent] Guest agent is running
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Not forwarding TCP [::]:22
> [hostagent] Not forwarding TCP 127.0.0.53:53
> [hostagent] Not forwarding TCP 127.0.0.54:53
> [hostagent] The final requirement 1 of 1 is satisfied
> READY. Run `limactl shell colima` to open the shell.
INFO[4644] done
marc@USLC02C30YTMD6T ~ %

AND IT COMPLETED! testing. Wow took almost 78 minutes total.

marc-ostrow commented 5 months ago

After quite a bit of time, the hostagent was shutdown, and another try was attempted which succeeded. It took almost 78 minutes to create the default, foreign arch (aarch64 on intel macos) colima instance. I tested running NGINX and downloading and extracting the 7 layers took a bit of time too, but it worked. Shelling into the container I get what I expect:

marc@USLC02C30YTMD6T ~ % docker context list
NAME       DESCRIPTION                               DOCKER ENDPOINT                                  ERROR
colima *   colima                                    unix:///Users/marc/.colima/default/docker.sock
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
marc@USLC02C30YTMD6T ~ % docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                                   NAMES
09f27df44dd2   nginx     "/docker-entrypoint.…"   2 minutes ago   Up 2 minutes   0.0.0.0:8080->80/tcp, :::8080->80/tcp   web
marc@USLC02C30YTMD6T ~ % docker exec -it web bash
root@09f27df44dd2:/# uname -a
Linux 09f27df44dd2 6.5.0-10-generic #10-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 13 18:28:22 UTC 2023 aarch64 GNU/Linux
root@09f27df44dd2:/# exit
exit
marc@USLC02C30YTMD6T ~ % uname -a
Darwin USLC02C30YTMD6T 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10 PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64 x86_64
marc@USLC02C30YTMD6T ~ %

nginx-welcome