Inability to start colima on GitHub macOS-14 runners: "HV_UNSUPPORTED"

beelux commented 5 months ago

Description

GitHub just released their macos-14 runners in Beta, and they don't seem to be able to start colima. The symptom seems similar to #791 or even https://github.com/abiosoft/colima/issues/786#issuecomment-1693629650: starting colima fails on an HV_UNSUPPORTED issue.

I've tried appliying the hotfix, but the signature doesn't seem to be the issue, as a non-hotfixed workflow also does not complain:

time="2024-01-30T18:27:00Z" level=info msg="QEMU binary \"/opt/homebrew/bin/qemu-system-aarch64\" seems properly signed with the \"com.apple.security.hypervisor\" entitlement"

While the .colima/_lima/colima/ha.stderr.log log shows an issue:

{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/runner/.colima/_lima/colima/serial*.log\")","time":"2024-01-30T18:53:22Z"}
(...)
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2024-01-30T18:53:22Z"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2024-01-30T18:53:22Z"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/runner/.colima/_lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=runner -o ControlMaster=auto -o ControlPath=\"/Users/runner/.colima/_lima/colima/ssh.sock\" -o ControlPersist=yes -p 49217 127.0.0.1 -- /bin/bash]","time":"2024-01-30T18:53:22Z"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: Error: HV_UNSUPPORTED","time":"2024-01-30T18:53:22Z"}
{"level":"info","msg":"Driver stopped due to error: \"signal: abort trap\"","time":"2024-01-30T18:53:22Z"}

Note that the quoted logs here are shortened.

The full logs are accessible here: https://github.com/beelux/toltec-toolchain/actions/runs/7716039166/job/21031938411

Is this even a colima issue, or should I open an issue on the actions/runner-images repo?

Version

colima version 0.6.7
git commit: ba1be00e9aec47f2c1ffdacfb7e428e465f0b58a
limactl version 0.19.1
qemu-img version 8.2.0
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

Operating System

[ ] macOS Intel <= 13 (Ventura)
[ ] macOS Intel >= 14 (Sonoma)
[ ] Apple Silicon <= 13 (Ventura)
[X] Apple Silicon >= 14 (Sonoma)
[ ] Linux

Output of `colima status`

time="2024-01-30T19:15:46Z" level=fatal msg="colima is not running"

Reproduction Steps

Running a workflow on a GitHub macos-14 aarch64 runner.

Use macos-14
Install docker and colima through brew
Start colima

Snippet from a workflow file:

  armbuild:
    name: Build latest arm64
    runs-on: macos-14
    steps:
      - name: Checkout the Git repository
        uses: actions/checkout@v2                                                                                                                                                                                   
      - name: Setup Docker
        run: |
          brew install docker colima docker-buildx
      - name: Start Docker container runtimes
        run: |
          colima start

Expected behaviour

Colima should start without crashing

Additional context

No response

jandubois commented 5 months ago

I suspect the runner is a VM on a machine with an M1 chip, which doesn't support nested virtualization (you can't run a VM inside the VM). It requires M2 or later architecture...

The same limitation was true on the Windows and Linux runners until about 2 weeks ago when Github moved the runners to new hardware that does support nested virtualization.

The macOS Intel runners have supported nested virtualization as long as I can remember.

beelux commented 5 months ago

@jandubois Thanks a lot for clearing this up so quickly! The runners are indeed running on M1 chips. Somehow the nested virtualization limitation didn't cross my mind even though I actually use an M1 chip machine.

I'm guessing that there's thus no "good" way to run colima on those M1 runners, outside of emulation.

I just realized that this actually got brought up on https://github.com/actions/runner-images/issues/9254#issuecomment-1917916016, which also mentioned a line in the macOS larger runner docs:

Due to a limitation of Apple's Virtualization Framework, which our hypervisor uses, nested-virtualization is not supported by arm64 runners.

Thanks again, I was stuck and wasn't exactly sure where to ask.

abiosoft / colima