Can't use rosetta for x86_64?

[pidgezero-one]

Description

Not sure if this is actually a bug, but I wasn't sure of a more appropriate place to put it.

For background, I'm experiencing problems using Puppeteer in a Docker container. I use Colima and don't have any alternatives installed. My container platform is linux/amd64 to match the production environment for this app, but my personal host device for developing it is a M1 mac running Ventura 13.5.2.

In a minimal container, puppeteer processes fail with a ProtocolError: Protocol error (Target.createTarget): Target closed. error. I've been trying to solve this issue for about three weeks now, and I think that it might be related to my host system architecture being incompatible with the docker container's Chrome binary, and me having messed up some configuration somewhere to make that matter.

According to the issue I opened in the Puppeteer repo, the library should be working fine with my OS and chosen container platform. I was suggested to use Rosetta in Docker Desktop, but I use Colima and not Docker Desktop.

The command I'm using to start colima is is:

colima start --cpu 4 --memory 8 --disk 100 --arch x86_64 --vm-type=vz --vz-rosetta --mount-type virtiofs

When this starts running, I see the following:

 `mountType: 9p` is experimental
> "Attempting to download the image" arch=x86_64 digest="sha512:f761b807fe9ba345968df72c07f8c5abcae0c4a44976fe5595c0ff748ef693841221a70e663986c700b027cea32b7cac24d5490d4c721593c39f2b8840c362a2" location="https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.5/alpine-lima-clm-3.18.0-x86_64.iso"
> Using cache "/Users/myname/Library/Caches/lima/download/by-url-sha256/blahblahblah/data"
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/myname/.lima/colima/serial*.log")
> SSH Local Port: 50777
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
...

Where I'm concerned about here is that I've specified vz, but it's telling me that QEMU is running instead. This is not the case if I use --arch aarch64 instead of --arch x86_64 (in that case it says it's starting VZ), but that doesn't solve my problem, since the container platform absolutely needs to be amd/linux64 and not ARM.

This is also right after running colima delete.

Is this a known issue, or something that's not possible, or am I just doing something wrong?

Version

Colima Version: 0.5.5 Lima Version: 0.17.2 Qemu Version: 8.1.0

Operating System

• [ ] macOS Intel <= 12 (Monterrey)
• [ ] macOS Intel >= 13 (Ventura)
• [ ] macOS M1 <= 12 (Monterrey)
• [X] macOS M1 >= 13 (Ventura)
• [ ] Linux

Output of colima status

INFO[0000] colima is running using macOS Virtualization.Framework INFO[0000] arch: x86_64 INFO[0000] runtime: docker INFO[0000] mountType: virtiofs INFO[0000] socket: unix:///Users/myname/.colima/default/docker.sock

Reproduction Steps

1. Start colima: colima start --cpu 4 --memory 8 --disk 100 --arch x86_64 --vm-type=vz --vz-rosetta --mount-type virtiofs
2. Check out this repo: https://github.com/juananpe/puppeteer-in-docker-in-m1
3. Run docker build . --platform=linux/amd64 -t puppeteerm1 -f Dockerfile
4. Run docker run -it --init puppeteerm1 /bin/bash
5. Run node test.js

Expected behaviour

Expect the provided repo code to output an image file to the working directory of the docker container.

Additional context

No response

[AkihiroSuda]

You have to create an aarch64 VM with --vz-rosetta, and you can run docker (build|run) --platform=linux/amd64 inside it

[pidgezero-one]

You have to create an aarch64 VM with --vz-rosetta, and you can run docker (build|run) --platform=linux/amd64 inside it

Thanks, so to do this, I could just create a new Colima profile:

colima start --profile rosetta --cpu 4 --memory 8 --disk 100 --arch aarch64 --vm-type=vz --vz-rosetta --mount-type virtiofs

And then use it with docker context use colima-rosetta, and then proceed to docker run --platform=linux/amd64 ...

Is that right?

[1ikeadragon]

I created an aarch64 vm and used all the suggested flags but when I get into it, I can't run 32-bit binaries and I can't debug 64-bit binaries either. So essentially, rosetta and vz setup doesn't work! I had to rollback to painfully slow qemu and make an 86_64 arch VM.

https://github.com/1ikeadragon/pwntainer

This is what I'm using for now.

[abiosoft]

@1ikeadragon how are you starting Collima? Can you share the exact commands.

[1ikeadragon]

@abiosoft Exact command is the one mentioned in this issue: colima start --profile rosetta --cpu 4 --memory 8 --disk 100 --arch aarch64 --vm-type=vz --vz-rosetta --mount-type virtiofs

this does start an aarch64 vm with rosetta and all that and can run x86_64 binaries but as I mentioned, can't debug it.

[abiosoft]

@1ikeadragon can you run this command and see if you're able to run 32-bit binaries afterwards?

docker run --privileged --rm tonistiigi/binfmt --install all

[1ikeadragon]

@abiosoft I could run the binaries anyway the issue as I had mentioned is I can't debug them. This is output after following what you said:

Container cmd:

docker run --security-opt seccomp=unconfined --privileged --cap-add=SYS_PTRACE -v ./:/bal -it ubuntu:latest bash

[abiosoft]

@1ikeadragon looking at the readme of your project (https://github.com/1ikeadragon/pwntainer). I see this command.

colima start -p x64 -a x86_64 -c 8 -m 4 -d 10 --mount-type virtiofs --ssh-agent --vm-type vz --vz-rosetta

Actually, once you set arch to x86_64, vz and rosetta are not used and only QEMU emulation is used. The three flags are actually ignored --mount-type virtiofs, --vm-type vz, --vz-rosetta

There is a third option which usually only works for simply containers. You can disable Rosetta and allow QEMU to do only container emulation. The VM itself would still be native arm64 but the container would be emulated as x86_64.

[1ikeadragon]

@abiosoft yes I'm aware, just didn't update the readme. Tell me more about the third option you mention. As you have understood the goal is to debug x86_64 binaries smoothly.

If I'm not mistaken you're suggesting:

colima start -p x64 -a aarch64 -c 8 -m 4 -d 10 --vm-type qemu
docker run --platform=linx/amd64 --security-opt seccomp=unconfined --privileged --cap-add=SYS_PTRACE -p 31337:31337 -v ./:/pwn -it pwn:pwn bash

[abiosoft]

@1ikeadragon yeah, exactly.

[1ikeadragon]

@abiosoft that doesn't work. We run into PTRACE_NOT_IMPLEMENTED error. I tried everything before what I settled for haha :p

[abiosoft]

@1ikeadragon yeah, full emulation would work more reliably but it would be very slow. If you can deal with the slow speed then I guess that is your best bet for now.

[1ikeadragon]

Yep @abiosoft it's slower but stable and gets the job done so I'm fine w it. Since I'm just running CTF tools like pwndbg, ROPGadget, etc.

This is definitely too slow for anything production or hosting.

Do you have any tips for making docker-buildx times faster?