jemalloc fails to compile because ASM fails

[joeshaw]

Description

This is https://github.com/jemalloc/jemalloc/issues/2318. In short, an assembly check that jemalloc does in autoconf returns a value of 0 for "number of significant virtual address bits" instead of a real value, and the build later fails.

My configuration:

colima 0.5.2 lima 0.14.2 macOS 13.1 Intel MacBook Pro 16", 2019

$ colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: 9p
INFO[0000] socket: unix:///Users/jshaw/.colima/default/docker.sock

$ limactl list
NAME      STATUS     SSH                VMTYPE    ARCH      CPUS    MEMORY    DISK      DIR
colima    Running    127.0.0.1:58929    qemu      x86_64    6       12GiB     128GiB    ~/.lima/colima

The linked issue has a nice short reproduction case here: https://github.com/jemalloc/jemalloc/issues/2318#issuecomment-1352281840

[joeshaw]

I've confirmed that this only happens with qemu and not vz.

qemu:

colima:/Users/jshaw$ gcc -o test test.c && ./test && cat conftest.out;echo
0

vz:

colima:/Users/jshaw$ gcc -o test test.c && ./test && cat conftest.out;echo
48

Unfortunately vz isn't working for me due to https://github.com/abiosoft/colima/issues/552 -- the VM is locking up hard after a short amount of time.

[afbjorklund]

You probably need to set the CPU type, the default "qemu64" doesn't have so many features...

On Linux, it returns 48 in the VM. Due to to cpuType: host. Are you emulating, from arm64 ?

[afbjorklund]

Nice, trying to set cpuType crashes the emulator. And also the error reporting.

INFO[0000] [hostagent] Driver stopped due to error: %!q(<nil>) 

DEBU[0000] [hostagent] qemu[stderr]: qemu-system-x86_64: -accel tcg,thread=multi,tb-size=512: Invalid parameter 'tb-size'

Seems like it needs an older syntax, for older qemu versions.

• https://github.com/lima-vm/lima/issues/320

[joeshaw]

No, I am running on an Intel Mac.

limactl info has this inside it:

    "defaultTemplate": {
        "vmType": "qemu",
        "arch": "x86_64",
[...]
        "cpuType": {
            "aarch64": "cortex-a72",
            "riscv64": "rv64",
            "x86_64": "host"
        },

and inside the colima template yaml I have:

# Architecture of the virtual machine (x86_64, aarch64, host).
# Default: host
arch: host

[...]

# The CPU type for the virtual machine (requires vmType `qemu`).
# Options available for host emulation can be checked with: `qemu-system-$(arch) -cpu help`.
# Instructions are also supported by appending to the cpu type e.g. "qemu64,+ssse3".
# Default: host
cpuType: host

[afbjorklund]

Possibly related to the hvf accelerator then, since both kvm and tcg return 48 here.

It should be possible to add it to the cpuType, like host,phys-bits=39 (or something)

Tried setting to something else first, think I misunderstood the default values a little...

DEBU[0000] [hostagent] qemu[stderr]: qemu-system-x86_64: warning: Host physical bits (39) does not match phys-bits property (48)

[afbjorklund]

Apparently this is supposed to happen in the QEMU CPU "realize" function, but it doesn't.

target/i386/hvf/hvf-cpu.c:    acc->cpu_realizefn = host_cpu_realizefn;
target/i386/kvm/kvm-cpu.c:    acc->cpu_realizefn = kvm_cpu_realizefn;
target/i386/tcg/tcg-cpu.c:    acc->cpu_realizefn = tcg_cpu_realizefn;

There are some comments saying realize order is important, maybe also for phys_bits ?