purpleprotocol / mimalloc_rust

A Rust wrapper over Microsoft's MiMalloc memory allocator
MIT License
486 stars 42 forks source link

Failure compilation on windows cross-compile #125

Closed xMAC94x closed 2 weeks ago

xMAC94x commented 2 weeks ago

Hey folks, we at veloren use cross-compile from linux for windows builds. since the latest version we get a compilation error which looks like this. Seems to be a C constant missing, however I couldn't find neither ERROR_COMMITMENT_LIMIT nor ERROR_COMMITMENT_MINIMUM.

Is there a new version available which works on windows? e.g. with a never mimalloc subrepo ?

Note: windows, linux and macos compilation on respectives hosts works flawlessly, it seems to be just the cross compilation. BaseOS us ubuntu 22.04

Any Idea what might be wrong ?

warning: libmimalloc-sys@0.1.39: In file included from c_src/mimalloc/src/prim/prim.c:12,
warning: libmimalloc-sys@0.1.39:                  from c_src/mimalloc/src/static.c:38:
warning: libmimalloc-sys@0.1.39: c_src/mimalloc/src/prim/windows/prim.c: In function 'win_is_out_of_memory_error':
warning: libmimalloc-sys@0.1.39: c_src/mimalloc/src/prim/windows/prim.c:212:10: error: 'ERROR_COMMITMENT_MINIMUM' undeclared (first use in this function); did you mean 'ERROR_COMMITMENT_LIMIT'?
warning: libmimalloc-sys@0.1.39:   212 |     case ERROR_COMMITMENT_MINIMUM:
warning: libmimalloc-sys@0.1.39:       |          ^~~~~~~~~~~~~~~~~~~~~~~~
warning: libmimalloc-sys@0.1.39:       |          ERROR_COMMITMENT_LIMIT
warning: libmimalloc-sys@0.1.39: c_src/mimalloc/src/prim/windows/prim.c:212:10: note: each undeclared identifier is reported only once for each function it appears in
error: failed to run custom build command for `libmimalloc-sys v0.1.39`

Full logs: https://gitlab.com/veloren/veloren-docker-ci/-/jobs/7300415963

``` Compiling windows v0.54.0 The following warnings were emitted during compilation: warning: libmimalloc-sys@0.1.39: In file included from c_src/mimalloc/src/prim/prim.c:12, warning: libmimalloc-sys@0.1.39: from c_src/mimalloc/src/static.c:38: warning: libmimalloc-sys@0.1.39: c_src/mimalloc/src/prim/windows/prim.c: In function 'win_is_out_of_memory_error': warning: libmimalloc-sys@0.1.39: c_src/mimalloc/src/prim/windows/prim.c:212:10: error: 'ERROR_COMMITMENT_MINIMUM' undeclared (first use in this function); did you mean 'ERROR_COMMITMENT_LIMIT'? warning: libmimalloc-sys@0.1.39: 212 | case ERROR_COMMITMENT_MINIMUM: warning: libmimalloc-sys@0.1.39: | ^~~~~~~~~~~~~~~~~~~~~~~~ warning: libmimalloc-sys@0.1.39: | ERROR_COMMITMENT_LIMIT warning: libmimalloc-sys@0.1.39: c_src/mimalloc/src/prim/windows/prim.c:212:10: note: each undeclared identifier is reported only once for each function it appears in error: failed to run custom build command for `libmimalloc-sys v0.1.39` Caused by: process didn't exit successfully: `/dockercache/veloren/target/release/build/libmimalloc-sys-1bd5bb56500e943c/build-script-build` (exit status: 1) --- stdout OPT_LEVEL = Some(3) TARGET = Some(x86_64-pc-windows-gnu) OUT_DIR = Some(/dockercache/veloren/target/x86_64-pc-windows-gnu/release/build/libmimalloc-sys-9eb4a440d2a97933/out) HOST = Some(x86_64-unknown-linux-gnu) cargo:rerun-if-env-changed=CC_x86_64-pc-windows-gnu CC_x86_64-pc-windows-gnu = None cargo:rerun-if-env-changed=CC_x86_64_pc_windows_gnu CC_x86_64_pc_windows_gnu = None cargo:rerun-if-env-changed=TARGET_CC TARGET_CC = None cargo:rerun-if-env-changed=CC CC = None cargo:rerun-if-env-changed=CROSS_COMPILE CROSS_COMPILE = None RUSTC_LINKER = Some(/usr/bin/x86_64-w64-mingw32-gcc) cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT RUSTC_WRAPPER = None cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS CRATE_CC_NO_DEFAULTS = None DEBUG = Some(false) CARGO_CFG_TARGET_FEATURE = Some(cmpxchg16b,fxsr,lahfsahf,sse,sse2,sse3) cargo:rerun-if-env-changed=CFLAGS_x86_64-pc-windows-gnu CFLAGS_x86_64-pc-windows-gnu = None cargo:rerun-if-env-changed=CFLAGS_x86_64_pc_windows_gnu CFLAGS_x86_64_pc_windows_gnu = None cargo:rerun-if-env-changed=TARGET_CFLAGS TARGET_CFLAGS = None cargo:rerun-if-env-changed=CFLAGS CFLAGS = None cargo:warning=In file included from c_src/mimalloc/src/prim/prim.c:12, cargo:warning= from c_src/mimalloc/src/static.c:38: cargo:warning=c_src/mimalloc/src/prim/windows/prim.c: In function 'win_is_out_of_memory_error': cargo:warning=c_src/mimalloc/src/prim/windows/prim.c:212:10: error: 'ERROR_COMMITMENT_MINIMUM' undeclared (first use in this function); did you mean 'ERROR_COMMITMENT_LIMIT'? cargo:warning= 212 | case ERROR_COMMITMENT_MINIMUM: cargo:warning= | ^~~~~~~~~~~~~~~~~~~~~~~~ cargo:warning= | ERROR_COMMITMENT_LIMIT cargo:warning=c_src/mimalloc/src/prim/windows/prim.c:212:10: note: each undeclared identifier is reported only once for each function it appears in --- stderr error occurred: Command "/usr/bin/x86_64-w64-mingw32-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-I" "c_src/mimalloc/include" "-I" "c_src/mimalloc/src" "-Wall" "-Wextra" "-DMI_DEBUG=0" "-o" "/dockercache/veloren/target/x86_64-pc-windows-gnu/release/build/libmimalloc-sys-9eb4a440d2a97933/out/98cfcaec7182b1d8-static.o" "-c" "c_src/mimalloc/src/static.c" with args x86_64-w64-mingw32-gcc did not execute successfully (status code exit status: 1). warning: build failed, waiting for other jobs to finish... ```
nathaniel-daniel commented 2 weeks ago

Those are windows error code constants. Either mimalloc messed up their includes somehow, or you messed up your cross-compilation environment. Giving the relevant file a quick once-over, I don't see anything immediately wrong with it, but I might have missed something. Maybe try to compile mimalloc separately in a similar environment?

xMAC94x commented 2 weeks ago

I can give it a try, we just reverted the latest version here: https://gitlab.com/veloren/veloren/-/merge_requests/4522/diffs?commit_id=54e3cef22362825b3c998c746b48845484a1c9b9 and the result is a flawless compilation https://gitlab.com/veloren/veloren-docker-ci/-/jobs/7302239817

Edit the failing function was added since the last version libmimalloc-sys@0.1.38 and libmimalloc-sys@0.1.39 thats mimalloc 61418436 -> 8c532c32 Also the latest tag is not passing CI completely (though the logs seems to no longer exist)

xMAC94x commented 2 weeks ago

Here is a minimal reproduction scenario:

git clone  https://github.com/purpleprotocol/mimalloc_rust.git
cd mimalloc_rust
git submodule update --init --recursive

create file Dockerfile

FROM rust:1.79.0-bullseye

RUN apt-get update \
    && DEBIAN_FRONTEND=noninteractive \
    && apt-get install -y --no-install-recommends --assume-yes \
        # Needed to cross compile to windows
        gcc-mingw-w64-x86-64 \
        gcc \
        libc-dev \
        python3 \
        g++-mingw-w64-x86-64 \
    # Cleanup extra cached files
    && rm -rf /var/lib/apt/lists/*;

RUN mkdir -p /root/.cargo  &&  echo \
        "[target.x86_64-pc-windows-gnu]\nlinker = \"/usr/bin/x86_64-w64-mingw32-gcc\"\n" \
        >> /root/.cargo/config \
    && rustup target add x86_64-pc-windows-gnu

COPY . /opt
WORKDIR /opt
RUN cargo build --target=x86_64-pc-windows-gnu --release

run file docker build .

verify that old version still works: git checkout v0.1.42 , repeat docker build steps from above

Imberflur commented 2 weeks ago

looks like that installs mingw 8 and version 10 is needed: version correspondence https://www.mingw-w64.org/downloads/ commit adding error code https://github.com/mirror/mingw-w64/commit/558907c052696dc20fea9a5c5140fae7c51ed2b8

NeuralModder commented 2 weeks ago

Quick test would be to change the rust:1.79.0-bullseye in the Dockerfile to rust:1.79.0-bookworm, as that has mingw 10. Don't currently have docker set up on my computer to test.

NeuralModder commented 2 weeks ago

Can confirm: switching to bookworm indeed fixes the problem, which verifies that a mingw version requirement was the culprit.

This issue can therefore be closed.

xMAC94x commented 2 weeks ago

Thanks for analysing this, I am going to close this issue, the workaround is: update your OS to one with the respective mingw version of at least: libmimalloc-sys@v0.1.38 -> 8 libmimalloc-sys@v0.1.39 -> 10 if you want to use this crate.