search

dotnet / dotnet-buildtools-prereqs-docker

Used to maintain the Docker images hosted at the mcr.microsoft.com/dotnet-buildtools/prereqs image repository
MIT License
53 stars 99 forks source link

Multiple issues when cross-compiling .NET runtime using official docker images #1002

Closed lauxjpn closed 2 months ago

lauxjpn commented 2 months ago

The issue

I am trying to build the .NET runtime for different platforms, by using the Dockerfiles for .NET Core Builds.

There are multiple issues with the cross-compilation process.

Lets assume the following simple dockerfile to cross-compile the dotnet runtime on an x64 machine for arm64:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug

If I build this dockerfile, I get the following error:

No usable version of clang found.

The reason is that while .../prereqs:ubuntu-22.04-coredeps (the base image used by .../prereqs:ubuntu-22.04-cross-arm64) installs llvm/clang via script, it does not set the installed version as the default:

Symbolic links missing for default llvm/clang usage ``` root@e269ef91fec7:/src/dotnet/runtime# ls -lah /usr/bin/*clang* lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/clang++-18 -> ../lib/llvm-18/bin/clang++ lrwxrwxrwx 1 root root 24 Mar 17 02:35 /usr/bin/clang-18 -> ../lib/llvm-18/bin/clang lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/clang-cpp-18 -> ../lib/llvm-18/bin/clang-cpp lrwxrwxrwx 1 root root 25 Mar 17 02:35 /usr/bin/clangd-18 -> ../lib/llvm-18/bin/clangd root@e269ef91fec7:/src/dotnet/runtime# ls -lah /usr/bin/*llvm* lrwxrwxrwx 1 root root 33 Mar 17 02:35 /usr/bin/llvm-addr2line-18 -> ../lib/llvm-18/bin/llvm-addr2line lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-ar-18 -> ../lib/llvm-18/bin/llvm-ar lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-as-18 -> ../lib/llvm-18/bin/llvm-as lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-bcanalyzer-18 -> ../lib/llvm-18/bin/llvm-bcanalyzer lrwxrwxrwx 1 root root 37 Mar 17 02:35 /usr/bin/llvm-bitcode-strip-18 -> ../lib/llvm-18/bin/llvm-bitcode-strip lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-cat-18 -> ../lib/llvm-18/bin/llvm-cat lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-cfi-verify-18 -> ../lib/llvm-18/bin/llvm-cfi-verify lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-config-18 -> ../lib/llvm-18/bin/llvm-config lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-cov-18 -> ../lib/llvm-18/bin/llvm-cov lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-c-test-18 -> ../lib/llvm-18/bin/llvm-c-test lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-cvtres-18 -> ../lib/llvm-18/bin/llvm-cvtres lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-cxxdump-18 -> ../lib/llvm-18/bin/llvm-cxxdump lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-cxxfilt-18 -> ../lib/llvm-18/bin/llvm-cxxfilt lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-cxxmap-18 -> ../lib/llvm-18/bin/llvm-cxxmap lrwxrwxrwx 1 root root 42 Mar 17 02:35 /usr/bin/llvm-debuginfo-analyzer-18 -> ../lib/llvm-18/bin/llvm-debuginfo-analyzer lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-debuginfod-18 -> ../lib/llvm-18/bin/llvm-debuginfod lrwxrwxrwx 1 root root 39 Mar 17 02:35 /usr/bin/llvm-debuginfod-find-18 -> ../lib/llvm-18/bin/llvm-debuginfod-find lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-diff-18 -> ../lib/llvm-18/bin/llvm-diff lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-dis-18 -> ../lib/llvm-18/bin/llvm-dis lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-dlltool-18 -> ../lib/llvm-18/bin/llvm-dlltool lrwxrwxrwx 1 root root 33 Mar 17 02:35 /usr/bin/llvm-dwarfdump-18 -> ../lib/llvm-18/bin/llvm-dwarfdump lrwxrwxrwx 1 root root 33 Mar 17 02:35 /usr/bin/llvm-dwarfutil-18 -> ../lib/llvm-18/bin/llvm-dwarfutil lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-dwp-18 -> ../lib/llvm-18/bin/llvm-dwp lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-exegesis-18 -> ../lib/llvm-18/bin/llvm-exegesis lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-extract-18 -> ../lib/llvm-18/bin/llvm-extract lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-gsymutil-18 -> ../lib/llvm-18/bin/llvm-gsymutil lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-ifs-18 -> ../lib/llvm-18/bin/llvm-ifs lrwxrwxrwx 1 root root 41 Mar 17 02:35 /usr/bin/llvm-install-name-tool-18 -> ../lib/llvm-18/bin/llvm-install-name-tool lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-jitlink-18 -> ../lib/llvm-18/bin/llvm-jitlink lrwxrwxrwx 1 root root 40 Mar 17 02:35 /usr/bin/llvm-jitlink-executor-18 -> ../lib/llvm-18/bin/llvm-jitlink-executor lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-lib-18 -> ../lib/llvm-18/bin/llvm-lib lrwxrwxrwx 1 root root 38 Mar 17 02:35 /usr/bin/llvm-libtool-darwin-18 -> ../lib/llvm-18/bin/llvm-libtool-darwin lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-link-18 -> ../lib/llvm-18/bin/llvm-link lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-lipo-18 -> ../lib/llvm-18/bin/llvm-lipo lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-lto-18 -> ../lib/llvm-18/bin/llvm-lto lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-lto2-18 -> ../lib/llvm-18/bin/llvm-lto2 lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-mc-18 -> ../lib/llvm-18/bin/llvm-mc lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-mca-18 -> ../lib/llvm-18/bin/llvm-mca lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-ml-18 -> ../lib/llvm-18/bin/llvm-ml lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-modextract-18 -> ../lib/llvm-18/bin/llvm-modextract lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-mt-18 -> ../lib/llvm-18/bin/llvm-mt lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-nm-18 -> ../lib/llvm-18/bin/llvm-nm lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-objcopy-18 -> ../lib/llvm-18/bin/llvm-objcopy lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-objdump-18 -> ../lib/llvm-18/bin/llvm-objdump lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-opt-report-18 -> ../lib/llvm-18/bin/llvm-opt-report lrwxrwxrwx 1 root root 29 Mar 17 02:35 /usr/bin/llvm-otool-18 -> ../lib/llvm-18/bin/llvm-otool lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-pdbutil-18 -> ../lib/llvm-18/bin/llvm-pdbutil lrwxrwxrwx 1 root root 38 Mar 17 02:35 /usr/bin/llvm-PerfectShuffle-18 -> ../lib/llvm-18/bin/llvm-PerfectShuffle lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-profdata-18 -> ../lib/llvm-18/bin/llvm-profdata lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-profgen-18 -> ../lib/llvm-18/bin/llvm-profgen lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-ranlib-18 -> ../lib/llvm-18/bin/llvm-ranlib lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-rc-18 -> ../lib/llvm-18/bin/llvm-rc lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-readelf-18 -> ../lib/llvm-18/bin/llvm-readelf lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-readobj-18 -> ../lib/llvm-18/bin/llvm-readobj lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-readtapi-18 -> ../lib/llvm-18/bin/llvm-readtapi lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-reduce-18 -> ../lib/llvm-18/bin/llvm-reduce lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-remarkutil-18 -> ../lib/llvm-18/bin/llvm-remarkutil lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-rtdyld-18 -> ../lib/llvm-18/bin/llvm-rtdyld lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-sim-18 -> ../lib/llvm-18/bin/llvm-sim lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-size-18 -> ../lib/llvm-18/bin/llvm-size lrwxrwxrwx 1 root root 29 Mar 17 02:35 /usr/bin/llvm-split-18 -> ../lib/llvm-18/bin/llvm-split lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-stress-18 -> ../lib/llvm-18/bin/llvm-stress lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-strings-18 -> ../lib/llvm-18/bin/llvm-strings lrwxrwxrwx 1 root root 29 Mar 17 02:35 /usr/bin/llvm-strip-18 -> ../lib/llvm-18/bin/llvm-strip lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-symbolizer-18 -> ../lib/llvm-18/bin/llvm-symbolizer lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-tblgen-18 -> ../lib/llvm-18/bin/llvm-tblgen lrwxrwxrwx 1 root root 35 Mar 17 02:35 /usr/bin/llvm-tli-checker-18 -> ../lib/llvm-18/bin/llvm-tli-checker lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-undname-18 -> ../lib/llvm-18/bin/llvm-undname lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-windres-18 -> ../lib/llvm-18/bin/llvm-windres lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-xray-18 -> ../lib/llvm-18/bin/llvm-xray ```

There are e.g. no symbolic links for /usr/bin/clang, but only for /usr/bin/clang-18. The dotnet runtime build system does not check the installed llvm/clang versions by default, but just tries to call clang, which then fails.

I have to explicitly specifiying the --clang18 parameter to build.sh, which requires me to know that the installed llvm version is 18:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug --clang18

(It probably would also work to add /usr/lib/llvm-18/bin/ to the PATH environment variable instead.)

This now starts the compilation process, until it hits the following error:

[ 23%] Building CXX object nativeaot/Runtime/eventpipe/CMakeFiles/eventpipe-enabled.dir/src/dotnet/runtime/src/native/containers/dn-umap.c.o
  [ 23%] Built target System.Security.Cryptography.Native.OpenSsl-Static
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:89:9: error: use of undeclared identifier 'va_start'
     89 |         va_start(ap, nFlags);
        |         ^
  [ 23%] Building CXX object nativeaot/Runtime/eventpipe/CMakeFiles/eventpipe-enabled.dir/src/dotnet/runtime/src/native/containers/dn-vector.c.o
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:91:9: error: use of undeclared identifier 'va_end'
     91 |         va_end(ap);
        |         ^
  [ 23%] Building CXX object gcinfo/CMakeFiles/gcinfo_win_x86.dir/simplerhash.cpp.o
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:126:9: error: use of undeclared identifier 'va_start'
  [ 23%] Building CXX object md/compiler/CMakeFiles/mdcompiler_ppdb.dir/regmeta.cpp.o
    126 |         va_start(ap, nFlags);
        |         ^
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:128:9: error: use of undeclared identifier 'va_end'
    128 |         va_end(ap);
        |         ^
  4 errors generated.
  make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:90: pal/src/CMakeFiles/coreclrpal.dir/cruntime/filecrt.cpp.o] Error 1
  make[1]: *** [CMakeFiles/Makefile2:2270: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2
  make[1]: *** Waiting for unfinished jobs....

Looks like the source code is incompatible with llvm/clang 18.

To fix this issue, I install and use llvm/clang 14 instead:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
# Install llvm-toolchain version 14 using official script.
RUN wget -O- https://apt.llvm.org/llvm.sh | bash -s -- 14 \
        clang \
        clang-tools \
        liblldb-dev \
        lld \
        lldb \
        llvm \
        python3-lldb \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug --clang14

If I build this docker file, the runtime build continues further, until I hit:

Generating native code
  objcopy: Unable to recognise the format of the input file `/src/dotnet/runtime/artifacts/bin/coreclr/linux.arm64.Debug/ilc/native/ilc'
/root/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-rc.1.23406.6/build/Microsoft.NETCore.Native.targets(376,5): error MSB3073: The command ""objcopy" --only-keep-debug "/src/dotnet/runtime/artifacts/bin/coreclr/linux.arm64.Debug/ilc/native/ilc" "/src/dotnet/runtime/artifacts/bin/coreclr/linux.arm64.Debug/ilc/native/ilc.dbg"" exited with code 1. [/src/dotnet/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

It appears that the wrong version of objcopy (the not-cross-compile version) is used by the build system.

There seems to be an ObjCopyName msbuild property that can be set to the name of the file that should be used instead, which would be llvm-objcopy-14:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
# Install llvm-toolchain version 14 using official script.
RUN wget -O- https://apt.llvm.org/llvm.sh | bash -s -- 14 \
        clang \
        clang-tools \
        liblldb-dev \
        lld \
        lldb \
        llvm \
        python3-lldb \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV OBJCOPYNAME=llvm-objcopy-14
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug --clang14

(Appending -p:ObjCopyName=llvm-objcopy-14 to the build.sh parameter list instead would probably work as well.)

This dockerfile is now finally able to finish the build process successfully.

Suggestions

It took me quite some time to analyze and fix these issues. Currently, it is unreasonably hard to build a cross-compiled version of the dotnet runtime for someone who has never done it before.

lbussell commented 2 months ago

[Triage] @richlander @sbomer do either of you have any input on this issue with cross-compiling the .NET runtime?

sbomer commented 2 months ago

@lauxjpn the image you are using (mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64) was specifically added for our .NET 7 builds, before we transitioned to using CBL-Mariner in official builds in .NET 8 (and Azure Linux 3.0 in .NET 9).

I'd recommend checking https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml which lists the images we use for cross-building in our ci. For 8.0.4, you'll have better luck with mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64.

richlander commented 2 months ago

There are good instructions here: https://github.com/dotnet/runtime/blob/b194416ea68e2d1c93a91fc7abf80eb2607b4831/docs/workflow/building/coreclr/linux-instructions.md

Also checkout https://github.com/dotnet/dotnet

lauxjpn commented 2 months ago

There are good instructions here: https://github.com/dotnet/runtime/blob/b194416ea68e2d1c93a91fc7abf80eb2607b4831/docs/workflow/building/coreclr/linux-instructions.md

Yes, all those runtime workflow docs are the first ones I read. But since the docs say This table of images might often become stale as we change our images as our requirements change. and because the table mentioned the long outdated Alpine 3.13 and Ubuntu 16.04 and I didn't recognize mariner, so I thought it is probably some very old distribution code name, I directly went to the referenced dotnet-buildtools-prereqs-docker repo instead and then chose a cross-compilation enabled dockerfile of a distribution that I recognized (so that I can easily extend it if necessary) and that seemed current.

Assumption is the mother of all mistakes.

@lauxjpn the image you are using (mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64) was specifically added for our .NET 7 builds, before we transitioned to using CBL-Mariner in official builds in .NET 8 (and Azure Linux 3.0 in .NET 9).

I'd recommend checking https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml which lists the images we use for cross-building in our ci. For 8.0.4, you'll have better luck with mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64.

Thanks, yes, the cbl-mariner-2.0-cross images do just work out-of-the-box.

I guess we can close this one.

richlander commented 2 months ago

The table is accurate, however, I grasp the assumption.

This issue has relevant context: https://github.com/dotnet/runtime/pull/100942

It is on my list of TODOs to update that table to reflect the state of that merged PR.

We use a cross build approach for everything and use very old targets to ensure that our users can run our binaries (for the most part) everywhere. It enables us to both satisfy secure supply chain requirements and offer binary compat. That's very hard to do well.