Update coredistools

[BruceForstall]

The CI build of coredistools is currently broken, due to the addition of Loongarch64 when the CI was disabled (https://github.com/dotnet/jitutils/pull/367), and other bit rot.

The CI is https://dev.azure.com/dnceng-public/public/_build?definitionId=253&_a=summary.

Also, the build is based on LLVM 13.0.1 (https://github.com/dotnet/jitutils/pull/351). Try to update to 16.0.1 to address failure to disassemble vextracti64x4 instruction: https://github.com/dotnet/runtime/issues/84967.

[Update: the vextracti64x4 problem was a bug in JIT encoding, now resolved.]

Update cross-build containers to use CBL-Mariner, and cross-build linux-x64, to match dotnet/runtime (https://github.com/dotnet/runtime/pull/84148, https://github.com/dotnet/runtime/pull/84863)

Building using LLVM 16.0.1 requires changing the build to build tblgen on Linux and Mac (like we do for Windows) since the LLVM releases don't contain all pre-built images we currently use to avoid building tblgen: https://github.com/llvm/llvm-project/releases. It looks like llvm-tblgen does exist in the CBL-Mariner images, but it's probably best to build it from source like Windows/Mac to ensure it is from the right build/source tree.

[BruceForstall]

@sbomer I'm planning to update the coredistools build to cross-build using CBL-Mariner images (instead of Ubuntu), and also cross-build for linux-x64. These will all build on a ubuntu-latest host. Is that appropriate? Or should we keep building as we did before, with older ubuntu18.04 cross-build arm/arm64 containers, and native linux-x64?

[BruceForstall]

Looks like switching to CBL-Mariner will be more work. The images are missing:

1. strip (from binutils)
2. which (can replace with type -p?)

[sbomer]

I don't have full context on coredistools, but I have a few thoughts:

• The approach sounds reasonable to me in general. It looks similar to how we build the compiler-rt profile library against a sysroot in https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/cbl-mariner/2.0/cross/arm64/Dockerfile, for example.
• The mariner images are the new source of truth for our rootfs, which would point to using them instead of the old cross-build images. It's also good to move to a more modern host OS for the build.
• You may run into issues because LLVM 16 changed to use C++ 17, but not all C++ 17 headers are available in the sysroot.
• You can probably replace strip with llvm-strip and which with command -v. Maybe type -p works the same - but in the runtime build scripts we have been using command -v.

[sbomer]

Also, the plan is to update the mariner images from LLVM 12 to LLVM 16. Once that change is made, you might be able to simplify the coredistools build by not downloading the llvm release. (tblgen looks like it will be available too).

[BruceForstall]

@sbomer It looks like Mariner doesn't have ld; it has lld instead. Do you know how to get cmake to use it?

$ ./build-coredistools.sh linux-x64 /crossrootfs/x64
/opt/code/obj/linux-x64 /opt/code
-- The C compiler identification is Clang 12.0.1
-- The CXX compiler identification is Clang 12.0.1
-- The ASM compiler identification is Clang
-- Found assembler: /usr/bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /usr/bin/clang
-- Check for working C compiler: /usr/bin/clang - broken
CMake Error at /usr/share/cmake-3.21/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/usr/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /opt/code/obj/linux-x64/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make -f Makefile cmTC_06ace/fast && /usr/bin/make  -f CMakeFiles/cmTC_06ace.dir/build.make CMakeFiles/cmTC_06ace.dir/build
    make[1]: Entering directory '/opt/code/obj/linux-x64/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_06ace.dir/testCCompiler.c.o
    /usr/bin/clang   -target x86_64-linux-gnu --sysroot=/crossrootfs/x64  -MD -MT CMakeFiles/cmTC_06ace.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_06ace.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_06ace.dir/testCCompiler.c.o -c /opt/code/obj/linux-x64/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_06ace
    /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_06ace.dir/link.txt --verbose=1
    /usr/bin/clang -target x86_64-linux-gnu --sysroot=/crossrootfs/x64  CMakeFiles/cmTC_06ace.dir/testCCompiler.c.o -o cmTC_06ace
    clang-12: error: unable to execute command: Executable "ld" doesn't exist!
    clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [CMakeFiles/cmTC_06ace.dir/build.make:100: cmTC_06ace] Error 1
    make[1]: Leaving directory '/opt/code/obj/linux-x64/CMakeFiles/CMakeTmp'
    make: *** [Makefile:127: cmTC_06ace/fast] Error 2

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:54 (project)```

[BruceForstall]

And/or, why doesn't clang-12 in the Mariner container use lld by default?

[BruceForstall]

If I set

LDFLAGS=-fuse-ld=lld

I instead get the error:

CMake Error at /usr/share/cmake-3.21/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/usr/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /opt/code/obj/linux-x64/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make -f Makefile cmTC_202af/fast && /usr/bin/make  -f CMakeFiles/cmTC_202af.dir/build.make CMakeFiles/cmTC_202af.dir/build
    make[1]: Entering directory '/opt/code/obj/linux-x64/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_202af.dir/testCCompiler.c.o
    /usr/bin/clang   -target x86_64-linux-gnu --sysroot=/crossrootfs/x64  -MD -MT CMakeFiles/cmTC_202af.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_202af.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_202af.dir/testCCompiler.c.o -c /opt/code/obj/linux-x64/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_202af
    /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_202af.dir/link.txt --verbose=1
    /usr/bin/clang -target x86_64-linux-gnu --sysroot=/crossrootfs/x64  -fuse-ld=lld  CMakeFiles/cmTC_202af.dir/testCCompiler.c.o -o cmTC_202af
    ld.lld: error: unable to find library -lgcc_s
    ld.lld: error: unable to find library -lgcc_s
    clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [CMakeFiles/cmTC_202af.dir/build.make:100: cmTC_202af] Error 1
    make[1]: Leaving directory '/opt/code/obj/linux-x64/CMakeFiles/CMakeTmp'
    make: *** [Makefile:127: cmTC_202af/fast] Error 2

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:54 (project)

[sbomer]

You could also try -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" like we do here: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/cc2cadabadbefa364a3c397426d66428833c17f6/src/cbl-mariner/2.0/cross/arm64/Dockerfile#L19

I believe the default for clang to use ld since it's the most common platform linker, and historically the llvm tools have been almost drop-in replacements for GNU tools (which used platform linker). I don't know if there's a way to configure it so that lld becomes the default - we also set -fuse-ld=lld in the runtime build.

To debug that failure I would pass -v as an argument during the build. This should make clang tell you where it found the gcc toolchain. You'll need to do two things:

1. For cross-builds, pass --sysroot=$ROOTFS_DIR. Here's how we do that in the compiler-rt builds: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/cc2cadabadbefa364a3c397426d66428833c17f6/src/cbl-mariner/2.0/cross/arm64/Dockerfile#L8. That should be enough for it to locate the gcc toolchain in the sysroot in most cases.
2. For x64 cross-build, even with --sysroot, in my experience clang preferred the gcc installation on the host. In this case you need to specify the gcc installation yourself with --gcc-toolchain=$ROOTFS_DIR/usr.

[jakobbotsch]

It seems like our current coredistools do not support ldapur instruction that we recently started generating with https://github.com/dotnet/runtime/pull/89681. I see several instances of decode failures when running diffs of arm64 collections when we hit these instructions. So we may need to continue pushing on #370... what is the status of it?

[BruceForstall]

I haven't tried it in a while; I was waiting for the Mariner images to mature and update to a newer LLVM (with C++17 support). I believe those Mariner updates have happened (https://github.com/dotnet/runtime/pull/87190?).

[BruceForstall]

Need to undo https://github.com/dotnet/runtime/pull/91668