Tracking issue for illumos and Solaris x86-64 port work

Cut from https://github.com/dotnet/runtime/issues/4173.

Given below is a high-level list of work items for Solaris x86-64 port:

[x] Native configurations (#34756)
[x] CoreCLR native components (#35173)
- ~~awaiting next release of libunwind https://github.com/libunwind/libunwind/releases/tag/v1.5-rc1 or higher with changes from https://github.com/libunwind/libunwind/pull/171, for Solaris support.~~
- upstream PR was merged, libunwind is updated to 1.5 by @sdmaclea in #36027.
[ ] PAL tests
- one test is failing is due to #35362.
- another one is related to raising thread priority, after lowering it from the same value as a non-root user. this either requires implementation using priocntl(2) directly for SunOS targets, or wait for https://www.illumos.org/issues/4963.
[x] Libraries native components (#34867)
- on SmartOS x86_64 with gcc 7x:
```
# from runtime repo root
./src/libraries/Native/build-native.sh -gcc
```
[x] Mono native components (#37560)
- Full mono (https://github.com/mono/mono) is already available on Solaris, requires some configurations for netcore mono in this repository.

[x] Installer native components (#34263)

on SmartOS x86_64 with gcc 7x:

# from runtime repo root
src/installer/corehost/build.sh -commithash $(git rev-parse HEAD) -gcc \
-apphostver 5.0.0-dev -hostver 5.0.0-dev -fxrver 5.0.0-dev -policyver 5.0.0-dev

[ ] MSBuild configurations
[x] CoreCLR managed components (#36266)
[ ] Libraries managed components
- Most of the partial classes can be shared with Linux (e.g. by moving it under Unix names) and modifying msbuild configurations.
[ ] Installer managed components
[x] CoreCLR tests (#37824)
[ ] Libraries tests
[ ] Mono tests
[ ] Installer tests
[ ] Packaging configurations
[x] RID (#37016)
[x] Cross compilation on Linux (dotnet/arcade#5584, dotnet/dotnet-buildtools-prereqs-docker#324, #37753)
- as done for other operating systems, e.g. FreeBSD: https://github.com/dotnet/arcade/blob/3443768/eng/common/cross/build-rootfs.sh#L243.
- script gets mirrored at https://github.com/dotnet/runtime/blob/master/eng/common/cross/build-rootfs.sh.
[x] SDK (dotnet/sdk#12198)
[ ] CI hook
- similar to https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/277 and https://github.com/dotnet/runtime/pull/34521
- external (GitHub Actions) CI has set up using the official illumos docker image: https://github.com/am11/runtime/blob/feature/sunos/ci/.github/workflows/main.yml
- example run: https://github.com/am11/runtime/actions/runs/145127885
- on tag push, it creates a GitHub release and upload artifacts (files under Shipping directory), e..g. https://github.com/am11/runtime/releases/tag/5.0.0-dev.1

Tagging subscribers to this area: @ViktorHofer Notify danmosemsft if you want to be subscribed.

CoreCLR native components is WIP with branch from libunwind that has all Solaris patches. If someone wants to pick up other items, feel free. :)

I might like to pick up this work. Any advice on next steps etc.?

@gwr, welcome! That would be a huge help! 🙂

I think the remaining items require specific domain knowledge. The next step is to port libraries like System.Diagnostics.Process (process management), System.Net (networking) and System.IO.FileSystem (filesystem). There are only few platform-dependent base libraries like these but they are necessary to execute dotnet/sdk on the platform. For System.Diagnostics.Process, the Sun's procfs binary based interface vs. linux' text based one would require us to update the existing abstractions.

For build, I have so far used cross-build setup. Basically, on x64 system with docker:

$ cd dotnet-runtime
$ docker run --rm -v$(pwd):/runtime -e ROOTFS_DIR=/crossrootfs/x64 \
    mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-illumos-20220531132048-f13d79e \
     /runtime/build.sh -c Release -os illumos -cross -gcc

# then copy runtime tarball from artifacts/packages/Release/Shipping/dotnet-runtime-9.0.0-dev-illumos-x64.tar.gz
# to the target machine and extract to be able to execute `./dotnet yourapp.dll` (yourapp.dll is built on linux,
# I can explain when we cross that bridge).

Here is a quick GitHub workflow: https://github.com/am11/CrossRepoCITesting/blob/master/.github/workflows/illumos-x64-runtime-build.yml we can employee it to upload build artifacts to a github release as well (I used to had that in my runtime fork but it was lost somewhere so I recreated a quick one in isolated repo).

Feel free to ask here or if you visit #illumos IRC channel (on libera.chat), I sometimes hang there as well.

Happy coding! 🎉

So one must use a cross-build setup? Interesting. (Makes me wonder how the first one was done :) The easiest cross build host for me would be MacOS, if that's possible. When are you usually on IRC? I may need some help getting builds going.

Here's what happens when I try to build: https://gist.github.com/gwr/3520dfbf14190e9225e8214f434ca38e Looks like the configuration etc. is not quite right yet.

Thanks

So one must use a cross-build setup? Interesting.

We can build coreclr and mono (the runtimes) on e.g. illumos host, but to build runtime libraries (.NET assemblies), we need a working SDK on the platform. The working SDK requires System.Diagnostics.Process and few others. A basic helllo-world app (something which does not use process management and networking) should be working. Hence the cross-compilation requirement.

(Makes me wonder how the first one was done :)

The first C# compiler was written in C++/assembly for Windows decades ago (until roslyn came along which itself is written in N-1 version of C#). :)

The easiest cross build host for me would be MacOS, if that's possible.

Unfortunately, that's currently not an option. We only support Apple-platforms cross build on macOS host. One of the reason is lack of interest because it requires llvm-toolchain (typically installed with homebrew) instead of AppleClang which most users have installed. So docker build is the way to go.

When are you usually on IRC? I may need some help getting builds going.

Soon 😅

Sounds like maybe I should focus on coreclr and mono first, agreed? Do you have configuration and build instructions for those parts? I haven't found those yet. BTW, there's already a patch set around that lets mono build on illumos, in case that helps. I'm in US/Eastern timee zone. You? Thanks.

CoreCLR, mono and System.Private.CoreLib / System.Runtime etc. were ported and C# hello-world type of scenarios were tested on OpenIndiana (not recently though). I just ran a build against today's main branch and it succeeded:

https://github.com/am11/CrossRepoCITesting/actions/runs/9678215572

I'm in US/Eastern timee zone. You?

I'm in Finland/EET (UTC +2).

OK, got the container running correctly. Some progress, new road blocks. See my updated gist.

Looks like you are running into a package restore issue:

Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
  Attempting to install 'sdk v9.0.100-preview.5.24307.3' from public_location.
dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.5.24307.3/dotnet-sdk-9.0.100-preview.5.24307.3-linux-x64.tar.gz
dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.5.24307.3/dotnet-sdk-9.0.100-preview.5.24307.3-linux-x64.tar.gz size is 210667542 bytes.
dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.5.24307.3/dotnet-sdk-9.0.100-preview.5.24307.3-linux-x64.tar.gz
dotnet-install: Downloaded file size is 210667542 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Installed version is 9.0.100-preview.5.24307.3
dotnet-install: Adding to current process PATH: `/runtime/.dotnet`. Note: This change will be visible only when sourcing script.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
dotnet-install: Installation finished successfully.
/runtime/artifacts/toolset/restore.proj : error : Could not resolve SDK "Microsoft.DotNet.Arcade.Sdk". Exactly one of the probing messages below indicates why we could not resolve the SDK. Investigate and resolve that message to correctly specify the SDK.
/runtime/artifacts/toolset/restore.proj : error :   SDK resolver "Microsoft.DotNet.MSBuildWorkloadSdkResolver" returned null.
/runtime/artifacts/toolset/restore.proj : error :   Failed to download package 'Microsoft.DotNet.Arcade.Sdk.9.0.0-beta.24311.10' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/microsoft.dotnet.arcade.sdk/9.0.0-beta.24311.10/microsoft.dotnet.arcade.sdk.9.0.0-beta.24311.10.nupkg'.
/runtime/artifacts/toolset/restore.proj : error : The nupkg at 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/microsoft.dotnet.arcade.sdk/9.0.0-beta.24311.10/microsoft.dotnet.arcade.sdk.9.0.0-beta.24311.10.nupkg' is not valid.
/runtime/artifacts/toolset/restore.proj : error :   Central Directory corrupt.
/runtime/artifacts/toolset/restore.proj : error :   Invalid argument : '/tmp/NuGetScratchroot/8b876b45-004a-4c1d-93e3-10d53f6d2a8d/omne45ab.3pz'
/runtime/artifacts/toolset/restore.proj : error MSB4236: The SDK 'Microsoft.DotNet.Arcade.Sdk' specified could not be found.
Build failed with exit code 1. Check errors above.

If by retrying it still runs into the same error, we can probably rule out the network issue (because it just downloaded dotnet from dotnet-install.sh). You can verify by running curl -SL https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/flat2/microsoft.dotnet.arcade.sdk/9.0.0-beta.24311.10/microsoft.dotnet.arcade.sdk.9.0.0-beta.24311.10.nupkg -o /dev/null.

Then it comes down to filesystem share. On macOS with docker, I'm using gRPC FUSE sharing implementation because the default VirtioFS runs into issue when nuget opens too many nodes (e..g I found the other day that it uses LINQ query which VirtioFS on macOS 14.5 doesn't handle well: https://github.com/NuGet/Home/issues/13572).

I think the best way for development with VMware is probably to use a linux VM instead of their containerd vctl tool. The simplest way to achieve that is to create a Debian/Ubuntu VM and use prebuilt crossrootfs. I haven't used vctl, but something like this should give us the rootfs prebuilt setup as tar.gz archive:

#!/bin/bash

# run this script from mac host

# Variables
CONTAINER_NAME="tmp-dnb"
IMAGE_NAME="mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-illumos-20220531132048-f13d79e"
HOST_DESTINATION="/tmp/crossrootfs.tar.gz" # on host

# Start the container
vctl start --name $CONTAINER_NAME $IMAGE_NAME

# Execute commands inside the container
vctl exec -it $CONTAINER_NAME /bin/sh -c "tar -czvf /tmp/crossrootfs.tar.gz /crossrootfs"

# Copy the tar.gz to host
vctl copy $CONTAINER_NAME:/tmp/crossrootfs.tar.gz $HOST_DESTINATION

# Stop and remove the container
vctl stop $CONTAINER_NAME
vctl rm $CONTAINER_NAME

echo "Process completed successfully."

Then on host copy /tmp/crossrootfs.tar.gz to Debian VM, extract it at root and export ROOTFS_DIR=/crossrootfs/x64.

Then in the VM:

$ cd runtime

# one-time setup
$ eng/install-native-dependencies.sh

# repeatable build step:
$ ./build.sh clr+libs+packs -cross -os illumos

If the runtime was cloned in shared location, then you can copy artifacts/packages/Debug/Shipping/dotnet-runtime-9.0.0-dev-illumos-x64.tar.gz from macOS host to illumos VM and extract.

OK, built and ubuntu-18.04 VM, added dep. packages, copied in /crossrootfs as advised. Trying this:

$ ./build.sh clr+libs+packs --cross --os illumos
[...snip...]
  Commencing build of "install" target in "CoreCLR component" for illumos.x64.Debug in /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug
  Invoking "/g/ws/dotnet/runtime/eng/native/gen-buildsys.sh" "/g/ws/dotnet/runtime/src/coreclr" "/g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug" x64 illumos clang Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet 
  Error: No usable version of clang found.
  Failed to generate "CoreCLR component" build project!
/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.

Build FAILED.

/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:08:03.19
Build failed with exit code 1. Check errors above.

Tried again with adding --gcc on the end:

$ ./build.sh clr+libs+packs --cross --os illumos --gcc
  Determining projects to restore...
[...snip...]
  Commencing CoreCLR Repo build
  __OutputRid: illumos-x64
  Setting up directories for build
  Checking prerequisites...
  Commencing build of "install" target in "CoreCLR component" for illumos.x64.Debug in /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug
  Invoking "/g/ws/dotnet/runtime/eng/native/gen-buildsys.sh" "/g/ws/dotnet/runtime/src/coreclr" "/g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug" x64 illumos gcc Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet 
  Error: No usable version of gcc found.
  Failed to generate "CoreCLR component" build project!
/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.

Build FAILED.

/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:03:58.19
Build failed with exit code 1. Check errors above.
gwr@ubuntu18:/g/ws/dotnet/runtime$

So the first could not find clang, and the second could not find gcc. Presumably that should be a cross-compiler, right? Where should I get that? Do I need to build that myself?

Thanks!

Yup, I forgot to add --gcc arg in the last comment. You can also use latest ubuntu (24.04). 18.04 is not a requirement.

To start fresh, rm -rf artifacts followed by ./build.. command.

Same again after removing artifacts and adding --gcc

  Commencing build of "install" target in "CoreCLR component" for illumos.x64.Debug in /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug
  Invoking "/g/ws/dotnet/runtime/eng/native/gen-buildsys.sh" "/g/ws/dotnet/runtime/src/coreclr" "/g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug" x64 illumos gcc Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet 
  Error: No usable version of gcc found.
  Failed to generate "CoreCLR component" build project!
/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.

Build FAILED.

/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:05:26.72
Build failed with exit code 1. Check errors above.

Do I need to do something to copy the cross gcc out of that docker image?

If you ran eng/install-native-dependencies.sh, it should had installed some version of gcc (via build-essential)?

https://github.com/dotnet/runtime/blob/aa343497e05510fdcd1e6a9e1abdd2843b34fd52/eng/install-native-dependencies.sh#L26

Nonetheless, it is a bug in our init-compiler script that we don't bypass for platforms where rootfs provides a separate toolchain for host machine to emit target code (illumos and android NDK). For android, we have a hack __Compiler="default", but for illumos lets just go with traditional environment variable magic:

export CLR_CC=/crossrootfs/x64/bin/x86_64-illumos-gcc export CLR_CXX=/crossrootfs/x64/bin/x86_64-illumos-g++

On 18.04, you may need a newer cmake on host:

# before
cmake --version

sudo apt purge cmake
wget -O cmake-install.sh https://github.com/Kitware/CMake/releases/download/v3.29.6/cmake-3.29.6-linux-x86_64.sh
sudo bash ./cmake-install.sh --skip-license --exclude-subdir --prefix=/usr/local
rm ./cmake-install.sh

# after
cmake --version

Here's the latest. I updated cmake as you advised, though it was already 2.39.6 before the update.

  Commencing CoreCLR Repo build
  __OutputRid: illumos-x64
  Setting up directories for build
  Checking prerequisites...
  Commencing build of "install" target in "CoreCLR component" for illumos.x64.Debug in /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug
  Invoking "/g/ws/dotnet/runtime/eng/native/gen-buildsys.sh" "/g/ws/dotnet/runtime/src/coreclr" "/g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug" x64 illumos gcc Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet 
  Not searching for unused variables given on the command line.
  loading initial cache file /g/ws/dotnet/runtime/eng/native/tryrun.cmake
  CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  -- Configuring incomplete, errors occurred!
  Failed to generate "CoreCLR component" build project!
/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.

Build FAILED.

/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:01:29.76
Build failed with exit code 1. Check errors above.
gwr@ubuntu18:/g/ws/dotnet/runtime$ cmake --version
cmake version 3.29.6

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Oh, looks like "apt install" choked on liblldb-dev We need to tell it a version. Which one? Looks like this is OK: liblldb-6.0-dev Getting farther along now...

Here's the latest. I did not expect faiures in here. Do I have version mixing or something?

  [  9%] Linking CXX static library libpalrt.a
  [  9%] Built target palrt
  [  9%] Building CXX object inc/CMakeFiles/corguids.dir/__/pal/prebuilt/idl/sospriv_i.cpp.o
  [  9%] Built target corguids
  [  9%] Built target precompiled_asm
  [  9%] Building CXX object gcinfo/CMakeFiles/gcinfo.dir/arraylist.cpp.o
  [  9%] Generating eventpipe/eventpipehelpers.cpp, eventpipe/dotnetruntime.cpp, eventpipe/dotnetruntimerundown.cpp, eventpipe/dotnetruntimestress.cpp, eventpipe/dotnetruntimeprivate.cpp
  [  9%] Building CXX object vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx.gch
  [  9%] Building CXX object gcinfo/CMakeFiles/gcinfo.dir/gcinfoencoder.cpp.o
  [  9%] Building CXX object gcinfo/CMakeFiles/gcinfo.dir/simplerhash.cpp.o
  [  9%] Built target gcinfo
  [  9%] Building CXX object vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx.o
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/contract.h:235,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/stdmacros.h:21,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:90,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx:5,
                   from <command-line>:
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h: In instantiation of 'struct has_m_revision<SBuffer, void>':
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h:268:29:   required by substitution of 'template<class Dummy> typename std::enable_if<(! has_m_revision<CONTAINER>::value), void>::type CheckedIteratorBase<SBuffer>::Resync<Dummy>(const SBuffer*) [with Dummy = SBuffer]'
  /g/ws/dotnet/runtime/src/coreclr/inc/sbuffer.inl:1729:96:   required from here
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h:206:1: error: 'int SBuffer::m_revision' is protected within this context
   DEFINE_MEMBER_EXISTENCE_CHECK(m_revision);
   ^
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/sstring.h:46,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/ex.h:18,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/metadata.h:16,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/util.hpp:15,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:97,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx:5,
                   from <command-line>:
  /g/ws/dotnet/runtime/src/coreclr/inc/sbuffer.h:520:9: note: declared protected here
       int m_revision;
           ^~~~~~~~~~
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/contract.h:235,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/stdmacros.h:21,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:90,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx:5,
                   from <command-line>:
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h:206:1: error: 'int SBuffer::m_revision' is protected within this context
   DEFINE_MEMBER_EXISTENCE_CHECK(m_revision);
   ^
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/sstring.h:46,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/ex.h:18,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/metadata.h:16,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/util.hpp:15,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:97,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx:5,
                   from <command-line>:
  /g/ws/dotnet/runtime/src/coreclr/inc/sbuffer.h:520:9: note: declared protected here
       int m_revision;
           ^~~~~~~~~~
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/contract.h:235,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/stdmacros.h:21,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:90,
                   from /g/ws/dotnet/runtime/src/native/eventpipe/ep-rt-config.h:22,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.cpp:4,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx:4:
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h: In instantiation of 'struct has_m_revision<SBuffer, void>':
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h:268:29:   required by substitution of 'template<class Dummy> typename std::enable_if<(! has_m_revision<CONTAINER>::value), void>::type CheckedIteratorBase<SBuffer>::Resync<Dummy>(const SBuffer*) [with Dummy = SBuffer]'
  /g/ws/dotnet/runtime/src/coreclr/inc/sbuffer.inl:1729:96:   required from here
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h:206:31: error: 'int SBuffer::m_revision' is protected within this context
   DEFINE_MEMBER_EXISTENCE_CHECK(m_revision);
  /g/ws/dotnet/runtime/src/coreclr/inc/check.h:28:58: note: in definition of macro 'DEFINE_MEMBER_EXISTENCE_CHECK'
   struct has_##Member<T, void_t<decltype(std::declval<T>().Member)>> : std::true_type {};
                                                            ^~~~~~
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/sstring.h:46,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/ex.h:18,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/metadata.h:16,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/util.hpp:15,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:97,
                   from /g/ws/dotnet/runtime/src/native/eventpipe/ep-rt-config.h:22,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.cpp:4,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx:4:
  /g/ws/dotnet/runtime/src/coreclr/inc/sbuffer.h:520:9: note: declared protected here
       int m_revision;
           ^~~~~~~~~~
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/contract.h:235,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/stdmacros.h:21,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:90,
                   from /g/ws/dotnet/runtime/src/native/eventpipe/ep-rt-config.h:22,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.cpp:4,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx:4:
  /g/ws/dotnet/runtime/src/coreclr/inc/iterator.h:206:31: error: 'int SBuffer::m_revision' is protected within this context
   DEFINE_MEMBER_EXISTENCE_CHECK(m_revision);
  /g/ws/dotnet/runtime/src/coreclr/inc/check.h:28:58: note: in definition of macro 'DEFINE_MEMBER_EXISTENCE_CHECK'
   struct has_##Member<T, void_t<decltype(std::declval<T>().Member)>> : std::true_type {};
                                                            ^~~~~~
  In file included from /g/ws/dotnet/runtime/src/coreclr/inc/sstring.h:46,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/ex.h:18,
                   from /g/ws/dotnet/runtime/src/coreclr/inc/metadata.h:16,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/util.hpp:15,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/common.h:97,
                   from /g/ws/dotnet/runtime/src/native/eventpipe/ep-rt-config.h:22,
                   from /g/ws/dotnet/runtime/src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.cpp:4,
                   from /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx:4:
  /g/ws/dotnet/runtime/src/coreclr/inc/sbuffer.h:520:9: note: declared protected here
       int m_revision;
           ^~~~~~~~~~
  vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/build.make:75: recipe for target 'vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx.o' failed
  make[2]: *** [vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/Unity/unity_0_cxx.cxx.o] Error 1
  CMakeFiles/Makefile2:4173: recipe for target 'vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/all' failed
  make[1]: *** [vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/all] Error 2
  make[1]: *** Waiting for unfinished jobs....
  vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/build.make:94: recipe for target 'vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx.gch' failed
  make[2]: *** [vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/cmake_pch.hxx.gch] Error 1
  CMakeFiles/Makefile2:4146: recipe for target 'vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/all' failed
  make[1]: *** [vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/all] Error 2
  Makefile:135: recipe for target 'all' failed
  /g/ws/dotnet/runtime/src/coreclr
  Failed to build "CoreCLR component".
  make: *** [all] Error 2
/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 2.

Build FAILED.

/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -x64 -debug gcc -cross -os illumos -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll"" exited with code 2.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:07:11.42
Build failed with exit code 1. Check errors above.
gwr@ubuntu18:/g/ws/dotnet/runtime$

The upper part of the logs should look something like this:

   -- The C compiler identification is GNU 8.4.0
  -- The CXX compiler identification is GNU 8.4.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /crossrootfs/x64/bin/x86_64-illumos-gcc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /crossrootfs/x64/bin/x86_64-illumos-g++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done

Ah, I was able to repro it with Debug build. I haven't tested debug build of runtime in a while. Looks like it has regressed. If you are going to work on runtime libraries instead of coreclr itself, then you can use Release build of runtime and Debug build of libraries:

./build.sh clr+libs+packs -rc Release -lc Debug

I'm fixing the debug build of runtime meanwhile.

Fix was easy: https://github.com/dotnet/runtime/pull/104118. Thanks for the catch! 😅

Thanks for the fix for -c Debug. Here's my latest try:

./build.sh clr+libs+packs -c Debug -cross -os illumos -gcc
  Commencing CoreCLR Repo build
  __OutputRid: illumos-x64
  Setting up directories for build
  Checking prerequisites...
  Commencing build of "install" target in "CoreCLR component" for illumos.x64.Debug in /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug
  Invoking "/g/ws/dotnet/runtime/eng/native/gen-buildsys.sh" "/g/ws/dotnet/runtime/src/coreclr" "/g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug" x64 illumos gcc Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet 
  Not searching for unused variables given on the command line.
  loading initial cache file /g/ws/dotnet/runtime/eng/native/tryrun.cmake
  -- The linker identification is GNU ld (GNU Binutils) 2.33.1
  Copyright (C) 2019 Free Software Foundation, Inc.
  This program is free software; you may redistribute it under the terms of
  the GNU General Public License version 3 or (at your option) a later version.
  This program has absolutely no warranty.
  Detected SunOS amd64
  -- Configuring done (3.1s)
  -- Generating done (20.6s)
  -- Build files have been written to: /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug
  /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug /g/ws/dotnet/runtime/src/coreclr
  Executing make install -j 4
  [  0%] Built target singlefilehost_exports
  make[2]: Warning: File 'interop/CMakeFiles/interop.dir/compiler_depend.make' has modification time 0.0038 s in the future
  make[2]: Warning: File 'Corehost.Static/hostmisc/CMakeFiles/hostmisc.dir/compiler_depend.make' has modification time 0.0063 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  0%] Built target hostmisc
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  0%] Built target interop
  make[2]: Warning: File 'libs-native/System.Net.Security.Native/CMakeFiles/System.Net.Security.Native-Static.dir/compiler_depend.make' has modification time 0.0028 s in the future
  make[2]: Warning: File 'libs-native/System.Globalization.Native/CMakeFiles/System.Globalization.Native-Static.dir/compiler_depend.make' has modification time 0.0065 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  0%] Built target System.Net.Security.Native-Static
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  0%] Built target System.Globalization.Native-Static
  [  1%] Built target System.Native-Static
  [  2%] Built target System.IO.Compression.Native-Static
  make[2]: Warning: File 'pal/src/CMakeFiles/coreclrpal_dac.dir/compiler_depend.make' has modification time 0.002 s in the future
  make[2]: Warning: File 'libs-native/System.Security.Cryptography.Native/CMakeFiles/objlib.dir/compiler_depend.make' has modification time 0.0004 s in the future
  make[2]: Warning: File 'pal/src/eventprovider/dummyprovider/CMakeFiles/eventprovider.dir/compiler_depend.make' has modification time 0.0062 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  2%] Built target coreclrpal_dac
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  3%] Built target objlib
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  4%] Built target eventprovider
  make[2]: Warning: File 'minipal/Unix/CMakeFiles/coreclrminipal.dir/compiler_depend.make' has modification time 0.00045 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  4%] Built target coreclrminipal
  make[2]: Warning: File 'containers/CMakeFiles/dn-containers.dir/compiler_depend.make' has modification time 0.00078 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  5%] Built target dn-containers
  [  5%] Built target gc_pal
  make[2]: Warning: File 'debug/dbgutil/CMakeFiles/dbgutil.dir/compiler_depend.make' has modification time 0.0015 s in the future
  make[2]: Warning: File 'nativeresources/CMakeFiles/nativeresourcestring.dir/compiler_depend.make' has modification time 0.0033 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  5%] Built target dbgutil
  [  5%] Built target eventing_headers
  [  5%] Built target gc_vxsort
  make[2]: Warning: File 'inc/CMakeFiles/corguids.dir/compiler_depend.make' has modification time 0.0004 s in the future
  make[2]: Warning: File 'palrt/CMakeFiles/palrt.dir/compiler_depend.make' has modification time 0.0016 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  5%] Built target nativeresourcestring
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  5%] Built target corguids
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [  5%] Built target palrt
  [  5%] Built target precompiled_asm
  make[2]: Warning: File 'vm/eventing/eventpipe/CMakeFiles/eventpipe_objs.dir/compiler_depend.make' has modification time 0.0031 s in the future
  [  9%] Built target libunwind
  [  9%] Built target gcinfo
  make[2]: Warning: File 'vm/eventing/eventpipe/CMakeFiles/eventpipe_gen_objs.dir/compiler_depend.make' has modification time 0.0017 s in the future
  make[2]: Warning: File 'md/runtime/CMakeFiles/mdruntime_wks.dir/compiler_depend.make' has modification time 0.0073 s in the future
  make[2]: Warning: File 'md/compiler/CMakeFiles/mdcompiler_wks.dir/compiler_depend.make' has modification time 0.00054 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 10%] Built target mdruntime_wks
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 11%] Built target mdcompiler_wks
  make[2]: Warning: File 'md/enc/CMakeFiles/mdruntimerw_wks.dir/compiler_depend.make' has modification time 0.0029 s in the future
  make[2]: Warning: File 'md/ceefilegen/CMakeFiles/ceefgen.dir/compiler_depend.make' has modification time 0.0062 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 11%] Built target ceefgen
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 11%] Built target mdruntimerw_wks
  make[2]: Warning: File 'vm/wks/CMakeFiles/cee_wks.dir/compiler_depend.make' has modification time 0.0015 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 11%] Built target eventpipe_gen_objs
  [ 11%] Built target eventpipe_objs
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 11%] Built target cee_wks
  [ 12%] Built target cordbee_wks
  [ 12%] Built target coreclr_exports
  [ 12%] Built target coreclr_def
  [ 12%] Built target comfloat_wks
  make[2]: Warning: File 'dlls/mscorrc/CMakeFiles/mscorrc.dir/compiler_depend.make' has modification time 0.01 s in the future
  [ 12%] Built target v3binder
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 12%] Built target bcltype
  [ 12%] Built target mscorrc
  make[2]: Warning: File 'unwinder/CMakeFiles/unwinder_wks.dir/compiler_depend.make' has modification time 0.0009 s in the future
  make[2]: Warning: File 'unwinder/CMakeFiles/unwinder_dac.dir/compiler_depend.make' has modification time 0.0015 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 14%] Built target unwinder_wks
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 14%] Built target unwinder_dac
  make[2]: Warning: File 'md/runtime/CMakeFiles/mdruntime_dac.dir/compiler_depend.make' has modification time 0.0019 s in the future
  make[2]: Warning: File 'md/enc/CMakeFiles/mdruntimerw_dac.dir/compiler_depend.make' has modification time 0.00023 s in the future
  [ 14%] Built target mdcompiler_dac
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 15%] Built target mdruntime_dac
  [ 15%] Built target mdruntimerw_dac
  [ 15%] Built target mscordaccore_exports
  make[2]: Warning: File 'dlls/mscordac/CMakeFiles/mscordacobj.dir/compiler_depend.make' has modification time 0.0088 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 16%] Built target mscordacobj
  [ 16%] Built target cordbee_dac
  [ 17%] Built target mdcompiler-dbi
  make[2]: Warning: File 'md/runtime/CMakeFiles/mdruntime-dbi.dir/compiler_depend.make' has modification time 0.0022 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 17%] Built target mdruntime-dbi
  make[2]: Warning: File 'md/enc/CMakeFiles/mdruntimerw-dbi.dir/compiler_depend.make' has modification time 0.0018 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 17%] Built target mdruntimerw-dbi
  make[2]: Warning: File 'md/datasource/CMakeFiles/mddatasource_dbi.dir/compiler_depend.make' has modification time 0.0021 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 17%] Built target mddatasource_dbi
  [ 17%] Built target mscordbi_exports
  make[2]: Warning: File 'libs-native/System.IO.Ports.Native/CMakeFiles/System.IO.Ports.Native-Static.dir/compiler_depend.make' has modification time 0.0057 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 20%] Built target utilcodestaticnohost
  [ 20%] Built target System.IO.Ports.Native-Static
  make[2]: Warning: File 'test-watchdog/CMakeFiles/watchdog.dir/compiler_depend.make' has modification time 0.00048 s in the future
  make[2]: Warning: File 'hosts/corerun/CMakeFiles/corerun.dir/compiler_depend.make' has modification time 0.0016 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 21%] Built target watchdog
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 21%] Built target corerun
  make[2]: Warning: File 'tools/aot/jitinterface/CMakeFiles/jitinterface_x64.dir/compiler_depend.make' has modification time 0.003 s in the future
  make[2]: Warning: File 'containers/CMakeFiles/dn-containers-no-lto.dir/compiler_depend.make' has modification time 0.0024 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 21%] Built target dn-containers-no-lto
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 22%] Built target jitinterface_x64
  make[2]: Warning: File 'nativeaot/Bootstrap/base/CMakeFiles/bootstrapper.dir/compiler_depend.make' has modification time 0.0059 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 22%] Built target bootstrapper
  [ 23%] Built target cordbdi
  make[2]: Warning: File 'nativeaot/Bootstrap/base/CMakeFiles/stdc++compat.dir/compiler_depend.make' has modification time 0.0027 s in the future
  [ 23%] Built target aot_eventing_headers
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 23%] Built target stdc++compat
  make[2]: Warning: File 'nativeaot/Bootstrap/dll/CMakeFiles/bootstrapperdll.dir/compiler_depend.make' has modification time 0.0083 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 23%] Built target bootstrapperdll
  [ 23%] Built target RuntimeAsmHelpers
  make[2]: Warning: File 'nativeaot/Runtime/Full/CMakeFiles/standalonegc-disabled.dir/compiler_depend.make' has modification time 0.0059 s in the future
  make[2]: Warning: File 'nativeaot/Runtime/Full/CMakeFiles/standalonegc-enabled.dir/compiler_depend.make' has modification time 0.0079 s in the future
  [ 28%] Built target cee_dac
  make[2]: Warning: File 'nativeaot/Runtime/Full/CMakeFiles/Runtime.VxsortDisabled.dir/compiler_depend.make' has modification time 0.0095 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 28%] Built target Runtime.VxsortDisabled
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 28%] Built target standalonegc-disabled
  [ 28%] Built target standalonegc-enabled
  [ 29%] Built target Runtime.VxsortEnabled
  make[2]: Warning: File 'nativeaot/Runtime/eventpipe/CMakeFiles/eventpipe-shared-objects.dir/compiler_depend.make' has modification time 0.005 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 29%] Built target eventpipe-shared-objects
  make[2]: Warning: File 'nativeaot/Runtime/eventpipe/CMakeFiles/eventpipe-disabled.dir/compiler_depend.make' has modification time 0.005 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 30%] Built target eventpipe-disabled
  make[2]: Warning: File 'md/runtime/CMakeFiles/mdruntime_ppdb.dir/compiler_depend.make' has modification time 0.005 s in the future
  make[2]: Warning: File 'md/compiler/CMakeFiles/mdcompiler_ppdb.dir/compiler_depend.make' has modification time 0.0044 s in the future
  [ 31%] Built target clrgcexp
  [ 31%] Built target clrgc
  make[2]: Warning: File 'md/staticmd/CMakeFiles/mdstaticapi_ppdb.dir/compiler_depend.make' has modification time 0.0055 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 31%] Built target mdstaticapi_ppdb
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 31%] Built target mdruntime_ppdb
  make[2]: Warning: File 'md/enc/CMakeFiles/mdruntimerw_ppdb.dir/compiler_depend.make' has modification time 0.0032 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 32%] Built target mdcompiler_ppdb
  make[2]: Warning: File 'dlls/mscorpe/CMakeFiles/mscorpe.dir/compiler_depend.make' has modification time 0.0018 s in the future
  make[2]: Warning: File 'md/staticmd/CMakeFiles/mdstaticapi.dir/compiler_depend.make' has modification time 0.0018 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 32%] Built target mdstaticapi
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 34%] Built target mdruntimerw_ppdb
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 34%] Built target mscorpe
  make[2]: Warning: File 'gcinfo/CMakeFiles/gcinfo_universal_arm64.dir/compiler_depend.make' has modification time 0.00075 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 34%] Built target gcinfo_universal_arm64
  make[2]: Warning: File 'gcinfo/CMakeFiles/gcinfo_unix_x64.dir/compiler_depend.make' has modification time 0.0023 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 34%] Built target gcinfo_unix_x64
  [ 34%] Built target gcinfo_win_x64
  [ 35%] Built target gcinfo_universal_arm
  [ 35%] Built target jit_exports
  [ 35%] Built target spmi_exports
  [ 35%] Built target gcinfo_win_x86
  [ 35%] Built target debug-pal
  [ 36%] Built target System.Security.Cryptography.Native.OpenSsl-Static
  make[2]: Warning: File 'vm/wks/CMakeFiles/cee_wks_mergeable.dir/compiler_depend.make' has modification time 0.0042 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 36%] Built target cee_wks_mergeable
  [ 40%] Built target coreclrpal
  [ 42%] Built target Runtime.WorkstationGC
  [ 44%] Built target Runtime.ServerGC
  make[2]: Warning: File 'ildasm/exe/CMakeFiles/ildasm.dir/compiler_depend.make' has modification time 0.0022 s in the future
  [ 44%] Built target eventpipe-enabled
  [ 45%] Built target ilasm
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 45%] Built target ildasm
  [ 55%] Built target cee_wks_core
  [ 65%] Built target clrjit
  [ 67%] Built target clrjit_universal_arm64_x64
  [ 69%] Built target clrjit_unix_x64_x64
  [ 74%] Built target clrjit_win_x64_x64
  [ 75%] Built target superpmi
  [ 76%] Built target mcs
  [ 77%] Built target superpmi-shim-collector
  [ 78%] Built target superpmi-shim-counter
  [ 78%] Built target superpmi-shim-simple
  [ 83%] Built target clrjit_universal_arm_x64
  [ 84%] Built target StressLogAnalyzer
  [ 88%] Built target clrjit_win_x86_x64
  make[2]: Warning: File 'debug/runtimeinfo/CMakeFiles/cdac_data_descriptor.dir/compiler_depend.make' has modification time 0.00043 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 88%] Built target cdac_data_descriptor
  [ 89%] Built target utilcode
  [ 89%] Built target cdac_contract_descriptor
  make[2]: Warning: File 'dlls/mscoree/coreclr/CMakeFiles/coreclr.dir/compiler_depend.make' has modification time 0.00094 s in the future
  [ 91%] Built target utilcode_dac
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 91%] Built target coreclr
  [ 91%] Built target dactablerva_header
  [ 96%] Built target clrjit_static
  make[2]: Warning: File 'dlls/mscoree/coreclr/CMakeFiles/coreclr_static.dir/compiler_depend.make' has modification time 0.00047 s in the future
  make[2]: Warning: File 'debug/daccess/CMakeFiles/daccess.dir/compiler_depend.make' has modification time 0.0001 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 96%] Built target coreclr_static
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 97%] Built target daccess
  make[2]: Warning: File 'dlls/mscordac/CMakeFiles/mscordaccore.dir/compiler_depend.make' has modification time 0.01 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 97%] Built target mscordaccore
  make[2]: Warning: File 'dlls/mscordbi/CMakeFiles/mscordbi.dir/compiler_depend.make' has modification time 0.0031 s in the future
  make[2]: warning:  Clock skew detected.  Your build may be incomplete.
  [ 97%] Built target mscordbi
  [ 97%] Built target runtimeinfo
  [100%] Built target singlefilehost
  Install the project...
  -- Install configuration: "DEBUG"
  /g/ws/dotnet/runtime/src/coreclr
  Repo successfully built.
  Product binaries are available at /g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug
  cdac-build-tool -> /g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll
  Executing "/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -x64 -debug gcc -os illumos -hostarch x64 -hostos linux -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll" -component crosscomponents
  Commencing CoreCLR Repo build
  __OutputRid: illumos-x64
  Setting up directories for build
  Checking prerequisites...
  Commencing build of "  crosscomponents " target in "CoreCLR component" for illumos.x64.Debug in /g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/x64
  Invoking "/g/ws/dotnet/runtime/eng/native/gen-buildsys.sh" "/g/ws/dotnet/runtime/src/coreclr" "/g/ws/dotnet/runtime/artifacts/obj/coreclr/illumos.x64.Debug/x64" x64 linux gcc Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet -DCLR_CROSS_COMPONENTS_BUILD=1 
  -- The linker identification is GNU ld (GNU Binutils) 2.33.1
  Copyright (C) 2019 Free Software Foundation, Inc.
  Not searching for unused variables given on the command line.
  This program is free software; you may redistribute it under the terms of
  the GNU General Public License version 3 or (at your option) a later version.
  This program has absolutely no warranty.
  Detected Linux x86_64
  -- Performing Test HAVE_LTTNG_TRACEPOINT_H
  -- Performing Test HAVE_LTTNG_TRACEPOINT_H - Failed
  CMake Error at pal/src/configure.cmake:921 (message):
    Cannot find liblttng-ust-dev.  Try installing liblttng-ust-dev (or the
    appropriate packages for your platform)
  Call Stack (most recent call first):
    pal/src/CMakeLists.txt:26 (include)

  -- Configuring incomplete, errors occurred!
  Failed to generate "CoreCLR component" build project!
/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -x64 -debug gcc -os illumos -hostarch x64 -hostos linux -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll" -component crosscomponents" exited with code 1.

Build FAILED.

/g/ws/dotnet/runtime/src/coreclr/runtime.proj(108,5): error MSB3073: The command ""/g/ws/dotnet/runtime/src/coreclr/build-runtime.sh" -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -x64 -debug gcc -os illumos -hostarch x64 -hostos linux -outputrid illumos-x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/g/ws/dotnet/runtime/.dotnet/dotnet" -cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=/g/ws/dotnet/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll" -component crosscomponents" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:02:16.54
Build failed with exit code 1. Check errors above.

Looking at src/coreclr/pal/src/config/configure.cmake line 911 it seems the build skipped the CLR_CMAKE_TARGET_SUNOS section. I tried making it take that path, but things didn't look better. Suggestions?

Looks like sudo eng/install-native-dependencies.sh didn't succeed on your machine.

it seems the build skipped the CLR_CMAKE_TARGET_SUNOS section. I tried making it take that path, but things didn't look better.

This is by design. The coreclr build succeeded and this next one is cross-components for host OS (Linux) so it should be taking Linux path where we expect lttng-ust to be present.

As far as I can tell, lttng-ust is there.

On Fri, Jun 28, 2024 at 3:23 AM Adeel Mujahid @.***> wrote:

it seems the build skipped the CLR_CMAKE_TARGET_SUNOS section. I tried making it take that path, but things didn't look better.

This is by design. The coreclr build succeeded and this next one is cross-components for host OS (Linux) so it should be taking Linux path where we expect lttng-ust to be present.

— Reply to this email directly, view it on GitHub https://github.com/dotnet/runtime/issues/34944#issuecomment-2196302615, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG33HHQK3W35JNUAS52OLTZJUFO3AVCNFSM6AAAAABJ4PTVTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJWGMYDENRRGU . You are receiving this because you were mentioned.Message ID: @.***>

Could you check find artifacts/obj -name CMakeError.log why it's failing?

Seems to be working in docker: https://github.com/am11/CrossRepoCITesting/actions/runs/9702500203/job/26778504623

  Checking prerequisites...
  Commencing build of "  crosscomponents " target in "CoreCLR component" for illumos.x64.Debug in /runtime/artifacts/obj/coreclr/illumos.x64.Debug/x64
  Invoking "/runtime/eng/native/gen-buildsys.sh" "/runtime/src/coreclr" "/runtime/artifacts/obj/coreclr/illumos.x64.Debug/x64" x64 linux gcc Debug ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="illumos" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCDAC_BUILD_TOOL_BINARY_PATH=/runtime/artifacts/bin/coreclr/illumos.x64.Debug/cdac-build-tool/cdac-build-tool.dll -DCLR_DOTNET_HOST_PATH=/runtime/.dotnet/dotnet -DCLR_CROSS_COMPONENTS_BUILD=1 
  Not searching for unused variables given on the command line.
  -- The C compiler identification is GNU 7.5.0
  -- The CXX compiler identification is GNU 7.5.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/gcc-7 - skipped
  -- Detecting C compile features
...

  -- Performing Test HAVE_MAP_HUGETLB - Success
  -- Performing Test HAVE_LTTNG_TRACEPOINT_H
  -- Performing Test HAVE_LTTNG_TRACEPOINT_H - Success
  -- Looking for sysctlbyname
  -- Looking for sysctlbyname - not found
...

Strange, find does not show any CMakeError.log files under artifacts. I'm on IRC -- maybe you can help me through this faster there? Thanks.

They have changed the format in recent version; instead of CMakeError.log we now get CMakeConfigureLog.yaml. Lets continue to IRC.

I tried building a rootfs with this command:

sudo ./eng/common/cross/build-rootfs.sh x64 illumos

I had to install some extra packages to get GCC to build on Ubuntu 22.04. These are not currently included in the eng/install-native-dependencies.sh script. Should they be added there or perhaps in cross-building.md?

sudo apt install libgmp3-dev libmpfr-dev libmpc-dev

Edit: I see the issue of pkgsrc and illumos sysroot being mis-matched causing a linker error like "undefined reference to __stack_chk_fail@ILLUMOS_0.37" is already being talked about at https://github.com/illumos/sysroot/issues/3

Longer description of the linking problem.

I then attempted to build with: ```bash ./build.sh clr+libs+packs -c Debug -cross -os illumos -gcc ``` I got these linker errors: > .tools/rootfs/x64/x86_64-sun-solaris2.10/bin/ld: .tools/rootfs/x64/lib/libz.so: undefined reference to '__stack_chk_fail@ILLUMOS_0.37' > .tools/rootfs/x64/x86_64-sun-solaris2.10/bin/ld: .tools/rootfs/x64/lib/libz.so: undefined reference to '__stack_chk_guard@ILLUMOS_0.37' I suspect this is caused by the fact the the `build-rootfs.sh` script downloads an illumos sysroot from 2018 while the zlib version is downloaded from the latest pkgsrc. pkgsrc for illumos [is building](https://pkgsrc.smartos.org/install-on-illumos/) on a newer version of illumos (20210826) that includes the `-fstack-protector` support added in 2020 in https://github.com/illumos/illumos-gate/commit/6a817834d81cc75ce12d0d393320837b1fec1e85 .

Building using the Docker method does work though:

docker run --rm -v$(pwd):/runtime -e ROOTFS_DIR=/crossrootfs/x64 \
    mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-illumos-20220531132048-f13d79e \
     /runtime/build.sh -c Release -os illumos -cross -gcc

Glad to hear you are trying it out @AustinWise! The cross-compilation is only required until we get some of the shared framework libraries ported for SDK, after that we can build everything on the illumos device / VM. The best way is to copy crossrootfs dir from the docker into a linux host (e.g. at /crossrootfs) and set ROOTFS_DIR=/crossrootfs/x64. Then in vscode, we can do the library port work. System.Diagnostics.Process and System.Net are two libs needed for SDK/MSBuild to function on illumos.

BTW, we ported coreclr on SmartOS VM itself by running coreclr-specific build script (src/coreclr/build-runtime.sh), after that I've used this docker to port CoreLib / System.Runtime etc.

N.B. the manpages are a bit dated in some areas, so I was reading https://github.com/illumos/illumos-gate as a source of truth. The community #illumos @libera.chat helped a lot.

I had done this earlier, and forgot about it:

If you ran eng/install-native-dependencies.sh, it should had installed some version of gcc (via build-essential)? [...] Nonetheless, it is a bug in our init-compiler script that we don't bypass for platforms where rootfs provides a separate toolchain for host machine to emit target code (illumos and android NDK). For android, we have a hack __Compiler="default", but for illumos lets just go with traditional environment variable magic:

export CLR_CC=/crossrootfs/x64/bin/x86_64-illumos-gcc export CLR_CXX=/crossrootfs/x64/bin/x86_64-illumos-g++

OK, I figured this out. We previously set CLR_CC and CLR_CXX in the shell I've been running in (to the /crossrootfs tools) That probably breaks native compile. I had to remove the artifacts and rebuild, then it worked.

Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 01:47:24.23

Yea! After removing the errant environment settings and deleting artifacts, the build worked. On to the next phase...

Thanks for the help!

Good news! That explains and sorry I didn't realized earlier that those environment variables override for the whole build, including cross-components.

To run a simple app on illumos, first create an app on linux host:

# assuming pwd is runtime
$ export PATH=$PATH:$(pwd)/.dotnet
$ cd ..
$ dotnet new console -n helloworld
$ cd helloworld
$ dotnet build

Then:

copy ./bin/Debug/net8.0/ and ../runtime/artifacts/packages/Debug/Shipping/dotnet-runtime-9.0.0-dev-illumos-x64.tar.gz to illumos machine
extract the tar.gz, say at ~/.dotnet
export PATH=$PATH:~/.dotnet
and run dotnet bin/Debug/net8.0/helloworld.dll

For libraries port work, we can modify the code on linux, rebuild just that specific library, e.g.

$ ./dotnet.sh build -p:TargetOS=illumos src/libraries/System.Diagnostics.Process/src

it shows lines like this:


  Determining projects to restore...
  All projects are up-to-date for restore.
  ILLink.RoslynAnalyzer -> /Users/adeel/projects/runtime4/artifacts/bin/ILLink.RoslynAnalyzer/Debug/netstandard2.0/ILLink.RoslynAnalyzer.dll
  ILLink.CodeFixProvider -> /Users/adeel/projects/runtime4/artifacts/bin/ILLink.CodeFixProvider/Debug/netstandard2.0/ILLink.CodeFixProvider.dll
  ILCompiler.DependencyAnalysisFramework -> /Users/adeel/projects/runtime4/artifacts/bin/ILCompiler.DependencyAnalysisFramework/Debug/ILCompiler.DependencyAnalysisFramework.dll
  Mono.Linker -> /Users/adeel/projects/runtime4/artifacts/bin/Mono.Linker/ref/Debug/net9.0/illink.dll
  Mono.Linker -> /Users/adeel/projects/runtime4/artifacts/bin/Mono.Linker/Debug/net9.0/illink.dll
  ILLink.Tasks -> /Users/adeel/projects/runtime4/artifacts/bin/ILLink.Tasks/Debug/net9.0/ILLink.Tasks.dll
  ILLink.RoslynAnalyzer -> /Users/adeel/projects/runtime4/artifacts/bin/ILLink.RoslynAnalyzer/Debug/netstandard2.0/ILLink.RoslynAnalyzer.dll
  Microsoft.Interop.SourceGeneration -> /Users/adeel/projects/runtime4/artifacts/bin/Microsoft.Interop.SourceGeneration/Debug/netstandard2.0/Microsoft.Interop.SourceGeneration.dll
  System.Runtime -> /Users/adeel/projects/runtime4/artifacts/bin/System.Runtime/ref/Debug/net9.0/System.Runtime.dll
  System.Diagnostics.FileVersionInfo -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.FileVersionInfo/ref/Debug/net9.0/System.Diagnostics.FileVersionInfo.dll
  System.Collections.NonGeneric -> /Users/adeel/projects/runtime4/artifacts/bin/System.Collections.NonGeneric/ref/Debug/net9.0/System.Collections.NonGeneric.dll
  System.Collections -> /Users/adeel/projects/runtime4/artifacts/bin/System.Collections/ref/Debug/net9.0/System.Collections.dll
  LibraryImportGenerator -> /Users/adeel/projects/runtime4/artifacts/bin/LibraryImportGenerator/Debug/netstandard2.0/Microsoft.Interop.LibraryImportGenerator.dll
  System.Runtime.InteropServices -> /Users/adeel/projects/runtime4/artifacts/bin/System.Runtime.InteropServices/ref/Debug/net9.0/System.Runtime.InteropServices.dll
  ComInterfaceGenerator -> /Users/adeel/projects/runtime4/artifacts/bin/ComInterfaceGenerator/Debug/netstandard2.0/Microsoft.Interop.ComInterfaceGenerator.dll
  System.ComponentModel -> /Users/adeel/projects/runtime4/artifacts/bin/System.ComponentModel/ref/Debug/net9.0/System.ComponentModel.dll
  System.ObjectModel -> /Users/adeel/projects/runtime4/artifacts/bin/System.ObjectModel/ref/Debug/net9.0/System.ObjectModel.dll
  System.ComponentModel.Primitives -> /Users/adeel/projects/runtime4/artifacts/bin/System.ComponentModel.Primitives/ref/Debug/net9.0/System.ComponentModel.Primitives.dll
  System.Collections.Specialized -> /Users/adeel/projects/runtime4/artifacts/bin/System.Collections.Specialized/ref/Debug/net9.0/System.Collections.Specialized.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/ref/Debug/net9.0/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-osx/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-maccatalyst/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-tvos/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-ios/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-linux/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-freebsd/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0-windows/System.Diagnostics.Process.dll
  System.Diagnostics.Process -> /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0/System.Diagnostics.Process.dll

so when we will have illumos flavor we can just copy its dll artifacts/bin/System.Diagnostics.Process/Debug/net9.0-illumos/System.Diagnostics.Process.dll and maybe also the arch-agnostic one as well /Users/adeel/projects/runtime4/artifacts/bin/System.Diagnostics.Process/Debug/net9.0/System.Diagnostics.Process.dll (depends on if the change was made in common files or only the arch-specific ones), to the illumos VM and replace them under ~/.dotnet directory.

@gwr, I read your updated gist. https://gist.githubusercontent.com/gwr/3520dfbf14190e9225e8214f434ca38e/raw/Debugging.txt. Congrats on the first helloworld run! :)

That 256 GB is coming from getrlimit here: https://github.com/dotnet/runtime/blob/9528c1544e180652162ed73ef1f3f71faca9c070/src/coreclr/gc/unix/gcenv.unix.cpp#L1174-L1207

Side note, by default W^X is enabled which we can disable by DOTNET_EnableWriteXorExecute=0 if that helps anything. It uses this impl on illumos: https://github.com/dotnet/runtime/blob/9528c1544e180652162ed73ef1f3f71faca9c070/src/coreclr/minipal/Unix/doublemapping.cpp#L51-L56

Regarding:

I also tried adding to helloworld.runtimeconfig.json:
   "configProperties": {
     "System.GC.HeapHardLimit": 33554432,
     ...
   }
but for some reason, that seems to be ignored.

lets try updating helloworld/Program.cs on linux:

Console.WriteLine("Hello, World!");

foreach ((string key, object value) in GC.GetConfigurationVariables())
{
    Console.WriteLine($"{key}: {value}");
}

then execute dotnet build, copy bin/Debug/net9.0/helloworld.dll directory to illumos VM and re-run with configProperties change. It should print GCHeapHardLimit:209715201 line with and without DOTNET_GCHeapHardLimit=1C00000.

I'd like to deal next with the privilege proc_lock_memory which we can do by adding an attribute to the executable. However, I'll need to add a small bit of OS-specific code somewhere after main() begins but before the mlock() call in PAL_InitializeCoreCLR(). Can you suggest a good place to add that OS-specific code? Thanks.

We can wrap it in PAL_InitializeCoreCLR(), because that's where the first mlock() call being made.

I've fixed available memory determination for illumos. See: https://github.com/dotnet/runtime/compare/main...gwr:dotnet-runtime:illumos2

Today, it ends up using: https://github.com/dotnet/runtime/blob/9a4329d65c4d272e26b2c693341ef4b26a5bc8c8/src/coreclr/gc/unix/gcenv.unix.cpp#L1276

and SYSCONF_PAGES is defined as _SC_AVPHYS_PAGES, is it different than available = sysconf(_SC_AVPHYS_PAGES) * getpagesize()?

I misread that #define SYSCONF_PAGES, thinking it was APPLE specific (hm, ifndef...) I'm not sure why the existing code didn't use _SC_AVPHYS_PAGES on illumos. But anyway, why not use that, where available, instead of reading /proc/whatever? The sysconf call should be much lower cost.

I have opened https://github.com/dotnet/runtime/issues/104211 to discuss it further with a suggestion to use kstat (if applicable). This is a general tracking issue for the port so it was getting a bit off-topic. Lets open more granular issues for targeted discussions so area owners and interested folks can chime in and easily follow the details.

When running a hellow world, I get a segfault like this:

WKS::in_range_for_segment (seg=0xfffd940f026f72d0, add=0xfffffc7f5bc00028 "") at src/coreclr/gc/gc.cpp:3480
WKS::seg_mapping_table_segment_of (o=0xfffffc7f5bc00028 "") at src/coreclr/gc/gc.cpp:4585
0xfffffc7fe3d1a51b in WKS::gc_heap::find_segment (interior=0xfffffc7f5bc00028 "", small_segment_only_p=0) at src/coreclr/gc/gc.cpp:26331
0xfffffc7fe3d5adf9 in WKS::GCHeap::IsHeapPointer (this=0x7132f0, vpObject=0xfffffc7f5bc00028, small_heap_only=false) at src/coreclr/gc/gc.cpp:49008
0xfffffc7fe3d5c095 in WKS::GCHeap::Alloc (this=0x7132f0, context=0xfffffc7fef121a58, size=8184, flags=66) at src/coreclr/gc/gc.cpp:49515

Full stack trace in case that's interesting

#0  WKS::in_range_for_segment (seg=0xfffd940f026f72d0, add=0xfffffc7f5bc00028 "") at /tank/externsrc/dotnet/runtime/src/coreclr/gc/gc.cpp:3480
#1  WKS::seg_mapping_table_segment_of (o=0xfffffc7f5bc00028 "") at /tank/externsrc/dotnet/runtime/src/coreclr/gc/gc.cpp:4585
#2  0xfffffc7fe3d1a51b in WKS::gc_heap::find_segment (interior=0xfffffc7f5bc00028 "", small_segment_only_p=0) at /tank/externsrc/dotnet/runtime/src/coreclr/gc/gc.cpp:26331
#3  0xfffffc7fe3d5adf9 in WKS::GCHeap::IsHeapPointer (this=0x7132f0, vpObject=0xfffffc7f5bc00028, small_heap_only=false) at /tank/externsrc/dotnet/runtime/src/coreclr/gc/gc.cpp:49008
#4  0xfffffc7fe3d5c095 in WKS::GCHeap::Alloc (this=0x7132f0, context=0xfffffc7fef121a58, size=8184, flags=66) at /tank/externsrc/dotnet/runtime/src/coreclr/gc/gc.cpp:49515
#5  0xfffffc7fe3b03b60 in Alloc (size=8184, flags=66) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/gchelpers.cpp:227
#6  0xfffffc7fe3b03e4c in AllocateSzArray (pArrayMT=0xfffffc7f6495b160, cElements=1020, flags=66) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/gchelpers.cpp:422
#7  0xfffffc7fe3b03c82 in AllocateSzArray (arrayType=..., cElements=1020, flags=GC_ALLOC_PINNED_OBJECT_HEAP) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/gchelpers.cpp:367
#8  0xfffffc7fe3b04fa4 in AllocateObjectArray (cElements=1020, elementType=..., bAllocateInPinnedHeap=1) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/gchelpers.cpp:897
#9  0xfffffc7fe3890324 in PinnedHeapHandleTable::AllocateHandles (this=0x826500, nRequested=1) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/appdomain.cpp:347
#10 0xfffffc7fe3891230 in BaseDomain::AllocateObjRefPtrsInLargeTable (this=0xfffffc7fe48adaa0 , nRequested=1, pStaticsInfo=0xfffffc7f64aeccf0, pMTToFillWithStaticBoxes=0x0) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/appdomain.cpp:700
#11 0xfffffc7fe39a487a in LoaderAllocator::AllocateGCHandlesBytesForStaticVariables (this=0xfffffc7fe48ae268 , pStaticsInfo=0xfffffc7f64aeccf0, cSlots=1, pMTToFillWithStaticBoxes=0x0) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/loaderallocator.cpp:2385
#12 0xfffffc7fe39c6a1c in MethodTable::EnsureStaticDataAllocated (this=0xfffffc7f64aecc80) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/methodtable.cpp:3793
#13 0xfffffc7fe388eef1 in FieldDesc::GetCurrentStaticAddress (this=0xfffffc7f64aec598) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/field.h:518
#14 0xfffffc7fe38badd3 in CastCache::Initialize () at /tank/externsrc/dotnet/runtime/src/coreclr/vm/castcache.cpp:131
#15 0xfffffc7fe3892285 in SystemDomain::LoadBaseSystemClasses (this=0xfffffc7fe48adaa0 ) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/appdomain.cpp:1240
#16 0xfffffc7fe3891b82 in SystemDomain::Init (this=0xfffffc7fe48adaa0 ) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/appdomain.cpp:1028
#17 0xfffffc7fe3f5c4a2 in EEStartupHelper () at /tank/externsrc/dotnet/runtime/src/coreclr/vm/ceemain.cpp:926
#18 0xfffffc7fe3f5cae4 in ::operator()(PVOID) const (__closure=0xfffffc7fffdfebba, p=0x0) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/ceemain.cpp:1064
#19 0xfffffc7fe3f5cc3e in EEStartup () at /tank/externsrc/dotnet/runtime/src/coreclr/vm/ceemain.cpp:1066
#20 0xfffffc7fe3f5b903 in EnsureEEStarted () at /tank/externsrc/dotnet/runtime/src/coreclr/vm/ceemain.cpp:303
#21 0xfffffc7fe38ff205 in CorHost2::Start (this=0x75d050) at /tank/externsrc/dotnet/runtime/src/coreclr/vm/corhost.cpp:100
#22 0xfffffc7fe388873e in coreclr_initialize (exePath=0x759130 "/root/dotnet/dotnet", appDomainFriendlyName=0xfffffc7fe950f044 "clrhost", propertyCount=10, propertyKeys=0x713ee0,  propertyValues=0x714650, hostHandle=0xfffffc7fffdfee38, domainId=0xfffffc7fffdfee28) at /tank/externsrc/dotnet/runtime/src/coreclr/dlls/mscoree/exports.cpp:310

Notice in seg_mapping_table_segment_of that the object was allocated at a very high address (0xfffffc7f5bc00028). It appears that mmap on illumos will allocate above the non-cannoical area on x64. This does not match the assumption in GCToOSInterface::GetVirtualMemoryMaxAddress that adresses above 128TB will NOT be used.

Let's look at seg_mapping_table_segment_of:

heap_segment* seg_mapping_table_segment_of (uint8_t* o)
{
    size_t index = (size_t)o >> gc_heap::min_segment_size_shr;
    seg_mapping* entry = &seg_mapping_table[index];

It's shifting down the object address to index into seg_mapping_table, which looks like it is allocated as one chuck (i.e., it's not a space mapping of memory). I have not looked into how this table works further, but I could imagine that changing it from covering 128TB of address space to coving the whole 16EB of 64-address space might require rethinking it.

Since I'm a bit out of my depth with how the GC works, I have worked around this problem by limiting the GC address space to 4GB. That commit along with setting DOTNET_GCHeapHardLimit=1C00000 is enough to make a "hello world" program work!

OK, what's next on the way to self hosting the dotnet build? Also, I have some research findings to discuss re. FlushProcessWriteBuffers. Let's chat about that when we're both on IRC.

@AustinWise, please see https://github.com/dotnet/runtime/issues/104211, we have just discussed this topic over there.

OK, what's next on the way to self hosting the dotnet build?

SDK requires functioning shared framework on a platform. So System.Diagnostics.Process would be the next library to port. I think this is where procfs/kstat etc. will come handy. https://github.com/dotnet/runtime/tree/main/src/libraries/System.Diagnostics.Process

@am11 I saw the discussion about about the physical memory limits and how they fallback to GetVirtualMemoryMaxAddress. I was just pointing out that GetVirtualMemoryMaxAddress is used in other places and returns an incorrect value for illumos. illumos use a larger maximum user-mode virtual memory address (0xFFFFFC7F.FFE00000) than other operating systems that CoreCLR currently supports.

Anyways, thanks for posting how to get cross building working. I'm now unblocked and can play with this on my system.

Yeah, I forgot that I had GCHeapHardLimit set in helloworld.runtimeconfig.json and when I took that out this failure is back:

$ ./dotnet helloworld/bin/Debug/net9.0/helloworld.dll
GC heap initialization failed with error 0x8007000E
Failed to create CoreCLR, HRESULT: 0x8007000E

I'm looking into it. It fails when mmap asks for 256 GB of anon pages.

Thread 2 hit Breakpoint 2, VirtualReserveInner (size=274877906944, 
    alignment=8192, flags=0, hugePagesFlag=0, committing=false)
    at /g/ws/dotnet/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:576
576     assert(!(flags & VirtualReserveFlags::WriteWatch) && "WriteWatch not supported on Unix");
(gdb) where
#0  VirtualReserveInner (size=274877906944, alignment=8192, flags=0, 
    hugePagesFlag=0, committing=false)
    at /g/ws/dotnet/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:576
#1  0x00007fff909735b9 in GCToOSInterface::VirtualReserve (size=274877906944, 
    alignment=8192, flags=0, node=65535)
    at /g/ws/dotnet/runtime/src/coreclr/gc/unix/gcenv.unix.cpp:626
#2  0x00007fff906f9a00 in WKS::virtual_alloc (size=274877906944, 
    use_large_pages_p=false, numa_node=65535)
    at /g/ws/dotnet/runtime/src/coreclr/gc/gc.cpp:5680
#3  0x00007fff907068af in WKS::gc_heap::initialize_gc (
    soh_segment_size=268435456, loh_segment_size=0, poh_segment_size=0)
    at /g/ws/dotnet/runtime/src/coreclr/gc/gc.cpp:14177
#4  0x00007fff9075a349 in WKS::GCHeap::Initialize (this=0x713420)
    at /g/ws/dotnet/runtime/src/coreclr/gc/gc.cpp:48482
#5  0x00007fff9095c526 in EEStartupHelper ()
    at /g/ws/dotnet/runtime/src/coreclr/vm/ceemain.cpp:886
#6  0x00007fff9095cc24 in <lambda(PVOID)>::operator()(PVOID) const (
    __closure=0x7fffbfffeb4a, p=0x0)
    at /g/ws/dotnet/runtime/src/coreclr/vm/ceemain.cpp:1064
#7  0x00007fff9095cd7e in EEStartup ()
    at /g/ws/dotnet/runtime/src/coreclr/vm/ceemain.cpp:1066
#8  0x00007fff9095ba43 in EnsureEEStarted ()
    at /g/ws/dotnet/runtime/src/coreclr/vm/ceemain.cpp:303
--Type <RET> for more, q to quit, c to continue without paging--
#9  0x00007fff902ff345 in CorHost2::Start (this=0x75cdd0)
    at /g/ws/dotnet/runtime/src/coreclr/vm/corhost.cpp:100
#10 0x00007fff9028887e in coreclr_initialize (
    exePath=0x7593b0 "/tank/ws/dnt/dotnet", 
    appDomainFriendlyName=0x7fff9150efc4 "clrhost", propertyCount=10, 
    propertyKeys=0x7140f0, propertyValues=0x714380, hostHandle=0x7fffbfffedc8, 
    domainId=0x7fffbfffedb8)
    at /g/ws/dotnet/runtime/src/coreclr/dlls/mscoree/exports.cpp:310
#11 0x00007fff91418baa in coreclr_t::create (libcoreclr_path=..., 
    exe_path=0x7593b0 "/tank/ws/dnt/dotnet", 
    app_domain_friendly_name=0x7fff9150efc4 "clrhost", properties=..., 
    inst=...)
    at /g/ws/dotnet/runtime/src/native/corehost/hostpolicy/coreclr.cpp:72
#12 0x00007fff9143faf5 in (anonymous namespace)::create_coreclr ()
    at /g/ws/dotnet/runtime/src/native/corehost/hostpolicy/hostpolicy.cpp:75
#13 0x00007fff91441858 in corehost_main (argc=2, argv=0x7fffbffff6a8)
    at /g/ws/dotnet/runtime/src/native/corehost/hostpolicy/hostpolicy.cpp:422
#14 0x00007fff9182b322 in execute_app (impl_dll_dir=..., init=0x7148c0, 
    argc=2, argv=0x7fffbffff6a8)
    at /g/ws/dotnet/runtime/src/native/corehost/fxr/fx_muxer.cpp:145
#15 0x00007fff9182d0c3 in (anonymous namespace)::read_config_and_execute (
    host_command=..., host_info=..., app_candidate=..., opts=..., new_argc=2, 
    new_argv=0x7fffbffff6a8, mode=muxer, is_sdk_command=false, out_buffer=0x0, 
--Type <RET> for more, q to quit, c to continue without paging--
    buffer_size=0, required_buffer_size=0x0)
    at /g/ws/dotnet/runtime/src/native/corehost/fxr/fx_muxer.cpp:532
#16 0x00007fff9182ecf5 in fx_muxer_t::handle_exec_host_command (
    host_command=..., host_info=..., app_candidate=..., opts=..., argc=2, 
    argv=0x7fffbffff6a8, argoff=1, mode=muxer, is_sdk_command=false, 
    result_buffer=0x0, buffer_size=0, required_buffer_size=0x0)
    at /g/ws/dotnet/runtime/src/native/corehost/fxr/fx_muxer.cpp:1007
#17 0x00007fff9182d2d2 in fx_muxer_t::execute (host_command=..., argc=2, 
    argv=0x7fffbffff6a8, host_info=..., result_buffer=0x0, buffer_size=0, 
    required_buffer_size=0x0)
    at /g/ws/dotnet/runtime/src/native/corehost/fxr/fx_muxer.cpp:578
#18 0x00007fff91820d41 in hostfxr_main_startupinfo (argc=2, 
    argv=0x7fffbffff6a8, host_path=0x6fb060 "/tank/ws/dnt/dotnet", 
    dotnet_root=0x7fffbffff588 "/tank/ws/dnt/", 
    app_path=0x6fb090 "/tank/ws/dnt/dotnet.dll")
    at /g/ws/dotnet/runtime/src/native/corehost/fxr/hostfxr.cpp:63
#19 0x0000000000403e0f in exe_start (argc=2, argv=0x7fffbffff6a8)
    at /g/ws/dotnet/runtime/src/native/corehost/corehost.cpp:243
#20 0x0000000000404146 in main (argc=2, argv=0x7fffbffff6a8)
    at /g/ws/dotnet/runtime/src/native/corehost/corehost.cpp:311
(gdb)

@am11 I saw the discussion about about the physical memory limits and how they fallback to GetVirtualMemoryMaxAddress. I was just pointing out that GetVirtualMemoryMaxAddress is used in other places and returns an incorrect value for illumos. illumos use a larger maximum user-mode virtual memory address (0xFFFFFC7F.FFE00000) than other operating systems that CoreCLR currently supports.

Yeah, thanks. It's not clear to me what DotNet wants when it calls GetVirtualMemoryMaxAddress. If the plan is to map anon pages through that entire range, then that's the "wrong question to ask". :)

Here are the prereq workarounds to make the app working:

~~vmem echo "export DOTNET_GCHeapHardLimit=0x1C00000" >> ~/.profile~~ not needed after https://github.com/dotnet/runtime/pull/104275
proc_lock echo sudo usermod -K defaultpriv='basic,proc_lock_memory' $(whoami) >> ~/.profile (see https://github.com/dotnet/runtime/issues/36632)
source it . ~/.profile or re-login

Yes, we need to figure out better solution which doesn't require any manual steps from the user and the one which is not more involved than the existing workarounds.., but until then, we can continue on libraries port because there is a lot to cover there.

Can you tell me what GetVirtualMemoryMaxAddress returns on Linux? (and the memory stats for the machine that runs on)

$ uname -a
Linux 72098fe3cbae 6.6.31-linuxkit #1 SMP Thu May 23 08:36:57 UTC 2024 aarch64 GNU/Linux

$ clang++ -xc++ - -DHOST_64BIT <<EOF
#include <iostream>
#include <sys/resource.h>

class GCToOSInterface {
public:
    size_t GetVirtualMemoryLimit();
    size_t GetVirtualMemoryMaxAddress();
};

size_t GCToOSInterface::GetVirtualMemoryLimit() {
    rlimit addressSpaceLimit;
    if ((getrlimit(RLIMIT_AS, &addressSpaceLimit) == 0) && (addressSpaceLimit.rlim_cur != RLIM_INFINITY)) {
        return addressSpaceLimit.rlim_cur;
    }
    // No virtual memory limit
    return GetVirtualMemoryMaxAddress();
}

size_t GCToOSInterface::GetVirtualMemoryMaxAddress() {
#ifdef HOST_64BIT
#ifndef TARGET_RISCV64
    static const uint64_t _128TB = (1ull << 47);
    return _128TB;
#else // TARGET_RISCV64
    static const uint64_t _256GB = (1ull << 38);
    return _256GB;
#endif // TARGET_RISCV64
#else
    return (size_t)-1;
#endif
}

int main() {
    GCToOSInterface interface;
    std::cout << "Virtual Memory Limit: " << interface.GetVirtualMemoryLimit() << std::endl;
    std::cout << "Virtual Memory Max Address: " << interface.GetVirtualMemoryMaxAddress() << std::endl;
    return 0;
}
EOF

$ ./a.out
Virtual Memory Limit: 140737488355328
Virtual Memory Max Address: 140737488355328

$ ulimit -v
unlimited

$ cat /proc/meminfo
MemTotal:       15072344 kB
MemFree:         1836804 kB
MemAvailable:   14130380 kB
Buffers:          265036 kB
Cached:         11423340 kB
SwapCached:          284 kB
Active:          3671848 kB
Inactive:        8488140 kB
Active(anon):     249108 kB
Inactive(anon):   233048 kB
Active(file):    3422740 kB
Inactive(file):  8255092 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1572860 kB
SwapFree:        1572080 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                76 kB
Writeback:             0 kB
AnonPages:        468148 kB
Mapped:           137548 kB
Shmem:             10544 kB
KReclaimable:     861444 kB
Slab:             957316 kB
SReclaimable:     861444 kB
SUnreclaim:        95872 kB
KernelStack:        5808 kB
PageTables:         3540 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     9109032 kB
Committed_AS:    2251004 kB
VmallocTotal:   133141626880 kB
VmallocUsed:        2832 kB
VmallocChunk:          0 kB
Percpu:             3560 kB
AnonHugePages:    380928 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

I think we can disable the GC regions feature for now, it's disabled for macOS (linux and freebsd seem to be working fine). It is relatively a new feature and adjustments have been made a few times to make it work properly on linux as well (https://github.com/dotnet/runtime/commits/main/src/coreclr/gc/unix/gcenv.unix.cpp).

diff --git a/src/coreclr/gc/gcpriv.h b/src/coreclr/gc/gcpriv.h
index 1005d002029..8b3b3eb1870 100644
--- a/src/coreclr/gc/gcpriv.h
+++ b/src/coreclr/gc/gcpriv.h
@@ -141,7 +141,7 @@ inline void FATAL_GC_ERROR()
 // This means any empty regions can be freely used for any generation. For
 // Server GC we will balance regions between heaps.
 // For now disable regions for standalone GC and macOS builds
-#if defined (HOST_64BIT) && !defined (BUILD_AS_STANDALONE) && !defined(__APPLE__)
+#if defined (HOST_64BIT) && !defined (BUILD_AS_STANDALONE) && !defined(__APPLE__) && !defined(__sun)
 #define USE_REGIONS
 #endif //HOST_64BIT && BUILD_AS_STANDALONE && !__APPLE__

dotnet / runtime

Tracking issue for illumos and Solaris x86-64 port work #34944