[Alpine Linux] `s390x` platform support

[ayakael]

Description

Dotnet6 on Fedora supports s390x. Dotnet6 on Alpine should do so, too.

Parallel thread on Alpine Gitlab

Major roadblocks

As there are no dotnet6 prebuilt bootstraps release for s390x, we have to build our own by crosscompiling from x86_64 to s390x. Crosscompilation scripts was kindly provided by the Fedora team, and added to a git repo on Apine's Gitlab instance for further work.

Hypothetical steps

1) clone aports, and cd aports/community/dotnet6-stage0 2) abuild deps to install build dependencies of dotnet 3) apk add quilt mono dotnet6-sdk 4) git clone https://gitlab.alpinelinux.org/ayakael/dotnet6-s390x (original version here) 5) cd dotnet6-s390x && ./dotnet-s390x-prepare 6) ./dotnet-s390x-build

cc. @nekopsykose @uweigand @omajid

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ayakael]

Current issue: As is currently the case, crosscompilation of runtime fails due to clang13 not having elf64_s390 emulation. How to get this working? Having no experience of crosscompilation, I'd appreciate if anyone has pointers for this.

[ayakael]

I made another attempt. Considering no s390x support from lld, I set LD=s390x-linux-gnu-ldand tried making a binutils-cross package for s390x by modifying binutils-cross-embedded aport to target s390x-linux-gnu. New errros:

  Invoking "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/eng/native/gen-buildsys.sh" "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr" "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release" s390x clang "" "0" Release ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=1 -DFEATURE_DISTRO_AGNOSTIC_SSL=1 
  WARN: Specific version of clang not found, falling back to use the one in PATH.
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Not searching for unused variables given on the command line.
  loading initial cache file /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/eng/native/tryrun.cmake
  -- The C compiler identification is Clang 13.0.1
  -- The CXX compiler identification is Clang 13.0.1
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - failed
  -- Check for working C compiler: /usr/bin/clang
  -- Check for working C compiler: /usr/bin/clang - broken
  CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:69 (message):
    The C compiler

      "/usr/bin/clang"

    is not able to compile a simple test program.

    It fails with the following output:

      Change Dir: /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp

      Run Build Command(s):/usr/bin/make -f Makefile cmTC_30712/fast && /usr/bin/make  -f CMakeFiles/cmTC_30712.dir/build.make CMakeFiles/cmTC_30712.dir/build
      make[1]: Entering directory '/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp'
      Building C object CMakeFiles/cmTC_30712.dir/testCCompiler.c.o
      /usr/bin/clang --target=s390x-linux-gnu --gcc-toolchain=//usr --sysroot=/    -MD -MT CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_30712.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -c /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp/testCCompiler.c
      Linking C executable cmTC_30712
      /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_30712.dir/link.txt --verbose=1
      /usr/bin/clang --target=s390x-linux-gnu --gcc-toolchain=//usr --sysroot=/ -Wl,--rpath-link=//lib/s390x-linux-gnu -Wl,--rpath-link=//usr/lib/s390x-linux-gnu -Wl,--rpath-link=//lib/s390x-linux-gnu -Wl,--rpath-link=//usr/lib/s390x-linux-gnu  CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -o cmTC_30712 
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: error adding symbols: file in wrong format
      clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
      make[1]: *** [CMakeFiles/cmTC_30712.dir/build.make:100: cmTC_30712] Error 1
      make[1]: Leaving directory '/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp'
      make: *** [Makefile:127: cmTC_30712/fast] Error 2

    CMake will not be able to correctly generate this project.
  Call Stack (most recent call first):
    CMakeLists.txt:16 (project)

  -- Configuring incomplete, errors occurred!
  See also "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeOutput.log".
  See also "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeError.log".
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Executing make   iltools  -j 24
  make: *** No rule to make target 'iltools'.  Stop.
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Failed to build "CoreCLR component".

[omajid]

cc @BahaVv

[uweigand]

This seems to be a problem with the cross-compile environment.

Specifically:

      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)

EM 62 is EM_X86_64, which the s390x specific ld doesn't understand. This is probably a problem with the cross-compiler configuration, since the compiler apparently attempts to pull in the host version of crt1.o into a cross-link, instead of the appropriate target version.

crt1.o is part of the libc package, so the question is whether/where you've installed the target libc. In my original Ubuntu-based scripts, multiple architecture versions of libc (and other libraries) can be installed into the same root directory, because of Debian's multi-architecture file system layout. This doesn't work on most other distros, so there you usually have to use something like a sysroot environment.

I see in the repo you've linked above that in dotnet-s390x-build you have this line:

ROOTFS_DIR=/ ./build.sh -arch s390x -c ${RUNTIME_CONF} -cross -clang

ROOTFS_DIR=/ is correct on Ubuntu because of the multi-arch file system layout mentioned above. On other distros, this is nearly certain to be incorrect. Instead, you'll have to create a sysroot directory containing target libraries somewhere, and then point ROOTFS_DIR to that directory.

[ayakael]

That strikes me as making total sense. I'll explore this further and come back to you.

[ayakael]

Okay, finally found time to setup a crossbuild environement on Alpine. I've managed to get parts of runtime built, but musl is apparently utterly unhappy with dotnet's mono. Many errors such as function-like macro '__GLIBC_PREREQ' is not defined and variable 'is_enum' set but not used [-Wunused-but-set-variable]. I assume I need to go through the build and turn all Werrors off, as I don't see any deal breaker erros.

[uweigand]

Okay, finally found time to setup a crossbuild environement on Alpine. I've managed to get parts of runtime built, but musl is apparently utterly unhappy with dotnet's mono. Many errors such as function-like macro '__GLIBC_PREREQ' is not defined and variable 'is_enum' set but not used [-Wunused-but-set-variable]. I assume I need to go through the build and turn all Werrors off, as I don't see any deal breaker erros.

Yes, it's never been built against musl, so there may well be some issues. The __GLIBC_PREREQ most likely comes from this:

#elif defined(__s390x__)

#define MONO_ARCH_HAS_MONO_CONTEXT 1

#include <sys/ucontext.h>

#if __GLIBC_PREREQ(2, 26)
typedef ucontext_t MonoContext;
#else
typedef struct ucontext MonoContext;
#endif

so this needs to be fixed for musl.

[ayakael]

Indeed, I'll have to clean up mono for musl. Considering that, I've setup a development protocol for crossbuilding on Alpine.

1) (done) Port your s390x crossbuild script to APKBUILD, allowing Alpine's abuild to manage dependencies. The goal is to have one APKBUILD for crossbuilding to any Alpine arches without prebuilt SDKs. See repo 2) Build non-crossbuild x64 build of runtime, roslyn, sdk, aspnetcore and installer. This will iron out any non-crossbuild problems with the build process. 3) Crossbuild to aarch64. Since I've already ironed out linux-musl-arm64 issues and patched that, this will iron out any crossbuild related issues. 4) Crossbuild to armhf. This will allow me to bootstrap armhf which is not available right now. 5) Crossbuild to s390x. Given all of the above, whatever issues I'll encounter will have to do with getting s390x working on musl.

I'm currently at step 2. I've succesfully got runtime built, but I'm stuck on roslyn due to the following:

CSC : error CS1705: Assembly 'Microsoft.CodeAnalysis' with identity 'Microsoft.CodeAnalysis, Version=3.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' uses 'System.Collections.Immutable, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version than referenced assembly 'System.Collections.Immutable' with identity 'System.Collections.Immutable, Version=1.1.36.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' [/home/user/projects/dotnet6-cross/src/source-build-tarball-6.0.108/src/roslyn/src/Tools/Source/CompilerGeneratorTools/Source/CSharpSyntaxGenerator/CSharpSyntaxGenerator.csproj]

Roslyn's restore seems really bent on using 1.1.36, any idea what lever I need to switch?

[ayakael]

Note that for the above I'm trying to build 6.0.108 / 6.0.8, as fixing 6.0.100 on current Alpine would be a lot of work for nothing.

[uweigand]

Looking at your repo, I see things like this:

_build_roslyn() {
        # without these exports, build still attempts a download of dotnet sdk
        export _InitializeDotNetCli="$_cli_root"
        export DOTNET_INSTALL_DIR="$_cli_root"
        export NUGET_PACKAGES="$_nugetdir"

        cd "$builddir"/src/roslyn
        ./eng/build.sh --restore --build --pack -c $_conf
        for I in artifacts/packages/$_conf/*/*.nupkg; do
                $_nuget add $I -Source "$_packagesdir"
        done
}

The point of my cross-build scripts was that you should allow downloading the SDK. Since it's an Intel-based cross-build, those will be available, and they'll provide all the exact versions that the build needs.

Once you've used these scripts to create an initial bootstrap SDK, you can then use that bootstrap SDK to (natively) build the actual final version of the SDK. This would typically use the source-build method, with doesn't require any additional downloads.

[ayakael]

Looking at your repo, I see things like this:

_build_roslyn() {
        # without these exports, build still attempts a download of dotnet sdk
        export _InitializeDotNetCli="$_cli_root"
        export DOTNET_INSTALL_DIR="$_cli_root"
        export NUGET_PACKAGES="$_nugetdir"

        cd "$builddir"/src/roslyn
        ./eng/build.sh --restore --build --pack -c $_conf
        for I in artifacts/packages/$_conf/*/*.nupkg; do
                $_nuget add $I -Source "$_packagesdir"
        done
}

The point of my cross-build scripts was that you should allow downloading the SDK. Since it's an Intel-based cross-build, those will be available, and they'll provide all the exact versions that the build needs.

Once you've used these scripts to create an initial bootstrap SDK, you can then use that bootstrap SDK to (natively) build the actual final version of the SDK. This would typically use the source-build method, with doesn't require any additional downloads.

Unfortunately whether we pull dotnet v6.0.100 from Microsoft or use Alpine's dotnet 6.0.108 package, the result is the same. Maybe its a matter of building v6.0.100, as it was in your script originally. I have terrible memories of that release, it was really problematic to build on musl.

[uweigand]

Unfortunately whether we pull dotnet v6.0.100 from Microsoft or use Alpine's dotnet 6.0.108 package, the result is the same. Maybe its a matter of building v6.0.100, as it was in your script originally. I have terrible memories of that release, it was really problematic to build on musl.

Hmm, I haven't tried building 6.0.108 myself yet. I'll see if I can get it to work.

[ayakael]

Unfortunately whether we pull dotnet v6.0.100 from Microsoft or use Alpine's dotnet 6.0.108 package, the result is the same. Maybe its a matter of building v6.0.100, as it was in your script originally. I have terrible memories of that release, it was really problematic to build on musl.

Hmm, I haven't tried building 6.0.108 myself yet. I'll see if I can get it to work.

Appreciate your help :) Next thing I should try is figuring out what flags source-build applies to roslyn's build

  <ItemGroup>
    <!--
      From roslyn Versions.props:
        The version of Roslyn we build Source Generators against that are built in this
         repository. This must be lower than MicrosoftNetCompilersToolsetVersion,
         but not higher than our minimum dogfoodable Visual Studio version, or else
         the generators we build would load on the command line but not load in IDEs.
      In source-build these don't need to be pinned and can use the source-built versions since it doesn't
      need to support VS.
    -->
    <ExtraPackageVersionPropsPackageInfo Include="SourceGeneratorMicrosoftCodeAnalysisVersion" Version="%24(MicrosoftCodeAnalysisCommonVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemCollectionsImmutableVersion" Version="%24(SystemCollectionsImmutableVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemReflectionMetadataVersion" Version="%24(SystemReflectionMetadataVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemRuntimeCompilerServicesUnsafeVersion" Version="%24(SystemRuntimeCompilerServicesUnsafeVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemTextEncodingCodePagesVersion" Version="%24(SystemTextEncodingCodePagesVersion)" />
  </ItemGroup

this from repos/roslyn.proj seems relevant.

[uweigand]

Hmm, I wasn't seeing any issue with building roslyn. To be specific, I'm using the version included in 6.0.108, which is Microsoft.NETCore.Compilers.4.0.1-1.22181.2.nupkg etc. built from the roslyn repo commit 487283bcd8d66693091f2800dcf1c8ae37cccdee, using a build command along the lines of:

./eng/build.sh --restore --build --pack -c Release /p:VersionSuffix=1.22181.2

[ayakael]

Silly me! It was caused by 0001-lift-version-of-Microsoft.CodeAnalysis.Common-depend.patch from source-build. I pulled roslyn's source code from the cached tarball produced by installer. After reversing that patch, it works. Looking back, I should've pulled the source code directly, but I have limited bandwidth right now so it would've expensive.

[ayakael]

Now at the stage of doing a control crossbuild to armhf, skipping aarch64 due to lack of time. I get this error when building runtime:

  /var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm.Release/ilasm: line 1: syntax error: unexpected word (expecting ")")
/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.net.sdk.il/6.0.0-rc.1.21415.6/targets/Microsoft.NET.Sdk.IL.targets(145,5): error MSB3073: The command ""/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm.Release/ilasm" -QUIET -NOLOGO -OPTIMIZE -DLL  -DEBUG=OPT -INCLUDE="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/version/" -OUTPUT="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/System.Runtime.CompilerServices.Unsafe.dll" -KEY="/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.dotnet.arcade.sdk/6.0.0-beta.22161.1/tools/snk/MSFT.snk" System.Runtime.CompilerServices.Unsafe.il" exited with code 2. [/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/src/libraries/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.ilproj]

Any idea what might cause this?

EDIT:

aarch64 has a similar error:

  /var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm64.Release/ilasm: line 0: syntax error: unterminated quoted string
/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.net.sdk.il/6.0.0-rc.1.21415.6/targets/Microsoft.NET.Sdk.IL.targets(145,5): error MSB3073: The command ""/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm64.Release/ilasm" -QUIET -NOLOGO -OPTIMIZE -DLL  -DEBUG=OPT -INCLUDE="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/version/" -OUTPUT="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/System.Runtime.CompilerServices.Unsafe.dll" -KEY="/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.dotnet.arcade.sdk/6.0.0-beta.22161.1/tools/snk/MSFT.snk" System.Runtime.CompilerServices.Unsafe.il" exited with code 2. [/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/src/libraries/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.ilproj]

EDIT 2 Linking /bin/sh to /bin/bash rather than Alpine's default /bin/busybox changed the error messages, now:

cannot execute binary file: Exec format error
/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.net.sdk.il/6.0.0-rc.1.21415.6/targets/Microsoft.NET.Sdk.IL.targets(145,5): error MSB3073: The command ""/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm.Release/ilasm" -QUIET -NOLOGO -OPTIMIZE -DLL  -DEBUG=OPT -INCLUDE="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/version/" -OUTPUT="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/System.Runtime.CompilerServices.Unsafe.dll" -KEY="/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.dotnet.arcade.sdk/6.0.0-beta.22161.1/tools/snk/MSFT.snk" System.Runtime.CompilerServices.Unsafe.il" exited with code 126. [/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/src/libraries/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.ilproj]

Thus ilasm is being built for arm, but not for x64...

EDIT 3 Fixed by setting ILAsmToolPath to a path containing an x64 ilasm.

[ayakael]

Terrific news! armhf crossbuild was a success- although I am now experiencing issues with the first full build of source-build due to build requiring stable versions of ILAsm, ILDAsm, and TestHost. Unfortunately, despite what your script expects, those nupkgs were generated in the NonShipping packages folder. Indeed, the version is 6.0.8-dev. With the /p:DotNetUseShippingVersions=true flag, I got it to 6.0.8-dev.22428.1, but unfortunately source build still does not recognize that as stable. I tried setting /p:PreReleaseVersionLabel=servicing /p:PB_VersionStamp=servicing /p:PackageVersionStamp=servicing for dev to become servicing to no avail. I'm sure it's a matter of putting in the correct flag. Would anyone have any pointers?

EDIT 1 No such luck when setting /p:VersionSuffix=22363 and executing sed 's|<IsShipping>false</IsShipping>|<IsShipping>true</IsShipping>|' -i src/coreclr/.nuget/Directory.Build.props to force Shipping.

EDIT 2 Fixed by manually modifying the nupkg file to set version to 6.0.0 in nuspec. source-build doesn't take anything else for those packages. It'd be nice to have another way to do this, but at least this hack seems to work.

[ayakael]

I got Mono fixed, and managed to crossbuild an SDK on dotnet7 :) Right now I just need an s390x VM to work out the kinks. Any idea where I might find that?

[uweigand]

I got Mono fixed, and managed to crossbuild an SDK on dotnet7 :) Right now I just need an s390x VM to work out the kinks. Any idea where I might find that?

You can get access to your own s390x VM here: https://linuxone.cloud.marist.edu

If the default setup of those machines doesn't meet your requirements, let me know and we can work something out.

[ayakael]

I made a request last week, and awaiting response from their team :). I've been moving along via our CI pipeline, but the build has been really unstable. Every build fails for different reason, so it makes me think that the issue is with the CI.

[ayakael]

Got the s390x VM setup, and it looks like dotnet6 is pretty stable build wise - the bootstrap can build on itself (thus only runtime, roslyn, sdk, aspnet, installer). As for the full sdk, I'm getting a bunch of CompilerServer: fatal error - server reports different hash version than build task errors despite setting UseSharedCompilation to false.

[uweigand]

Got the s390x VM setup, and it looks like dotnet6 is pretty stable build wise - the bootstrap can build on itself (thus only runtime, roslyn, sdk, aspnet, installer). As for the full sdk, I'm getting a bunch of CompilerServer: fatal error - server reports different hash version than build task errors despite setting UseSharedCompilation to false.

That's strange, the compile server normally works fine for me. This may be a problem with running different dotnet versions at the same time? Are there any old compile server tasks still left running?

[ayakael]

Got the s390x VM setup, and it looks like dotnet6 is pretty stable build wise - the bootstrap can build on itself (thus only runtime, roslyn, sdk, aspnet, installer). As for the full sdk, I'm getting a bunch of CompilerServer: fatal error - server reports different hash version than build task errors despite setting UseSharedCompilation to false.

That's strange, the compile server normally works fine for me. This may be a problem with running different dotnet versions at the same time? Are there any old compile server tasks still left running?

The error occurs in a pipeline environment, so it can't be due to running different dotnet versions. It only occurs on s390x.

[ayakael]

Fortunately, setting UseSharedCompilation=false in problematic repos/*.proj files does the trick. Next error: NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-musl-s390x'.. This was to be expected, which is why I crosscompiled runtime.linux-musl-s390x.Microsoft.NETCore.DotNetAppHost.6.0.9.nupkg and Microsoft.NETCore.App.Host.linux-musl-s390x.6.0.9.nupkg. Though note TestHost didn't seem to want to be crosscompiled.

[ayakael]

Fixed the apphost issue by doing as fedora did and disabling use of apphost in various components.

Unfortunately the CompileServer issue is basically unsurmountable on full source-build. While I could get the bootstrap components (runtime, roslyn, sdk, aspnetcore, installer) to build, the full stack doesn't pickup UseSharedCompilation=false consistently. source-build is currently the most problematic given it being a collection of various components. What could break the compile server like this for linux-musl-s390x?

@omajid, have you encountered this error when porting to fedora?

CompilerServer: fatal error - server reports different hash version than build task

[ayakael]

Disabling shared compilation via

local _sharedcompArray="
   Microsoft.CSharp.CurrentVersion.targets
   Microsoft.VisualBasic.CurrentVersion.targets
   Sdks/Microsoft.NET.Sdk.Razor/targets/Microsoft.NET.Sdk.Razor.Component.targets
   Sdks/Microsoft.NET.Sdk.Razor/targets/Microsoft.NET.Sdk.Razor.Compilation.targets
   Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets
   Roslyn/Microsoft.Managed.Core.targets
"
for i in $_sharedcompArray; do
  sed -i 's|<UseSharedCompilation>true</UseSharedCompilation>|<UseSharedCompilation>false</UseSharedCompilation>|' "$_cli_root"/sdk/*/$i
done

where _cli_root equals location of bootstrap seems to force the false value on this flag accross the board.

Although now build of newtonsoft-json fails with many errors of:

/home/build/dotnet6/community/dotnet6-build/src/bootstrap/sdk/6.0.109/Roslyn/Microsoft.CSharp.Core.targets(75,5): error : [ERROR] FATAL UNHANDLED EXCEPTION: System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space. [/home/build/dotnet6/community/dotnet6-build/src/dotnet-v6.0.109/src/source-build/artifacts/source-build/self/src/src/newtonsoft-json/Src/Newtonsoft.Json/Newtonsoft.Json.csproj]

The most recent version of newtonsoft-json gives the same error.

[ayakael]

Upgrading to 6.0.401, thus upgrading roslyn, seems to fix the shared compilation errors. The stack trace errors are still an issue though, suggesting that something is broken with my crosscompiled sdk.

edit Nope, shared compilation still broken... there's something fundamentally broken with my crossgen and I can't find what it is. But, one thing is for sure, both dotnet7 and dotnet6 give me System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. errors

[ayakael]

It turns out the issues that I'm encountering have less to do with s390x and more to do with mono-backed runtime being broken on musl. The stack issue is being tracked here. The shared compilation issue only exists on dotnet6, thus suggesting it has already been fixed and needing backporting. Shared compilation issue tracked here

[ayakael]

The stack issues seem to have been fixed. The current roadblock is a mono bug on musl making build of fsharp fail. It is being tracked here: https://github.com/dotnet/runtime/issues/76805. I suspect it is the last bug to deal with given fsharp being one of the last components to build.

[ayakael]

Moving right along now, it seems like we got a full source-build build for .NET 6.0 on s390x. For .NET 7.0, there seems to be a regression:

The process cannot access the file '/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/artifacts/obj/System.IO.Ports' because it is being used by another process

Indeed, it doesn't take long for this kind of error to appear in CI. I set ulimit -n 8192 to no avail.

[ayakael]

Closing as s390x support is now merged in dotnet6.

[uweigand]

That's really excellent news! Thanks for your efforts in making this happen, @ayakael !

[ayakael]

That's really excellent news! Thanks for your efforts in making this happen, @ayakael !

Thanks for your help in making it happen ^^ There is still one issue facing .net7 on s390x, but that's going to be tracked in https://github.com/dotnet/source-build/issues/3020

Now that mono is fixed for musl, ppc64le support is trivial on dotnet7 (we even got a bootstrap crossbuilt already). Our crossbuild aware package builder, designed off of your s390x script, makes it much easier to port to new platforms.

If anyone should want to model a script around our packabe builder, it reads very much like a shell script and it is available here

[ayakael]

I'm reopening this issue as s390x and ppc64le on Alpine is still plagued by stability problems, namely issues related to processes accessing the same file and process stalls which seem to be related to roslyn. These are hard to reproduce and I do not have the time / experience to work on this further. For anyone who wants to work further on this, I make the bootstraps available at: https://lab.ilot.io/ayakael/dotnet-stage0/-/releases