dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.36k stars 4.74k forks source link

dotnet 6 - Segfault -- Runtime compilation [crossgen2] #62727

Open zalaare opened 2 years ago

zalaare commented 2 years ago

Description

NOTE: I believe this is centered around LLVM/libc++/libc++abi toolchain

When compiling dotnet 6.0/7.0/8.0 either via dotnet/installer or dotnet/runtime direct compilation stops with error

Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/log/CrossgenCoreLib_Linux__x64__Release.log
/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/dotnet.sh /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 --perfmap-format-version:1  --embed-pgo-data -O /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap --perfmap-path:/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/dotnet.sh: line 27: 72063 Aborted                 (core dumped) ${dotnetPath} "$@"

I know enough to muddle through and collect logs and such (sometimes I need instruction) if needed, but dotnet proj files are out of my wheelhouse at this time.

Reproduction Steps

Either use docker tag dotnet-bugreport-1 OR dotnet-bugreport-1-sparse NOTE TAG dotnet-bugreport-1 IMAGE IS MASSIVE as it includes all the previously built sources [Assumption of a working Docker setup]

docker run -it --rm --privileged zalaare/zizzy:dotnet-bugreport-1
cd dotnet
makepkg -Cfsr --noconfirm -p ZPKBUILD.60.dotnet-bugreport-1

NOTES:

Expected behavior

I expect crossgen to not fail. Either by building it natively during compilation and using it (no segfault expected) or being able to skip it. UseCrossGen2=false only results in the file crossgen not being found.

Actual behavior

segfault on the execution of crossgen2.dll during the compilation of runtime [tag 6.0.1]

    Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/log/CrossgenCoreLib_Linux__x64__Release.log
    /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/dotnet.sh /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 --perfmap-format-version:1  --embed-pgo-data -O /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap --perfmap-path:/home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
    /home/build/dotnet/src/installer/src/runtime.3a25a7f1cc446b60678ed25c9d829420d6321eba/artifacts/source-build/self/src/dotnet.sh: line 27: 72063 Aborted                 (core dumped) ${dotnetPath} "$@"

--stacktrace-- strace --log-- runtime-portable.log

Regression?

source-build continues to work fine on dotnet/5 and dotnet/3

Known Workarounds

No response

Configuration

This is for my custom docker focused distro I maintain. The architecture is amd64 only at this time.

Other information

No response

EDIT - fixed some instructions that didn't work EDIT2 - added actual error from log to OP for ease of reading EDIT3 - updated for tag 6.0.1

ghost commented 2 years ago

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details
### Description NOTE: I **believe** this is centered around **LLVM/libc++/libc++abi** toolchain When compiling dotnet 6.0 either via dotnet/installer or dotnet/runtime direct compilation stops with error I know enough to muddle through and collect logs and such (sometimes I need instruction) if needed, but dotnet proj files are out of my wheelhouse at this time. ### Reproduction Steps [Assumption of a working Docker setup] ```docker pull zalaare/zizzy:dotnet-bugreport-1 docker run -it --rm zalaare/zizzy:dotnet-bugreport-1 su - build cd dotnet makepkg -Cfsr --noconfirm -p ZPKBUILD.60.dotnet-bugreport-1 ``` NOTES: * i've left the previously failed compilation within the docker image if you want to examine logs and results prior to seeing the errors yourself. they exist under the src directory within /home/build/dotnet. * this is pretty close to my official build script for dotnet, however this one is specific to minimize time spent waiting. * this still takes a while to build up to runtime [runtime-portable] and fail * makepkg -Cfsr --noconfirm basically translates to build this package, first cleaning out existing sources, replace any packages already existing with this packages names, search and install any required dependencies for building, and remove these same dependencies upon exiting the build procedure. Oh and don't ask to install or remove, just do it. * the patch drop-command-line-api is for a build error during the compilation of command-line-api that seems to be a requisite of roslyn (however roslyn seems to build fine when i commented it out). This patch i'm hopeful will go away when i have a successful build of dotnet to use for bootstrapping. * the bootstrapped dotnet we download and use for building this was created by allowing (source-build) to fetch the dotnet online and then using the old buildbootstrapcli.sh steps to build the shims/native libs from 6.0.0-100 and piecing them back into their respective locations. I've had to do this for dotnet 2, 3 and 5 prior and it has worked okay enough to get me going each time. ### Expected behavior I expect crossgen to not fail. Either by building it natively during compilation and using it (no segfault expected) or being able to skip it. UseCrossGen2=false only results in the file crossgen not being found. ### Actual behavior segfault on the execution of crossgen2.dll during the compilation of runtime [tag 6.0.0] ``` Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to /home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/artifacts/log/CrossgenCoreLib_Linux__x64__Release.log /home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/dotnet.sh \ /home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll \ -o:/home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll \ -r:/home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll \ --targetarch:x64 --perfmap-format-version:1 --embed-pgo-data \ -O /home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll \ --perfmap --perfmap-path:/home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/ /home/build/dotnet/src/installer/src/runtime.4822e3c3aa77eb82b2fb33c9321f923cf11ddde6/artifacts/source-build/self/src/dotnet.sh: line 27: 163732 Aborted (core dumped) ${dotnetPath} "$@" ``` --stacktrace-- [strace](https://zalaare.freeddns.org/junkbin/strace-log) --log-- [runtime-portable.log](https://zalaare.freeddns.org/junkbin/runtime-portable.log) ### Regression? source-build continues to work fine on dotnet/5 and dotnet/3 ### Known Workarounds _No response_ ### Configuration This is for my custom docker focused distro I maintain. The architecture is amd64 only at this time. ### Other information _No response_
Author: zalaare
Assignees: -
Labels: `area-Infrastructure-coreclr`, `untriaged`
Milestone: -
mangod9 commented 2 years ago

Hi @zalaare, could you please elaborate on what is failing? Is it that precompiling private.corelib is failing during a source-build? Since you mention a new toolchain, wonder if you are on clang13 and hitting this: https://github.com/dotnet/runtime/issues/61671

zalaare commented 2 years ago

Hi @mangod9, thanks for looking at this. I assume you are right in that precompiling private.corelib might indeed be what is failing. Whatever step "Generating native image of System.Private.CoreLib" is.

I've added the error from the log to the OP to hopefully make it quicker and easier to see. I am using clang 12.0.1 for this as clang 13 has too many issues even with dotnet 3 and dotnet 5 atm.

zalaare commented 2 years ago

Just an update, the problem still exists and I've currently patched the running of crossgen out of the compilation process of runtime (and also aspnetcore). This does at least allow me to successfully build dotnet 6.0.

zalaare commented 1 year ago

Again an update, the problem still exists with dotnet 7.0. Since the milestone moved to 8, I assumed this would be the case.

zalaare commented 1 year ago

Again an update, the problem still exists with dotnet 8.0.

Generating native image of System.Private.CoreLib for linux.x64.Release. Logging to /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/dotnet.sh /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/crossgen2/crossgen2.dll -o:/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/System.Private.CoreLib.dll -r:/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/IL/*.dll --targetarch:x64 --targetos:linux -O /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/ /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/dotnet.sh: line 27: 170150 Segmentation fault (core dumped) ${dotnetPath} "$@" /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(102,5): error MSB3073: The command "/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/dotnet.sh /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/crossgen2/crossgen2.dll -o:/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/System.Private.CoreLib.dll -r:/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/IL/*.dll --targetarch:x64 --targetos:linux -O /zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/zizzy/build/dotnet/src/git.dotnet.dotnet/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/linux.x64.Release/" exited with code 139.