[SBRP] Segfault during official build, and when building with low memory

dagood commented 4 years ago

When I build soruce-build-reference-packages cd5a8c6f with low memory, it fails with this end of log:

...
"/work/src/referencePackages/buildReferencePackages.proj" (default target) (1) ->
(Build target) ->
  /work/src/referencePackages/buildReferencePackages.proj(55,5): error MSB3073: The command "$__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/ref
erencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Ext
ensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649
,CS3001,CS3002,CS3003,CS3009" exited with code 139.

    1305 Warning(s)
    1 Error(s)

There's a segfault in an Exec task. From the binlog (also shows up on console):

Exec
    Assembly = Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
    CommandLineArguments = $__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009
    Parameters
        Command = $__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009
    Microsoft (R) Build Engine version 16.3.2+e481bbf88 for .NET Core
    Copyright (C) Microsoft Corporation. All rights reserved.
    /work/.dotnet/sdk/3.0.101/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/work/.dotnet/sdk/3.0.101/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/work/.dotnet/sdk/3.0.101/dotnet.dll -maxcpucount -property:PackageOutputPath=/work/artifacts/reference-packages/ /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ -restore -target:pack -verbosity:m /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009 /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj
      Restore completed in 71.42 ms for /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj.
      System.Reflection.Extensions -> /work/artifacts/bin/system.reflection.extensions/4.3.0/ref/netstandard1.0/System.Reflection.Extensions.dll
      Successfully created package '/work/artifacts/reference-packages/System.Reflection.Extensions.4.3.0.nupkg'.
    Warnings
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5131: ...
        /work/.dotnet/sdk/3.0.101/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(198,5): warning NU5048: ...
    /tmp/tmp9059cebb54e44b989a9188feed9b95e1.exec.cmd: line 2: 48710 Segmentation fault      (core dumped) $__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009
    Errors
        /work/src/referencePackages/buildReferencePackages.proj(55,5): error MSB3073: The command "$__DOTNET_CMD pack --output /work/artifacts/reference-packages/ /work/src/referencePackages/src/system.reflection.extensions/4.3.0/System.Reflection.Extensions.csproj /bl:/work/artifacts/log/system.reflection.extensions/4.3.0/System.Reflection.Extensions.binlog /p:ArtifactsBinDir=/work/artifacts/bin/ /p:ArtifactsObjDir=/work/artifacts/obj/ /p:RepoRoot=/work/ /nowarn:NU1603,NU1605,NU5125,CS0109,CS0169,CS0618,CS0649,CS3001,CS3002,CS3003,CS3009" exited with code 139. [/work/src/referencePackages/buildReferencePackages.proj]

This same error has been showing up as flakiness in the SBRP official build.

When I run the build locally, I see an error like this in `dmesg | grep -C10 'out of memory':

ory values in pages):
[10633484.918482] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[10633484.918483] [2331245]  1000 2331245     1005      691    40960      121             0 bash
[10633484.918485] [2331280]  1000 2331280   842737    12282   499712     4282             0 dotnet
[10633484.918486] [2331298]  1000 2331298   894790    10661   483328     6906             0 dotnet
[10633484.918487] [2331320]  1000 2331320      596      145    40960       24             0 sh
[10633484.918488] [2331322]  1000 2331322   842819    13750   499712     4119             0 dotnet
[10633484.918489] [2331340]  1000 2331340   932950    23921   606208     1298             0 dotnet
[10633484.918489] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=24768a8d1b80e0c40481a7aeca8cd63844eb99c524c2af1ee236ac3dcca2de8b,mems_allowed=0,oom_memcg=/d
ocker/24768a8d1b80e0c40481a7aeca8cd63844eb99c524c2af1ee236ac3dcca2de8b,task_memcg=/docker/24768a8d1b80e0c40481a7aeca8cd63844eb99c524c2af1ee236ac3dcca2de8b,task=dotnet,pid
=2331340,uid=1000
[10633484.918525] Memory cgroup out of memory: Killed process 2331340 (dotnet) total-vm:3731800kB, anon-rss:43584kB, file-rss:52100kB, shmem-rss:0kB, UID:1000 pgtables:59
2kB oom_score_adj:0
[10633484.921746] oom_reaper: reaped process 2331340 (dotnet), now anon-rss:4kB, file-rss:0kB, shmem-rss:0kB

The inner binlog appears 100% successful, the error appears to be "outside" the inner MSBuild but still "inside" the Exec.

(I don't know if this is the exact OOM kill that caused the error--I ran some other simulations to see what it does and don't know how to differentiate them. The build takes long enough that I didn't want to just rerun it.)

A few things we can try:

Remove libraries that we're 100% confident no future builds will use to reduce overall footprint.
Investigate situations that might be using more memory than necessary.
- I wonder if calling so many Execs with inner builds might be causing some particularly bad behavior in the binlog logger. I want to try this out with the 5.0 changes (no more Execs) to see if it still happens there, and play with the limit.
Upgrade the distro version.
Upgrade the .NET Core version.
Disable binlogs. (😢 please, no.)

I haven't tried to actually get a core dump, not sure if it'll actually have anything valuable in this situation. Maybe it would let MSBuild improve its error logging or handle this without a segfault in a future version.

I used --memory 512MB on my docker run command to repro the SBRP error.

dagood commented 4 years ago

I want to try this out with the 5.0 changes (no more Execs) to see if it still happens there, and play with the limit.

The dev branch starts to fail somewhere between 1.5 to 2 GB. (Without binlogs enabled: default in that branch. Not that I know if this affects it.)

dseefeld commented 4 years ago

Perhaps adding something to log memory usage to see how much memory the build is consuming at different stages to help identify memory requirements for the build.

MichaelSimons commented 1 month ago

[Triage] Closing - We haven't heard of any reports over the past several years of this being an issue.

dotnet / source-build

[SBRP] Segfault during official build, and when building with low memory #1687