dotnet / dotnet-docker

Docker images for .NET and the .NET Tools.
https://hub.docker.com/_/microsoft-dotnet
MIT License
4.48k stars 1.94k forks source link

Collect core dumps from tests #3377

Open mthalman opened 2 years ago

mthalman commented 2 years ago

On rare occasions, I sometimes see a test randomly fail due to the container crashing.

Here's an example of such a failure:

  Failed Microsoft.DotNet.Docker.Tests.RuntimeImageTests.VerifyAppScenario(imageData: Version='3.1', VersionString='3.1', HasCustomSdk='False', SdkOS='bionic', Arch='Arm64', IsArm='True', OS='bionic', IsDistroless='False', Rid='linux-arm64') [24 s]
  Error Message:
   System.InvalidOperationException : Failed to execute docker build -t 3.1-build-132858653826941415 --target build --build-arg sdk_image=mcr.microsoft.com/dotnet/nightly/sdk:3.1-bionic-arm64v8 --build-arg runtime_image=mcr.microsoft.com/dotnet/nightly/runtime:3.1-bionic-arm64v8 --build-arg runtime_deps_image=mcr.microsoft.com/dotnet/nightly/runtime-deps:3.1-bionic-arm64v8 /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/bin/Debug/net6.0/consoleApp132858653757239058
Exit code: 134
Standard Error: The command '/bin/sh -c dotnet build --no-restore' returned a non-zero code: 134
  Stack Trace:
     at Microsoft.DotNet.Docker.Tests.DockerHelper.Execute(String args, Boolean ignoreErrors, Boolean autoRetry, ITestOutputHelper outputHelper) in /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/DockerHelper.cs:line 127
   at Microsoft.DotNet.Docker.Tests.DockerHelper.ExecuteWithLogging(String args, Boolean ignoreErrors, Boolean autoRetry) in /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/DockerHelper.cs:line 142
   at Microsoft.DotNet.Docker.Tests.DockerHelper.Build(String tag, String dockerfile, String target, String contextDir, Boolean pull, String[] buildArgs) in /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/DockerHelper.cs:line 50
   at Microsoft.DotNet.Docker.Tests.ImageScenarioVerifier.BuildTestAppImage(String stageTarget, String contextDir, String[] customBuildArgs) in /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/ImageScenarioVerifier.cs:line 159
   at Microsoft.DotNet.Docker.Tests.ImageScenarioVerifier.Execute() in /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/ImageScenarioVerifier.cs:line 55
   at Microsoft.DotNet.Docker.Tests.RuntimeImageTests.VerifyAppScenario(ProductImageData imageData) in /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/RuntimeImageTests.cs:line 28
--- End of stack trace from previous location ---
  Standard Output Messages:
 Executing: docker run --name 3.1-create-console-132858653757239218 -w /app  mcr.microsoft.com/dotnet/nightly/sdk:3.1-bionic-arm64v8 dotnet new console --framework netcoreapp3.1 --no-restore
 Getting ready...
 The template "Console Application" was created successfully.
 Execution Elapsed Time: 00:00:06.0189880
 Executing: docker cp 3.1-create-console-132858653757239218:/app /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/bin/Debug/net6.0/consoleApp132858653757239058
 Execution Elapsed Time: 00:00:00.3620946
 Executing: docker container rm -f 3.1-create-console-132858653757239218
 3.1-create-console-132858653757239218
 Execution Elapsed Time: 00:00:00.1730087
 Executing: docker build -t 3.1-build-132858653826941415 --target build --build-arg sdk_image=mcr.microsoft.com/dotnet/nightly/sdk:3.1-bionic-arm64v8 --build-arg runtime_image=mcr.microsoft.com/dotnet/nightly/runtime:3.1-bionic-arm64v8 --build-arg runtime_deps_image=mcr.microsoft.com/dotnet/nightly/runtime-deps:3.1-bionic-arm64v8 /repo/dotnet-docker/tests/Microsoft.DotNet.Docker.Tests/bin/Debug/net6.0/consoleApp132858653757239058
 Sending build context to Docker daemon  7.168kB

 Step 1/11 : ARG sdk_image
 Step 2/11 : ARG runtime_image
 Step 3/11 : ARG runtime_deps_image
 Step 4/11 : FROM $sdk_image as build
  ---> e623ec9d9504
 Step 5/11 : EXPOSE 80
  ---> Using cache
  ---> 0967370806e0
 Step 6/11 : WORKDIR app
  ---> Using cache
  ---> 4e093925e2b1
 Step 7/11 : COPY NuGet.config .
  ---> Using cache
  ---> 7ad0b7e86846
 Step 8/11 : COPY *.csproj .
  ---> 16c3f75c6188
 Step 9/11 : RUN dotnet restore
  ---> Running in 8e56a4a7e6d8
   Determining projects to restore...
   Restored /app/app.csproj (in 517 ms).
 Removing intermediate container 8e56a4a7e6d8
  ---> d1c4de8f2edc
 Step 10/11 : COPY . .
  ---> fccf0396e101
 Step 11/11 : RUN dotnet build --no-restore
  ---> Running in c6c7f24a1840
 Microsoft (R) Build Engine version 16.7.2+b60ddb6f4 for .NET
 Copyright (C) Microsoft Corporation. All rights reserved.

   app -> /app/bin/Debug/netcoreapp3.1/app.dll

 Build succeeded.
     0 Warning(s)
     0 Error(s)

 Time Elapsed 00:00:04.79
 \x1b[91mmalloc_consolidate(): invalid chunk size
 \x1b[0m\x1b[91mAborted (core dumped)
 \x1b[0m
 The command '/bin/sh -c dotnet build --no-restore' returned a non-zero code: 134

These types of failures are nearly impossible to investigate today because they rarely happen, there is no known repro, and no crash dump is being collected by the test.

I propose that the tests be configured to automatically collect the core dump from the container and publish it as an artifact in the pipeline so that it can be investigated.

Mantools1 commented 2 years ago

On rare occasions