search

dotnet / diagnostics

This repository contains the source code for various .NET Core runtime diagnostic tools and documents.
MIT License
1.18k stars 354 forks source link

`dotnet-dump collect` -> EndOfStreamException; Process-being-dumped: "[createdump] Target process is alive" #5045

Closed johnterickson closed 1 day ago

johnterickson commented 3 days ago

Description

I had a long running process going over night and when I checked it this morning it was showing a bunch of timeout errors (see below) so I decided to capture a dump. This did nothing for a long time and then ultimately failed, though the debuggee-side had messages that made me think it may have succeeded (but it did not).

My alias is jerick if you want to ping me on teams.

---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
  at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)

Configuration

C:\Users\jerick>wsl --version
WSL version: 2.3.24.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.65
MSRDC version: 1.2.5620
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26100.2033
john@CPC-jeric-LCTKK:~/PackageAnalyzer/PackageAnalyzer$ dotnet-dump --version
8.0.547301+358ff9556edceba1e8689fbfd1fc67588d8ab6cb
john@CPC-jeric-LCTKK:~/PackageAnalyzer/PackageAnalyzer$ dotnet --info
.NET SDK:
 Version:           8.0.110
 Commit:            87a66bb3d1
 Workload version:  8.0.100-manifests.9143487c

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  24.04
 OS Platform: Linux
 RID:         ubuntu.24.04-x64
 Base Path:   /usr/lib/dotnet/sdk/8.0.110/

.NET workloads installed:
 Workload version: 8.0.100-manifests.9143487c
There are no installed workloads to display.

Host:
  Version:      8.0.10
  Architecture: x64
  Commit:       81cabf2857

.NET SDKs installed:
  8.0.110 [/usr/lib/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.10 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.10 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Regression?

john@CPC-jeric-LCTKK:~/PackageAnalyzer/PackageAnalyzer$ dotnet-dump collect -n PackageAnalyzer
PackageAnalyzer
Writing full to /home/john/PackageAnalyzer/PackageAnalyzer/core_20241110_143743
Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.Stream.ReadAtLeastCore(Span`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream)
   at System.IO.BinaryReader.InternalRead(Int32 numBytes)
   at System.IO.BinaryReader.ReadUInt16()
   at Microsoft.Diagnostics.NETCore.Client.IpcHeader.Parse(BinaryReader reader) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 55
   at Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(Stream stream) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 117
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(Stream stream) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 107
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 44
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 25
   at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(DumpType dumpType, String dumpPath, WriteDumpFlags flags) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:li
ne 171
   at Microsoft.Diagnostics.Tools.Dump.Dumper.Collect(IConsole console, Int32 processId, String output, Boolean diag, Boolean crashreport, DumpTypeOption type, String name) in /_/src/Tools/dotnet-dump/Dumper.cs:line 1
36
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__21_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__22_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__20_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()
john@CPC-jeric-LCTKK:~/PackageAnalyzer/PackageAnalyzer$
^C^C^C^C[createdump] Gathering state for process 3749281 PackageAnalyzer
[createdump] Writing full dump to file /home/john/PackageAnalyzer/PackageAnalyzer/core_20241110_143743
[createdump] Target process is alive
Killed
john@CPC-jeric-LCTKK:~/PackageAnalyzer/PackageAnalyzer/bin/Debug/net8.0$ ls -l /home/john/PackageAnalyzer/PackageAnalyzer/core_20241110_143743
ls: cannot access '/home/john/PackageAnalyzer/PackageAnalyzer/core_20241110_143743': No such file or directory
hoyosjs commented 1 day ago

Reaching out directly

hoyosjs commented 1 day ago

This is currently not reproducible and we don't have a way to know where the communication got lost such that it truncated the stream. Upon retry - this worked as expected. I am closing this as no-repro, but feel free to open an issue if this happens again.