search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.13k stars 4.7k forks source link

`dotnet run` in background will zombie child process and stall its own diagnostics server #54017

Open jander-msft opened 3 years ago

jander-msft commented 3 years ago

Description

When running dotnet run on a project as a background process (e.g. dotnet run &), the child process that is the application will be zombied and the diagnostics server in the dotnet run process is stalled.

Repro steps:

  1. mkdir TestApp
  2. cd TestApp
  3. dotnet new webapp
  4. dotnet run &
dotnet run &
[1] 1074
Building...

The expected bound addresses from the web application do not seem to be bound:

curl http://localhost:5000
curl: (7) Failed to connect to localhost port 5000: Connection refused

Output from ps aux (removed other processes):

ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jander    1074  3.5  0.5 3547660 66996 pts/0   Tl   08:53   0:00 dotnet run
jander    1093 15.6  0.0      0     0 pts/0    Z    08:53   0:01 [dotnet] <defunct>

Configuration

.NET 5.0.6 Ubuntu 18.04 x64 on WSL 2

dotnet --list-runtimes
Microsoft.AspNetCore.All 2.1.28 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.28 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.15 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.28 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.15 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
dotnet --list-sdks
2.1.816 [/usr/share/dotnet/sdk]
3.1.409 [/usr/share/dotnet/sdk]
5.0.300 [/usr/share/dotnet/sdk]

Regression?

Seems to repro for .NET Core 3.1 applications as well. However, I did have .NET Core 2.1 runtimes, .NET 5 runtimes, and the .NET 5 SDK installed on the OS.

Other information

Using nohup to run it in the background (e.g. nohup dotnet run &) seems to work correctly. Running dotnet run as a foreground process works as expected.

Attempting to use diagnostics tools to communicate with the diagnostics server in the dotnet run process has any of those diagnostics tools hang:

Attempting to use dotnet-dump on the dotnet run process (the dotnet-dump process will run indefinitely until process 1074 is killed):

dotnet dump collect --type Full -p 1074

Writing full to /home/jander/core_20210610_094401

Another example is using dotnet-monitor to enumerate processes on the system will hang indefinitely until process 1074 is killed:

dotnet monitor collect --no-auth
...
curl -k https://localhost:52323/processes

call stack of dotnet-monitor that is trying to issue the ProcessInfo command over the diagnostics port:

System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle socket, System.Net.Sockets.SocketFlags flags, System.Span<byte> buffer, byte[] socketAddress, ref int socketAddressLen = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out Interop.Error errno = SUCCESS)
System.Net.Sockets.dll!System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle socket, System.Span<byte> buffer, System.Collections.Generic.IList<System.ArraySegment<byte>> buffers, System.Net.Sockets.SocketFlags flags, byte[] socketAddress, ref int socketAddressLen, out int bytesReceived = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out System.Net.Sockets.SocketError errorCode = Success)
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext.ReceiveFrom(System.Memory<byte> buffer, ref System.Net.Sockets.SocketFlags flags = None, byte[] socketAddress = null, ref int socketAddressLen = 0, int timeout = -1, out int bytesReceived = 0)
System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle handle, byte[] buffer, int offset, int count, System.Net.Sockets.SocketFlags socketFlags, out int bytesTransferred)
System.Net.Sockets.dll!System.Net.Sockets.Socket.Receive(byte[] buffer = {byte[14]}, int offset = 0, int size, System.Net.Sockets.SocketFlags socketFlags, out System.Net.Sockets.SocketError errorCode = 2116193192)
System.Net.Sockets.dll!System.Net.Sockets.NetworkStream.Read(byte[] buffer, int offset, int size)
System.Private.CoreLib.dll!System.IO.BinaryReader.ReadBytes(int count = 14)
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcHeader.TryParse(System.IO.BinaryReader reader = {System.IO.BinaryReader})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint endpoint = {Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint}, Microsoft.Diagnostics.NETCore.Client.IpcMessage message = {Microsoft.Diagnostics.NETCore.Client.IpcMessage})
Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfo()
Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.EndpointInfo.FromProcessId(int processId = 1074)
Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.ClientEndpointInfoSource.GetEndpointInfoAsync.AnonymousMethod__2()
System.Private.CoreLib.dll!System.Threading.Tasks.Task<Microsoft.Diagnostics.Monitoring.EndpointInfo>.InnerInvoke()
System.Private.CoreLib.dll!System.Threading.Tasks.Task..cctor.AnonymousMethod__274_0(object obj)
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread = {System.Threading.Thread}, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot = Id = 2, Status = Running, Method = "Microsoft.Diagnostics.Monitoring.EndpointInfo <GetEndpointInfoAsync>b__2()", Result = "{Not yet computed}", System.Threading.Thread threadPoolThread)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread threadPoolThread)
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread threadPoolThread)
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Private.CoreLib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

I have core dumps of process 1074 (the dotnet run process), process 1142 (the dotnet-monitor process).

Related https://github.com/dotnet/dotnet-monitor/issues/399

cc @josalem

ghost commented 3 years ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
### Description When running `dotnet run` on a project as a background process (e.g. `dotnet run &`), the child process that is the application will be zombied and the diagnostics server in the `dotnet run` process is stalled. Repro steps: 1. mkdir TestApp 1. cd TestApp 1. dotnet new webapp 1. dotnet run & ``` dotnet run & [1] 1074 Building... ``` The expected bound addresses from the web application do not seem to be bound: ``` curl http://localhost:5000 curl: (7) Failed to connect to localhost port 5000: Connection refused ``` Output from `ps aux` (removed other processes): ``` ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jander 1074 3.5 0.5 3547660 66996 pts/0 Tl 08:53 0:00 dotnet run jander 1093 15.6 0.0 0 0 pts/0 Z 08:53 0:01 [dotnet] ``` ### Configuration .NET 5.0.6 Ubuntu 18.04 x64 on WSL 2 ``` dotnet --list-runtimes Microsoft.AspNetCore.All 2.1.28 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.28 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.15 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.28 [/usr/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.15 [/usr/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App] ``` ``` dotnet --list-sdks 2.1.816 [/usr/share/dotnet/sdk] 3.1.409 [/usr/share/dotnet/sdk] 5.0.300 [/usr/share/dotnet/sdk] ``` ### Regression? Unknown at this time, will attempt on prior .NET Core versions soon. ### Other information Using `nohup` to run it in the background (e.g. `nohup dotnet run &`) seems to work correctly. Running `dotnet run` as a foreground process works as expected. Attempting to use diagnostics tools to communicate with the diagnostics server in the `dotnet run` process has any of those diagnostics tools hang: Attempting to use dotnet-dump on the `dotnet run` process (the dotnet-dump process will run indefinitely until process 1074 is killed): ``` dotnet dump collect --type Full -p 1074 Writing full to /home/jander/core_20210610_094401 ``` Another example is using dotnet-monitor to enumerate processes on the system will hang indefinitely until process 1074 is killed: ``` dotnet monitor collect --no-auth ... curl -k https://localhost:52323/processes ``` call stack of dotnet-monitor that is trying to issue the ProcessInfo command over the diagnostics port: ``` System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle socket, System.Net.Sockets.SocketFlags flags, System.Span buffer, byte[] socketAddress, ref int socketAddressLen = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out Interop.Error errno = SUCCESS) System.Net.Sockets.dll!System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle socket, System.Span buffer, System.Collections.Generic.IList> buffers, System.Net.Sockets.SocketFlags flags, byte[] socketAddress, ref int socketAddressLen, out int bytesReceived = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out System.Net.Sockets.SocketError errorCode = Success) System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext.ReceiveFrom(System.Memory buffer, ref System.Net.Sockets.SocketFlags flags = None, byte[] socketAddress = null, ref int socketAddressLen = 0, int timeout = -1, out int bytesReceived = 0) System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle handle, byte[] buffer, int offset, int count, System.Net.Sockets.SocketFlags socketFlags, out int bytesTransferred) System.Net.Sockets.dll!System.Net.Sockets.Socket.Receive(byte[] buffer = {byte[14]}, int offset = 0, int size, System.Net.Sockets.SocketFlags socketFlags, out System.Net.Sockets.SocketError errorCode = 2116193192) System.Net.Sockets.dll!System.Net.Sockets.NetworkStream.Read(byte[] buffer, int offset, int size) System.Private.CoreLib.dll!System.IO.BinaryReader.ReadBytes(int count = 14) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcHeader.TryParse(System.IO.BinaryReader reader = {System.IO.BinaryReader}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint endpoint = {Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint}, Microsoft.Diagnostics.NETCore.Client.IpcMessage message = {Microsoft.Diagnostics.NETCore.Client.IpcMessage}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfo() Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.EndpointInfo.FromProcessId(int processId = 1074) Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.ClientEndpointInfoSource.GetEndpointInfoAsync.AnonymousMethod__2() System.Private.CoreLib.dll!System.Threading.Tasks.Task.InnerInvoke() System.Private.CoreLib.dll!System.Threading.Tasks.Task..cctor.AnonymousMethod__274_0(object obj) System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread = {System.Threading.Thread}, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot = Id = 2, Status = Running, Method = "Microsoft.Diagnostics.Monitoring.EndpointInfo b__2()", Result = "{Not yet computed}", System.Threading.Thread threadPoolThread) System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread threadPoolThread) System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread threadPoolThread) System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() System.Private.CoreLib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() ``` I have core dumps of process 1074 (the `dotnet run` process), process 1142 (the `dotnet-monitor` process). cc @josalem
Author: jander-msft
Assignees: -
Labels: `area-System.Net`, `untriaged`
Milestone: -
josalem commented 3 years ago

Is the process marked as defunct before attempting to do anything over the diagnostics IPC? This may be unrelated to the diagnostics server if so.

Do you see the same behavior if you do dotnet <path/to/dll> & or dotnet exec </path/to/dll> &?

jander-msft commented 3 years ago

Is the process marked as defunct before attempting to do anything over the diagnostics IPC? This may be unrelated to the diagnostics server if so.

Yes, the process is defunct before attempting diagnostics IPC. However, its the child process (1093) that is defunct, but the parent dotnet run process (1074) is still running and its (1074) diagnostics IPC is stalled.

I'll check the dotnet <path/to/dll> variants and see what happens.

jander-msft commented 3 years ago

Do you see the same behavior if you do dotnet <path/to/dll> & or dotnet exec </path/to/dll> &?

No, these seem to work correctly.

josalem commented 3 years ago

This smells like this old SDK/PAL issue: https://github.com/dotnet/sdk/issues/8050.

Let's take a look at the child process's call stack in the memory dump and see where it is in execution.

ghost commented 3 years ago

Tagging subscribers to this area: @tommcdon See info in area-owners.md if you want to be subscribed.

Issue Details
### Description When running `dotnet run` on a project as a background process (e.g. `dotnet run &`), the child process that is the application will be zombied and the diagnostics server in the `dotnet run` process is stalled. Repro steps: 1. mkdir TestApp 1. cd TestApp 1. dotnet new webapp 1. dotnet run & ``` dotnet run & [1] 1074 Building... ``` The expected bound addresses from the web application do not seem to be bound: ``` curl http://localhost:5000 curl: (7) Failed to connect to localhost port 5000: Connection refused ``` Output from `ps aux` (removed other processes): ``` ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jander 1074 3.5 0.5 3547660 66996 pts/0 Tl 08:53 0:00 dotnet run jander 1093 15.6 0.0 0 0 pts/0 Z 08:53 0:01 [dotnet] ``` ### Configuration .NET 5.0.6 Ubuntu 18.04 x64 on WSL 2 ``` dotnet --list-runtimes Microsoft.AspNetCore.All 2.1.28 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.28 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.15 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.28 [/usr/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.15 [/usr/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App] ``` ``` dotnet --list-sdks 2.1.816 [/usr/share/dotnet/sdk] 3.1.409 [/usr/share/dotnet/sdk] 5.0.300 [/usr/share/dotnet/sdk] ``` ### Regression? Unknown at this time, will attempt on prior .NET Core versions soon. ### Other information Using `nohup` to run it in the background (e.g. `nohup dotnet run &`) seems to work correctly. Running `dotnet run` as a foreground process works as expected. Attempting to use diagnostics tools to communicate with the diagnostics server in the `dotnet run` process has any of those diagnostics tools hang: Attempting to use dotnet-dump on the `dotnet run` process (the dotnet-dump process will run indefinitely until process 1074 is killed): ``` dotnet dump collect --type Full -p 1074 Writing full to /home/jander/core_20210610_094401 ``` Another example is using dotnet-monitor to enumerate processes on the system will hang indefinitely until process 1074 is killed: ``` dotnet monitor collect --no-auth ... curl -k https://localhost:52323/processes ``` call stack of dotnet-monitor that is trying to issue the ProcessInfo command over the diagnostics port: ``` System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle socket, System.Net.Sockets.SocketFlags flags, System.Span buffer, byte[] socketAddress, ref int socketAddressLen = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out Interop.Error errno = SUCCESS) System.Net.Sockets.dll!System.Net.Sockets.SocketPal.TryCompleteReceiveFrom(System.Net.Sockets.SafeSocketHandle socket, System.Span buffer, System.Collections.Generic.IList> buffers, System.Net.Sockets.SocketFlags flags, byte[] socketAddress, ref int socketAddressLen, out int bytesReceived = 0, out System.Net.Sockets.SocketFlags receivedFlags = 1761477460, out System.Net.Sockets.SocketError errorCode = Success) System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext.ReceiveFrom(System.Memory buffer, ref System.Net.Sockets.SocketFlags flags = None, byte[] socketAddress = null, ref int socketAddressLen = 0, int timeout = -1, out int bytesReceived = 0) System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle handle, byte[] buffer, int offset, int count, System.Net.Sockets.SocketFlags socketFlags, out int bytesTransferred) System.Net.Sockets.dll!System.Net.Sockets.Socket.Receive(byte[] buffer = {byte[14]}, int offset = 0, int size, System.Net.Sockets.SocketFlags socketFlags, out System.Net.Sockets.SocketError errorCode = 2116193192) System.Net.Sockets.dll!System.Net.Sockets.NetworkStream.Read(byte[] buffer, int offset, int size) System.Private.CoreLib.dll!System.IO.BinaryReader.ReadBytes(int count = 14) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcHeader.TryParse(System.IO.BinaryReader reader = {System.IO.BinaryReader}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(System.IO.Stream stream = {Microsoft.Diagnostics.NETCore.Client.ExposedSocketNetworkStream}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Microsoft.Diagnostics.NETCore.Client.IpcEndpoint endpoint = {Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint}, Microsoft.Diagnostics.NETCore.Client.IpcMessage message = {Microsoft.Diagnostics.NETCore.Client.IpcMessage}) Microsoft.Diagnostics.NETCore.Client.dll!Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfo() Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.EndpointInfo.FromProcessId(int processId = 1074) Microsoft.Diagnostics.Monitoring.dll!Microsoft.Diagnostics.Monitoring.ClientEndpointInfoSource.GetEndpointInfoAsync.AnonymousMethod__2() System.Private.CoreLib.dll!System.Threading.Tasks.Task.InnerInvoke() System.Private.CoreLib.dll!System.Threading.Tasks.Task..cctor.AnonymousMethod__274_0(object obj) System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread = {System.Threading.Thread}, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot = Id = 2, Status = Running, Method = "Microsoft.Diagnostics.Monitoring.EndpointInfo b__2()", Result = "{Not yet computed}", System.Threading.Thread threadPoolThread) System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread threadPoolThread) System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread threadPoolThread) System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() System.Private.CoreLib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() ``` I have core dumps of process 1074 (the `dotnet run` process), process 1142 (the `dotnet-monitor` process). Related https://github.com/dotnet/dotnet-monitor/issues/399 cc @josalem
Author: jander-msft
Assignees: -
Labels: `area-Diagnostics-coreclr`, `untriaged`
Milestone: -
josalem commented 3 years ago

@jander-msft let me know if you want some help investigating at the dumps.

jander-msft commented 3 years ago

Sync'd with @josalem. We found that this does not repro on MacOS X nor on Windows.

I tried this on Ubuntu 18.04 x64 in a virtual machine in Hyper-V and it does reproduce there too. So this seems to be at least an issue on Ubuntu, if not other Linux distros.

hoyosjs commented 3 years ago

@josalem Is this a 6.0 target issue? Can we repro this in 3.1/5.0 eventpipe?

josalem commented 3 years ago

@hoyosjs I don't believe this issue is due to EventPipe. I don't have a root cause, but this happens regardless of whether you configure EventPipe. I haven't been able to repro this on my machines, but Justin was able to under a Linux VM and container. We didn't check whether this happens for 3.1 or 5.0.

jander-msft commented 3 years ago

We didn't check whether this happens for 3.1 or 5.0.

I was able to repro this for both .NET Core 3.1 and .NET 5.0 (see "Regression?" in original description).

josalem commented 3 years ago

Ah good note, I missed that.

noahfalk commented 3 years ago

Given the number of bugs we have left in 6.0 I'm skeptical we'll get to this one in time. Given that the issue already exists in 3.1 and 5 if this is serious I assume we'll need a workaround elsewhere regardless.