search

dotnet / dotnet-monitor

This repository contains the source code for .NET Monitor - a tool that allows you to gather diagnostic data from running applications using HTTP endpoints
MIT License
642 stars 113 forks source link

Support for printing thread stack traces and heaps composition in dotnet-monitor #20

Open mleenhardt opened 4 years ago

mleenhardt commented 4 years ago

We wrote a much simpler version of dotnet monitor that uses Microsoft.Diagnostics.Runtime to help with high level debugging of remote processes running in Kubernetes. Our implementation returns a stack trace of all threads in a given process, as well as the type composition of all heaps, ordered by object size. This is very convenient to get a feel for what a process is doing without having to download a dump and load it in dotnet analyze.

e.g. In our implementation, http://localhost:52323/dump returns something that looks like

--------------------------------------------------------------------
    STACK TRACE
--------------------------------------------------------------------
Thread - OS 00015910 - Managed 1:
  25aaf7e070 7ffb6af86278 [GCFrame]
  25aaf7e1b8 7ffb6af86278 [HelperMethodFrame_1OBJ]
  25aaf7e2e0 7ffbc9a43725 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
  25aaf7e380 7ffbc9a43052 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
  25aaf7e3f0 7ffbc9a42dcf System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken)
  25aaf7e470 7ffbc9a61604 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
  25aaf7e4a0 7ffbc9a4432d System.Runtime.CompilerServices.TaskAwaiter.GetResult()
  25aaf7e4d0 7ffc275d6585 Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(Microsoft.Extensions.Hosting.IHost)
  25aaf7e500 7ffb6a5a1790 WebApplication2.Program.Main(System.String[])
  25aaf7e788 7ffbca0c6103 [GCFrame]
  25aaf7ed30 7ffbca0c6103 [GCFrame]

Thread - OS 00002a90 - Managed 2:
  25abbffa00 7ffb6af86278 [DebuggerU2MCatchHandlerFrame]

Thread - OS 00008898 - Managed 4:
  25ac37ed18 7ffb6af86278 [HelperMethodFrame]
  25ac37ee10 7ffbc9b2b91b System.Threading.Thread.Sleep(Int32)
  25ac37ee40 7ffb6a5a4535 WebApplication2.Program.DoWork()
  25ac37eeb0 7ffb6a5a40d1 WebApplication2.Program+<>c.<Main>b__0_0()
  25ac37eee0 7ffbc9a47bd3 System.Threading.Tasks.Task.InnerInvoke()
  25ac37ef20 7ffbc9a47b82 System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)
  25ac37ef50 7ffbc9a47b02 System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
  25ac37efa0 7ffbc9a477d7 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
  25ac37f040 7ffbc9a476b3 System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
  25ac37f080 7ffbc9a4764b System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread)
  25ac37f0b0 7ffbc9a4723c System.Threading.ThreadPoolWorkQueue.Dispatch()
  25ac37f160 7ffbc9a4706b System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
  25ac37f570 7ffbca0c6103 [DebuggerU2MCatchHandlerFrame]

[...]

--------------------------------------------------------------------
    HEAPS
--------------------------------------------------------------------

--------------------------------------------------------------------
GENERATION 0
System.String - 764954
System.Byte[] - 735445
System.Security.Cryptography.Oid - 735360
System.Char[] - 419616
System.Security.Cryptography.X509Certificates.X509Extension[] - 201128
System.Security.Cryptography.X509Certificates.X509Extension - 193440
[...]

--------------------------------------------------------------------
GENERATION 1
Free - 288

--------------------------------------------------------------------
GENERATION 2
Free - 768

Would it be feasible or would you be interested in supporting a similar feature? Maybe in two different endpoints /stacktrace and /heap?

rwkarg commented 4 years ago

A diagnostics summary like this would be nice to have available without needing to work with a dump or worry about the target architecture.

rwkarg commented 4 years ago

For an example use cases that would be interesting to me:

We will occasionally have (third-party, of course) dependencies that have un-synchronized access to a non-thread safe collection (like a Dictionary<K, V>). If we see three cores pegged on the process and thread stacks show three threads with a Dictionary<,>.Insert stack trace then we can easily diagnose where that is happening and push a fix to the upstream.

shirhatti commented 4 years ago

cc @josalem who was writing a similar tool a while ago- https://github.com/josalem/DotStack

davidfowl commented 4 years ago

I like this feature but I'd like to return a JSON representation of the stacks not a textual one so that we could build a nice UI on top (someday) 😄

rwkarg commented 4 years ago

I like this feature but I'd like to return a JSON representation of the stacks not a textual one so that we could build a nice UI on top (someday) 😄

Accept ftw

sukesh-ak commented 4 years ago

I could not get the sample by @josalem working, I guess due to change in API with unsupported format error etc. So I put together a sample with EventPipe trace code and stack related code borrowed from his sample. https://github.com/sukesh-ak/EventPipe-Diagnostics

Do let me know if you have any suggestions.

jander-msft commented 2 years ago

FYI, we've added a /stacks route to dotnet monitor in 7.0 RC 1 as an experimental feature:

At the release of 7.0, it will remain an experimental feature.

We are continuing to invest in this feature and hope to fully support it in a post-7.0 release. Please give it a try and let us know about any feedback you have for it!