dotnet monitor 6 - error during full /dump

[DevOnBike]

Description

during execution of /dump I get http 500 error.

Here is stack trace gathered from console: {"Timestamp":"2023-03-07T08:34:23.5112371Z","EventId":13,"LogLevel":"Error","Category":"Microsoft.AspNetCore.Server.Kestrel","Message":"Connection id \u00220HMOUSPLECPTL\u0022, Request id \u00220HMOUSPLECPTL:00000003\u0022: An unhandled exception was thrown by the application.","Exception":"System.IO.EndOfStreamException: Attempted to read past the end of the stream. at Microsoft.Diagnostics.NETCore.Client.StreamExtensions.ReadBytesAsync(Stream stream, Int32 length, CancellationToken cancellationToken) at Microsoft.Diagnostics.NETCore.Client.IpcHeader.ParseAsync(Stream stream, CancellationToken cancellationToken) at Microsoft.Diagnostics.NETCore.Client.IpcMessage.ParseAsync(Stream stream, CancellationToken cancellationToken) at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuationAsync(IpcEndpoint endpoint, IpcMessage message, CancellationToken cancellationToken) at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageAsync(IpcEndpoint endpoint, IpcMessage message, CancellationToken cancellationToken) at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDumpAsync(DumpType dumpType, String dumpPath, Boolean logDumpGeneration, CancellationToken token) at Microsoft.Diagnostics.Monitoring.WebApi.DumpService.DumpAsync(IEndpointInfo endpointInfo, DumpType mode, CancellationToken token) in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/DumpService.cs:line 77 at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003EcDisplayClass15_0.\u003C\u003CCaptureDump\u003Eb_0\u003Ed.MoveNext() in //src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 216 --- End of stack trace from previous location --- at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003EcDisplayClass31_0.\u003C\u003CInvokeForProcess\u003Eb0\u003Ed.MoveNext() in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 680 --- End of stack trace from previous location --- at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003Ec__DisplayClass33_0\u00601.\u003C\u003CInvokeForProcess\u003Eb0\u003Ed.MoveNext() in //src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 712 --- End of stack trace from previous location --- at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagControllerExtensions.InvokeService[T](ControllerBase controller, Func\u00601 serviceCall, ILogger logger) in //src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagControllerExtensions.cs:line 57 at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.InvokeForProcess[T](Func\u00602 func, Nullable\u00601 processKey, String artifactType) in //src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 702 at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.InvokeForProcess(Func\u00602 func, Nullable\u00601 processKey, String artifactType) in //src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 680 at lambda_method13(Closure , Object ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003EgAwaited|12_0(ControllerActionInvoker invoker, ValueTask\u00601 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeInnerFilterAsync\u003EgAwaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextExceptionFilterAsync\u003EgAwaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ExceptionContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeFilterPipelineAsync\u003Eg__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003EgAwaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.\u003CInvoke\u003Eg__AwaitRequestTask|60(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.Diagnostics.Tools.Monitor.RequestLimitMiddleware.Invoke(HttpContext context) in //src/Tools/dotnet-monitor/RequestLimitMiddleware.cs:line 67 at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)","State":{"Message":"Connection id \u00220HMOUSPLECPTL\u0022, Request id \u00220HMOUSPLECPTL:00000003\u0022: An unhandled exception was thrown by the application.","ConnectionId":"0HMOUSPLECPTL","TraceIdentifier":"0HMOUSPLECPTL:00000003","{OriginalFormat}":"Connection id \u0022{ConnectionId}\u0022, Request id \u0022{TraceIdentifier}\u0022: An unhandled exception was thrown by the application."},"Scopes":[{"Message":"SpanId:48e4111b0282f30c, TraceId:9a7953d4cfefa2d35816ddf38b0a7d4a, ParentId:0000000000000000","SpanId":"48e4111b0282f30c","TraceId":"9a7953d4cfefa2d35816ddf38b0a7d4a","ParentId":"0000000000000000"},{"Message":"ConnectionId:0HMOUSPLECPTL","ConnectionId":"0HMOUSPLECPTL"},{"Message":"RequestPath:/dump RequestId:0HMOUSPLECPTL:00000003","RequestId":"0HMOUSPLECPTL:00000003","RequestPath":"/dump"}]} {"Timestamp":"2023-03-07T08:34:25.5003267Z","EventId":52,"LogLevel":"Warning","Category":"Microsoft.Diagnostics.Tools.Monitor.ServerEndpointInfoSource","Message":"Unexpected timeout from process 9. Process will no longer be monitored.","State":{"Message":"Unexpected timeout from process 9. Process will no longer be monitored.","processId":"9","{OriginalFormat}":"Unexpected timeout from process {processId}. Process will no longer be monitored."},"Scopes":[]} {"Timestamp":"2023-03-07T08:34:25.5066625Z","EventId":44,"LogLevel":"Information","Category":"Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleService","Message":"Stopping collection rules.","State":{"Message":"Stopping collection rules.","{OriginalFormat}":"Stopping collection rules."},"Scopes":[{"Message":"TargetProcessId:9 TargetRuntimeInstanceCookie:4d514cb6c0564f7fa965ae249feb13ba","TargetProcessId":"9","TargetRuntimeInstanceCookie":"4d514cb6c0564f7fa965ae249feb13ba"}]} {"Timestamp":"2023-03-07T08:34:25.5079624Z","EventId":45,"LogLevel":"Information","Category":"Microsoft.Diagnostics.Tools.Monitor.CollectionRules.CollectionRuleService","Message":"All collection rules have stopped.","State":{"Message":"All collection rules have stopped.","{OriginalFormat}":"All collection rules have stopped."},"Scopes":[{"Message":"TargetProcessId:9 TargetRuntimeInstanceCookie:4d514cb6c0564f7fa965ae249feb13ba","TargetProcessId":"9","TargetRuntimeInstanceCookie":"4d514cb6c0564f7fa965ae249feb13ba"}]}

Configuration

dotnet monitor is running as a sidecar in kubernetes cluster. other endpoints are generally running and working properly (not giving errors). /dump endpoint is working for type=mini but full dump is problematic. dotnet monitor has given resource limit for memory = 2Gi monitored dotnet service has limit of 1Gi of memory.

Regression?

don't know, only this version I'm using

Other information

.net 6 app and dotnet monitor running with m$ images

{"version":"6.0.2+9774a8ccc1a015d47960ed76de1bc6a5a1227f2c","runtimeVersion":"6.0.5","diagnosticPortMode":"Listen","diagnosticPortName":"/etc/dotnet-monitor/dotnet-monitor.sock"}

[github-actions[bot]]

Welcome to dotnet-monitor!

Thanks for creating your first issue; let us know what you think of dotnet-monitor by filling out our survey.

[jander-msft]

.NET Monitor 6.0.2 is long out of date and unsupported. Please update to a newer version. Recommended tags to stay on the latest version within each major version would be mcr.microsoft.com/dotnet/monitor:6 or mcr.microsoft.com/dotnet/monitor:7.

[DevOnBike]

.NET Monitor 6.0.2 is long out of date and unsupported. Please update to a newer version. Recommended tags to stay on the latest version within each major version would be mcr.microsoft.com/dotnet/monitor:6 or mcr.microsoft.com/dotnet/monitor:7.

in 6.3.2+68aa7e8f1d65aa702a75525176bb72f2b50c1bba it is working fine