mikelle-rogers
commented
12 months ago Hi @szmcdull, what kind of computer are you working on?
Closed szmcdull closed 9 months ago
mikelle-rogers
commented
12 months ago Hi @szmcdull, what kind of computer are you working on?
szmcdull
commented
12 months ago Hi @szmcdull, what kind of computer are you working on?
It is a cloud server running CentOS linux
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"model name : Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
tommcdon
commented
11 months ago Hi @szmcdull! Thanks for all of the information! The dotnet-gcdump/dotnet-dump tools connect to the target process using a diagnostics IPC channel which is implemented as a Linux domain socket. At app startup, the runtime creates a domain socket in /tmp (or $TMPDIR if that environment variable is set) with the name dotnet-diagnostic-{pid}-{disambiguation_key}-socket (for example dotnet-diagnostic-416-19941-socket). The domain socket is created with srw------ permissions meaning only the owner of the socket (or root) has access. When the dotnet- tool is run it looks for the {temp}/dotnet-diagnostic-{pid}-{disambiguation_key}-socket file and tries to connect to it. The error we are seeing means that the dotnet- tool was not able to open that file.
So the likely explanation of the error is that the target process is running as a different user than the dotnet-gcdump/dotnet-dump tool. One workaround is to launch dotnet-gcdump/dotnet-dump as the same user, or as root by running it with sudo, for example sudo ~/.dotnet/tools/dotnet-dump <cmd>.
I apologize as the error message is not a very good one and should be addressed by https://github.com/dotnet/diagnostics/pull/4406 in the next release.
Please let us know if this helps.
szmcdull
commented
10 months ago Hi @szmcdull! Thanks for all of the information! The
dotnet-gcdump/dotnet-dumptools connect to the target process using a diagnostics IPC channel which is implemented as a Linux domain socket. At app startup, the runtime creates a domain socket in/tmp(or$TMPDIRif that environment variable is set) with the namedotnet-diagnostic-{pid}-{disambiguation_key}-socket(for exampledotnet-diagnostic-416-19941-socket). The domain socket is created withsrw------permissions meaning only the owner of the socket (or root) has access. When the dotnet- tool is run it looks for the{temp}/dotnet-diagnostic-{pid}-{disambiguation_key}-socketfile and tries to connect to it. The error we are seeing means that the dotnet- tool was not able to open that file.So the likely explanation of the error is that the target process is running as a different user than the
dotnet-gcdump/dotnet-dumptool. One workaround is to launchdotnet-gcdump/dotnet-dumpas the same user, or as root by running it withsudo, for examplesudo ~/.dotnet/tools/dotnet-dump <cmd>.I apologize as the error message is not a very good one and should be addressed by #4406 in the next release.
Please let us know if this helps.
@tommcdon I was running as the same user. All the processes are launched by me and I have only one account.
Skyppid
commented
10 months ago We're having the same issue with dotnet dump and dotnet-counters. SDK, Tool, Runtime all on latest .NET 8, App as well as the tool all run as root yet we get this error. It's a real bummer. We have severe issues I need to trace with these tools and can't get any data from our production cluster.
szmcdull
commented
10 months ago Today i was using dotnet-stack on the server and the same kind of error occurred
Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Process 13585 not running compatible .NET runtime.
at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.GetDefaultAddress(Int32 pid) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 332
at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.GetDefaultAddress() in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 265
at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.Connect(TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 241
at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 40
at Microsoft.Diagnostics.NETCore.Client.EventPipeSession.Start(IpcEndpoint endpoint, IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/EventPipeSession.cs:line 34
at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.StartEventPipeSession(IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 71
at Microsoft.Diagnostics.Tools.Stack.ReportCommandHandler.Report(CancellationToken ct, IConsole console, Int32 processId, String name, TimeSpan duration) in /_/src/Tools/dotnet-stack/ReportCommand.cs:line 81It was working good before. dotnet-stack version is 7.0.447801+d951821532fe44f5cbafbc339e5906592d6a5b36.
The program was compiled on Windows, using dotnet 8.0.100. The program project file is using net7.0 (<TargetFramework>net7.0</TargetFramework>)
szmcdull
commented
10 months ago Do these tools depend on createdump?
hoyosjs
commented
10 months ago @szmcdull sorry, wasn't around for the holidays. Yes, dotnet-dump's functionality relies on createdump. That being said - stacks doesn't depend on it. Can you please check two things:
1 - Does $TMPDIR/dotnet-diagnostics-<pid>-<number>-socket or /tmp/dotnet-diagnostics-<pid>-<number>-socket exists?
2 - If they don't - can you please check the environment of the target process? Is it possible DOTNET_EnableDiagnostics=0 is somewhere?
Additionally, is this process started in any special way (systemd for example) or does it use capabilities?
szmcdull
commented
10 months ago Currently I cannot find a process that is not working with the tools. Will check again if one is found
hoyosjs
commented
9 months ago @szmcdull gotcha. Sorry that it took a while to respond. I'll close this for now. In case you get this, feel free to open again and loop me directly, and thanks for the feedback.
Description
gcdump/dump: Process not running compatible .NET runtime
ps pick the last process in the above list:
pid 29080 is running exactly the same program, but it is not listed by dotnet-gcdump ps:
Configuration
$ dotnet --version 7.0.403 $ dotnet dotnet-gcdump --version 8.0.452401+966acd12b91675a4d06a7572ff47c587f827beaf $ dotnet-dump --version 8.0.452401+966acd12b91675a4d06a7572ff47c587f827beaf
Regression?
Other information