Closed Techtonictools closed 4 years ago
oot@debian:~/.dotnet/tools# dotnet tool update dotnet-counters --global Tool 'dotnet-counters' was successfully updated from version '3.1.120604' to version '3.1.135903'. root@debian:~/.dotnet/tools#
Still having the problem after updating the dotnet-counters tool from version '3.1.120604' to version '3.1.135903'. Please advise. root@debian:~/.dotnet/tools# ./dotnet-counters monitor -p 12618
`Press p to pause, r to resume, q to quit.
Status: Waiting for initial payload...
Failed to start the counter session: Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Process 12618 not running compatible .NET Core runtime.
at Microsoft.Diagnostics.NETCore.Client.IpcClient.GetTransport(Int32 processId) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 63
at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Int32 processId, IpcMessage message, IpcMessage& response) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 104
at Microsoft.Diagnostics.NETCore.Client.EventPipeSession..ctor(Int32 processId, IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/EventPipeSession.cs:line 30
at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.StartEventPipeSession(IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 39
at Microsoft.Diagnostics.Tools.Counters.CounterMonitor.<>c__DisplayClass17_0.<Start>b__1() in /_/src/Tools/dotnet-counters/CounterMonitor.cs:line 239
root@debian:~/.dotnet/tools#`
@Techtonictools Can you check if the target process is actually running? If it is, it would help if you can copy-paste the output from ls -l /tmp | grep dotnet-diagnostic-
, along with the pid of the process you're trying to monitor.
The PID is 12618 as shown by ps. top also shows it. The dotnet tool dotnet-trace doesn't see the process, it did before the VS upgrade and re-deploy. I didn't see any output from the ls to grep pipe command line so the directory listing of /tmp is shown.
`root@debian:/Filepath# ps -U TheUser
PID TTY TIME CMD
851 ? 00:00:00 systemd
852 ? 00:00:00 (sd-pam)
1072 ? 00:00:00 sshd
1073 pts/1 00:00:00 bash
1081 pts/1 00:00:00 su
11781 ? 00:00:00 sshd
11782 pts/2 00:00:00 bash
11785 pts/2 00:00:00 su
12618 ? 02:13:14 TheProgram
root@debian:/Filepath# ls -l /tmp | grep dotnet-diagnostic-,
root@debian:/Filepath# /root/.dotnet/tools/dotnet-trace ps
root@debian:/Filepath#
root@debian:/Filepath# ls /tmp
total 60K
drwxrwxrwt 15 root root 4.0K Jul 12 17:21 .
drwxr-xr-x 20 root root 4.0K Jun 26 16:29 ..
drwxr-xr-x 2 root root 4.0K Jul 12 16:29 3xyqbti4.5yi
drwxrwxrwt 2 root root 4.0K Jul 12 09:33 .font-unix
drwxrwxrwt 2 root root 4.0K Jul 12 09:33 .ICE-unix
drwxr-xr-x 2 root root 4.0K Jul 12 16:29 igqhsu1g.wjy
drwxrwxrwx 3 root root 4.0K Jul 12 16:27 NuGetScratch
drwxr-xr-x 2 root root 4.0K Jul 12 16:54 system-commandline-sentinel-files
drwx------ 3 root root 4.0K Jul 12 09:33 systemd-private-8e8c0284bfbb44f8a8b0635a3339c9f0-apache2.service-ZM4qyV
drwx------ 3 root root 4.0K Jul 12 09:33 systemd-private-8e8c0284bfbb44f8a8b0635a3339c9f0-systemd-timesyncd.service-mvAFxM
drwxrwxrwt 2 root root 4.0K Jul 12 09:33 .Test-unix
drwx------ 2 root root 4.0K Jul 12 09:34 vmware-root_462-826451927
drwxrwxrwt 2 root root 4.0K Jul 12 09:33 .X11-unix
drwxrwxrwt 2 root root 4.0K Jul 12 09:33 .XIM-unix
drwxr-xr-x 2 root root 4.0K Jul 12 16:27 z0dp4ct2.kas
root@debian:/Filepath#`
That's interesting... So the initial log that you posted in the issue is different from the error you're seeing after you upgraded the runtime. The first log you posted seems to suggest that the app has been shut down after running it because it did print few counter values until it shut down (it's not supposed to show the exceptions as it's an expected exception if the target app exits before dotnet-counters exits - that has been fixed in https://github.com/dotnet/diagnostics/pull/1322 and will be released with next release of dotnet-counters)
The second log (the one you posted after updating the tool) is suggesting that the pipe that all diagnostic tools use to attach to the target process is not created or deleted after it's been created. (We recently were notified of a very similar error in https://github.com/dotnet/diagnostics/issues/1318). The tool does not delete the pipe, so updating the version of the tool shouldn't have caused this behavior.
Do you happen to know if the target app is responsive? (i.e.. if it's a web app, does it still respond to requests?) Also, since it seems like you have root access to the host, would it be possible for you to attach a lldb and get a stack trace of the target app? Thanks for reporting this issue and helping us investigate the problem!
It is the same instance of TheProgram running in both cases. In the original post, the tool launched, the screen was flashing, I came back to it and saw the output as pasted above. The TheProgram started at about 1PM and is running actively progressing loading a database. Looking at the lldb dump, it looks like stop was called. It isn't clear how that happened. Maybe systemd is trying to kill it for using up too much resources. The system is very low on resources. VS won't connect to it now either. Feel free to close the issue.
`root@debian:~/.dotnet/tools# lldb process attach -p 12618
Warning: program arguments are ignored when attaching.
(lldb) process attach --pid 12618
Process 12618 stopped
* thread #1, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b700c libpthread.so.0`__pthread_cond_wait + 508
libpthread.so.0`__pthread_cond_wait:
-> 0x7f6cd69b700c <+508>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b7012 <+514>: ja 0x7f6cd69b7068 ; <+600>
0x7f6cd69b7014 <+516>: movl %r8d, %edi
0x7f6cd69b7017 <+519>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #2, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd65b0819 libc.so.6`__poll + 73
libc.so.6`__poll:
-> 0x7f6cd65b0819 <+73>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd65b081f <+79>: ja 0x7f6cd65b0852 ; <+130>
0x7f6cd65b0821 <+81>: movl %r8d, %edi
0x7f6cd65b0824 <+84>: movl %eax, 0xc(%rsp)
thread #3, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69bad0e libpthread.so.0`__libc_open64 + 206
libpthread.so.0`__libc_open64:
-> 0x7f6cd69bad0e <+206>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69bad14 <+212>: ja 0x7f6cd69bad46 ; <+262>
0x7f6cd69bad16 <+214>: movl %r8d, %edi
0x7f6cd69bad19 <+217>: movl %eax, 0x8(%rsp)
thread #4, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b700c libpthread.so.0`__pthread_cond_wait + 508
libpthread.so.0`__pthread_cond_wait:
-> 0x7f6cd69b700c <+508>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b7012 <+514>: ja 0x7f6cd69b7068 ; <+600>
0x7f6cd69b7014 <+516>: movl %r8d, %edi
0x7f6cd69b7017 <+519>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #5, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b73f9 libpthread.so.0`__pthread_cond_timedwait + 761
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7f6cd69b73f9 <+761>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b73ff <+767>: ja 0x7f6cd69b7503 ; <+1027>
0x7f6cd69b7405 <+773>: movl %r8d, %edi
0x7f6cd69b7408 <+776>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #6, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69ba667 libpthread.so.0`__libc_accept + 71
libpthread.so.0`__libc_accept:
-> 0x7f6cd69ba667 <+71>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69ba66d <+77>: ja 0x7f6cd69ba6a2 ; <+130>
0x7f6cd69ba66f <+79>: movl %r8d, %edi
0x7f6cd69ba672 <+82>: movl %eax, 0xc(%rsp)
thread #7, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b73f9 libpthread.so.0`__pthread_cond_timedwait + 761
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7f6cd69b73f9 <+761>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b73ff <+767>: ja 0x7f6cd69b7503 ; <+1027>
0x7f6cd69b7405 <+773>: movl %r8d, %edi
0x7f6cd69b7408 <+776>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #8, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b700c libpthread.so.0`__pthread_cond_wait + 508
libpthread.so.0`__pthread_cond_wait:
-> 0x7f6cd69b700c <+508>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b7012 <+514>: ja 0x7f6cd69b7068 ; <+600>
0x7f6cd69b7014 <+516>: movl %r8d, %edi
0x7f6cd69b7017 <+519>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #9, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69ba544 libpthread.so.0`__libc_read + 68
libpthread.so.0`__libc_read:
-> 0x7f6cd69ba544 <+68>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69ba54a <+74>: ja 0x7f6cd69ba584 ; <+132>
0x7f6cd69ba54c <+76>: movl %r8d, %edi
0x7f6cd69ba54f <+79>: movq %rax, 0x8(%rsp)
thread #10, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd65bb7ef libc.so.6`epoll_wait + 79
libc.so.6`epoll_wait:
-> 0x7f6cd65bb7ef <+79>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd65bb7f5 <+85>: ja 0x7f6cd65bb82a ; <+138>
0x7f6cd65bb7f7 <+87>: movl %r8d, %edi
0x7f6cd65bb7fa <+90>: movl %eax, 0xc(%rsp)
thread #11, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b700c libpthread.so.0`__pthread_cond_wait + 508
libpthread.so.0`__pthread_cond_wait:
-> 0x7f6cd69b700c <+508>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b7012 <+514>: ja 0x7f6cd69b7068 ; <+600>
0x7f6cd69b7014 <+516>: movl %r8d, %edi
0x7f6cd69b7017 <+519>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #12, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69ba544 libpthread.so.0`__libc_read + 68
libpthread.so.0`__libc_read:
-> 0x7f6cd69ba544 <+68>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69ba54a <+74>: ja 0x7f6cd69ba584 ; <+132>
0x7f6cd69ba54c <+76>: movl %r8d, %edi
0x7f6cd69ba54f <+79>: movq %rax, 0x8(%rsp)
thread #13, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69ba544 libpthread.so.0`__libc_read + 68
libpthread.so.0`__libc_read:
-> 0x7f6cd69ba544 <+68>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69ba54a <+74>: ja 0x7f6cd69ba584 ; <+132>
0x7f6cd69ba54c <+76>: movl %r8d, %edi
0x7f6cd69ba54f <+79>: movq %rax, 0x8(%rsp)
thread #14, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b700c libpthread.so.0`__pthread_cond_wait + 508
libpthread.so.0`__pthread_cond_wait:
-> 0x7f6cd69b700c <+508>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b7012 <+514>: ja 0x7f6cd69b7068 ; <+600>
0x7f6cd69b7014 <+516>: movl %r8d, %edi
0x7f6cd69b7017 <+519>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #15, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b73f9 libpthread.so.0`__pthread_cond_timedwait + 761
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7f6cd69b73f9 <+761>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b73ff <+767>: ja 0x7f6cd69b7503 ; <+1027>
0x7f6cd69b7405 <+773>: movl %r8d, %edi
0x7f6cd69b7408 <+776>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #16, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b73f9 libpthread.so.0`__pthread_cond_timedwait + 761
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7f6cd69b73f9 <+761>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b73ff <+767>: ja 0x7f6cd69b7503 ; <+1027>
0x7f6cd69b7405 <+773>: movl %r8d, %edi
0x7f6cd69b7408 <+776>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #17, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6c5f9d6804
-> 0x7f6c5f9d6804: addq $0x8, %rsp
0x7f6c5f9d6808: jmpq *%rax
0x7f6c5f9d680b: addb %bl, (%rcx)
0x7f6c5f9d680d: addl %eax, (%rcx)
thread #18, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b73f9 libpthread.so.0`__pthread_cond_timedwait + 761
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7f6cd69b73f9 <+761>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b73ff <+767>: ja 0x7f6cd69b7503 ; <+1027>
0x7f6cd69b7405 <+773>: movl %r8d, %edi
0x7f6cd69b7408 <+776>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
thread #19, name = 'TheProgram', stop reason = signal SIGSTOP
frame #0: 0x00007f6cd69b73f9 libpthread.so.0`__pthread_cond_timedwait + 761
libpthread.so.0`__pthread_cond_timedwait:
-> 0x7f6cd69b73f9 <+761>: cmpq $-0x1000, %rax ; imm = 0xF000
0x7f6cd69b73ff <+767>: ja 0x7f6cd69b7503 ; <+1027>
0x7f6cd69b7405 <+773>: movl %r8d, %edi
0x7f6cd69b7408 <+776>: callq 0x7f6cd69ba1f0 ; __pthread_disable_asynccancel
Executable module set to "/TheFIlepath".
Architecture set to: x86_64-pc-linux-gnu.
(lldb)`
The ETL program finished its load so I issued a systemctl stop TheProgram. No issues stopping. I restarted it and dotnet-counters is working now.
BTW, I appreciate the quick response.
Thanks for confirming that. Please feel free to reopen if you see the issue happening again.
I upgraded from Visual Studio Version 16.6.1 to 16.6.3, rebuilt my program, re-deployed it to a Debian 10.4 x64 machine and ./dotnet-counters monitor doesn't run because of an incompatibility. I don't know how to resolve this. Both the build machine and host have the same runtime version of Microsoft.NETCore.App 3.1.5. What am I doing wrong?
` Press p to pause, r to resume, q to quit. Status: Running
Installed runtimes:
apt shows the runtime is up-to-date
On Windows build machine:
` Sun 07/12/2020 16:03:15.79=> d:\Filepath dotnet --list-runtimes Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
The project cs file:
`