Question: dotnet-dump doesn't show the running dotnet process

msallin commented 3 years ago

We run integration tests based on Microsofft.AspNetCore.Mvc.Testing on our build server using dotnet test & XUnit. For some reason those tests deadlock(?) and so far we aren't able to reproduce it locally nor do we have a hint what the problem could be.

Thus we want to get a Dump from the build server. The build server is Jenkins respectively a Jenkins slave (Docker container with Red Hat Enterprise Linux Server 7.9). We are able to install dotnet-dump and we can successfully execute it. Afterwards we start the tests and wait until they deadlock. We see the process "dotnet test" on the slave but "dotnet-dump ps" does not return any processes.

The command "dotnet-dump ps" is executed with root privileges. What else could be the problem that the processes don't show up?

tommcdon commented 3 years ago

@mikem8361

hoyosjs commented 3 years ago

Do you know if the Jenkins container might be running it as a process with a private temp? all tools depend on both the dotnet-dump process and the target app to have the same temp directory on unix based systems. Do you have any way to get the app run by jenkins to print the temp path and the files in it? This could help narrow this down.

msallin commented 3 years ago

I'm not a Linux guys at all, so please take my statements with the necessary grain of salt.

Do you know if the Jenkins container might be running it as a process with a private temp?

There is no special setting. The container should use the default directory "/tmp" as temp dir, so should the processes running in it. Is there a way to see what the temp dir of "dotnet dump" is?

Do you have any way to get the app run by jenkins to print the temp path and the files in it?

The "app" is actually "dotnet test". But it runs a bunch of unit tests. I added the following code to one of the tests. Does it show what it should?

Console.WriteLine("Temp");
Console.WriteLine(Path.GetTempPath());
foreach (var tempContent in Directory.GetFileSystemEntries(Path.GetTempPath()))
{
    Console.WriteLine(tempContent);
}

The output is (replaced some things with *): 09:52:41 Temp 09:52:41 /tmp/ 09:52:41 /tmp/yum.log 09:52:41 /tmp/.Test-unix 09:52:41 /tmp/.X11-unix 09:52:41 /tmp/.XIM-unix 09:52:41 /tmp/.font-unix 09:52:41 /tmp/.ICE-unix 09:52:41 /tmp/ks-script-4sTfs_ 09:52:41 /tmp/MSBuild2268 09:52:41 /tmp/axxah5jd.kxa 09:52:41 /tmp/hsperfdata_baseuser 09:52:41 /tmp/resource-9215640403504503653 09:52:41 /tmp/resource-7549528422762870718 09:52:41 /tmp/resource-95182614806776500 09:52:41 /tmp/5jdd3w5j.eon 09:52:41 /tmp/qqo4prnu.2th 09:52:41 /tmp/tkqbsdkh.ra2 09:52:41 /tmp/djlydbtc.pnt 09:52:41 /tmp/.sonarqube 09:52:41 /tmp/el2s5mjb.kje 09:52:41 /tmp/MSBuild127565 09:52:41 /tmp/.X20-lock 09:52:41 /tmp/MSBuild10499 09:52:41 /tmp/MSBuild8787 09:52:41 /tmp/z1pfmavx.ecf 09:52:41 /tmp/MSBuild8796 09:52:41 /tmp/MSBuild8739 09:52:41 /tmp/MSBuild8754 09:52:41 /tmp/MSBuild8724 09:52:41 /tmp/MSBuild95032 09:52:41 /tmp/MSBuild95014 09:52:41 /tmp/mbdlsedg.4dy 09:52:41 /tmp/odz5xa3h.2cy 09:52:41 /tmp/e05iszcl.3eg 09:52:41 /tmp/mgzjlvp5.azx 09:52:41 /tmp/MSBuild10442 09:52:41 /tmp/MSBuild25041 09:52:41 /tmp/n5lhckfd.3tg 09:52:41 /tmp/MSBuild30281 09:52:41 /tmp/MSBuild30263 09:52:41 /tmp/MSBuild2296 09:52:41 /tmp/MSBuild26278 09:52:41 /tmp/MSBuild12554 09:52:41 /tmp/wkod1zrl.kr0 09:52:41 /tmp/MSBuild26318 09:52:41 /tmp/MSBuild12525 09:52:41 /tmp/MSBuild30298 09:52:41 /tmp/khuvk5xv.wxj 09:52:41 /tmp/MSBuild28768 09:52:41 /tmp/sope3wtp.qxu 09:52:41 /tmp/MSBuild26289 09:52:41 /tmp/MSBuild44600 09:52:41 /tmp/auek4hs1.s1n 09:52:41 /tmp/MSBuild26332 09:52:41 /tmp/NuGetScratch 09:52:41 /tmp/MSBuild17694 09:52:41 /tmp/.dotnet 09:52:41 /tmp/MSBuild44640 09:52:41 /tmp/MSBuild26304 09:52:41 /tmp/MSBuild25061 09:52:41 /tmp/MSBuild25010 09:52:41 /tmp/ggbq31q2.ur0 09:52:41 /tmp/qrxt3syp.0bk 09:52:41 /tmp/MSBuild35985 09:52:41 /tmp/MSBuild10484 09:52:41 /tmp/VBCSCompiler 09:52:41 /tmp/MSBuild36046 09:52:41 /tmp/MSBuild10470 09:52:41 /tmp/MSBuild26345 09:52:41 /tmp/MSBuild44674 09:52:41 /tmp/MSBuild17734 09:52:41 /tmp/NuGet 09:52:41 /tmp/MSBuild2282 09:52:41 /tmp/MSBuild17704 09:52:41 /tmp/MSBuild7679 09:52:41 /tmp/MSBuild7648 09:52:41 /tmp/MSBuild7697 09:52:41 /tmp/MSBuild7663 09:52:41 /tmp/phantomjs 09:52:41 /tmp/nw0akuee.yio 09:52:41 /tmp/o1qhSM+jVbMnsMYitcjDorrYUtA6VKeBosNetKJzQk4 09:52:41 /tmp/MSBuild127512 09:52:41 /tmp/jvcnvwgp.szy 09:52:41 /tmp/MSBuild2240 09:52:41 /tmp/MSBuild17760 09:52:41 /tmp/uaa2phza.j1t 09:52:41 /tmp/ihsio1ma.e0x 09:52:41 /tmp/MSBuild28784 09:52:41 /tmp/MSBuild44622 09:52:41 /tmp/azaryekb.2wd 09:52:41 /tmp/MSBuild35938 09:52:41 /tmp/MSBuild10432 09:52:41 /tmp/MSBuild34938 09:52:41 /tmp/MSBuild94984 09:52:41 /tmp/lqqsl0io.rvm 09:52:41 /tmp/dugsiohv.kiv 09:52:41 /tmp/MSBuild36002 09:52:41 /tmp/MSBuild35951 09:52:41 /tmp/tmpnTAMbM.tmp 09:52:41 /tmp/MSBuild30318 09:52:41 /tmp/MSBuild17745 09:52:41 /tmp/tmpEytdXc.tmp 09:52:41 /tmp/tepfstmx.ejx 09:52:41 /tmp/r5ewaptw.qz4 09:52:41 /tmp/MSBuild107753 09:52:41 /tmp/0x1re3xu.wuu 09:52:41 /tmp/MSBuild127549 09:52:41 /tmp/u5i15war.0s1 09:52:41 /tmp/nqvy5kea.cnp 09:52:41 /tmp/MSBuild35922 09:52:41 /tmp/MSBuild36031 09:52:41 /tmp/tmp2737Km.tmp 09:52:41 /tmp/MSBuild34882 09:52:41 /tmp/.Nes.Imports.Service_707ede43-918b-44a1-8bcf-4267f08653bc 09:52:41 /tmp/MSBuild34872 09:52:41 /tmp/MSBuild8772 09:52:41 /tmp/MSBuild8714 09:52:41 /tmp/MSBuild10456 09:52:41 /tmp/MSBuild34926 09:52:41 /tmp/MSBuild30245 09:52:41 /tmp/MSBuild34910 09:52:41 /tmp/q1vodhyd.cyf 09:52:41 /tmp/MSBuild44590 09:52:41 /tmp/MSBuild107762 09:52:41 /tmp/MSBuild107805 09:52:41 /tmp/MSBuild127580 09:52:41 /tmp/tt0qof42.qeo 09:52:41 /tmp/MSBuild2254 09:52:41 /tmp/MSBuild17716 09:52:41 /tmp/MSBuild12539 09:52:41 /tmp/MSBuild25025 09:52:41 /tmp/MSBuild107790 09:52:41 /tmp/3o3wbw5l.cnu 09:52:41 /tmp/MSBuild127536 09:52:41 /tmp/xaggd4lr.dkw 09:52:41 /tmp/clrebt5e.fv5 09:52:41 /tmp/MSBuild30233 09:52:41 /tmp/MSBuild44575 09:52:41 /tmp/jest_rs 09:52:41 /tmp/MSBuild94996 09:52:41 /tmp/jhs0uvx3.cfy 09:52:41 /tmp/4t3kdd3c.vls 09:52:41 /tmp/g0zwmccy.oid 09:52:41 /tmp/h05jzytw.e1q 09:52:41 /tmp/MSBuild107777 09:52:41 /tmp/MSBuild107817 09:52:41 /tmp/MSBuild127524 09:52:41 /tmp/cnmyzlqo.zij 09:52:41 /tmp/MSBuild2228 09:52:41 /tmp/ihgaucfk.ffl 09:52:41 /tmp/MSBuild44656 09:52:41 /tmp/pfrbdi5q.fzk 09:52:41 /tmp/CASESENSITIVETEST3b6abb6baf384e3383078919f4d71a74 09:52:41 /tmp/.SendingDataSystem.Core_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.dll 09:52:41 /tmp/.SendingDataSystem.Core_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.pdb 09:52:41 /tmp/.SendingDataSystem.Features_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.dll 09:52:41 /tmp/.SendingDataSystem.Features_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.pdb 09:52:41 /tmp/.SendingDataSystem.Service_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.dll 09:52:41 /tmp/.SendingDataSystem.Service_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.pdb 09:52:41 /tmp/.SendingDataSystem.Tests.Common_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.dll 09:52:41 /tmp/.SendingDataSystem.Tests.Common_600d5a8c-eeaa-45b8-a3ba-21b45a06c44d.pdb 09:52:41 /tmp/.MasterDataReleaseBuilder.Core_8de3e13e-6112-4d04-af4e-07088dbd7d65 09:52:41 /tmp/.MasterDataReleaseBuilder.Features_8de3e13e-6112-4d04-af4e-07088dbd7d65 09:52:41 /tmp/.MasterDataReleaseBuilder.Service_8de3e13e-6112-4d04-af4e-07088dbd7d65 09:52:41 /tmp/MSBuild34895 09:52:41 /tmp/.MasterDataReleaseBuilder.Core_44426035-3909-423e-94d6-b04450fb03c4.dll 09:52:41 /tmp/.MasterDataReleaseBuilder.Core_44426035-3909-423e-94d6-b04450fb03c4.pdb 09:52:41 /tmp/.MasterDataReleaseBuilder.Features_44426035-3909-423e-94d6-b04450fb03c4.dll 09:52:41 /tmp/.MasterDataReleaseBuilder.Features_44426035-3909-423e-94d6-b04450fb03c4.pdb 09:52:41 /tmp/.MasterDataReleaseBuilder.Service_44426035-3909-423e-94d6-b04450fb03c4.dll 09:52:41 /tmp/.MasterDataReleaseBuilder.Service_44426035-3909-423e-94d6-b04450fb03c4.pdb 09:52:41 /tmp/.MasterDataReleaseBuilder.Tests.Common_44426035-3909-423e-94d6-b04450fb03c4.dll 09:52:41 /tmp/.MasterDataReleaseBuilder.Tests.Common_44426035-3909-423e-94d6-b04450fb03c4.pdb 09:52:41 /tmp/tmpiJwXoR.tmp 09:52:41 /tmp/hsperfdata_root 09:52:41 /tmp/gitit-hostkey.rsa

hoyosjs commented 3 years ago

What version is the SDK/runtime (as seen by dotnet --info). I don't see ANY dotnet event pipes in there. This makes me wander why it disappeared. Some possibilities:

Somehow, you are running this before dotnet 3.1. This was when we introduced tooling support.
The temp directory is hidden by something like PrivateTemp. This is unlikely given your output.
Somehow, we were requested to shut down diagnostic capabilities. If you know the PID of the process (obtained from something like top/ps+grep/htop), I would poke the environment file (/proc/<PID>/environ) and see if there's an environment variable COMPlus_EnableDiagnostics being set to 0. (If you share such file's contents here, make sure to avoid sharing secrets as they are often in the environment block).

One more thing. Usually - at least in modern SDKs - dotnet test tends to have child processes that run your actual test. The executable is usually testhost. Look for it like sudo ps aux | grep -i testhost | grep -v grep

msallin commented 3 years ago

I found COMPlus_EnableDiagnostics was set to 0. I changed it to 1 and it works. Thank you!

dotnet / diagnostics

Question: dotnet-dump doesn't show the running dotnet process #2418