search

dotnet / diagnostics

This repository contains the source code for various .NET Core runtime diagnostic tools and documents.
MIT License
1.18k stars 354 forks source link

Debugging linux dump #48

Closed mikem8361 closed 5 years ago

mikem8361 commented 6 years ago

@ashishnegi commented on Fri Oct 13 2017

Hi I am trying to debug a crash dump on Linux. But I am not able to load any symbols.

Here are the steps I tried :

  1. I downloaded coreclr symbols from : 2.0.0 download which are symbols.

  2. I unzipped .symbols.nupkg into a folder with all .dbgs in one folder :

All *.so files are in folder allso.

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ ls -l
total 1181308
drwxrwxr-x 2 asnegi asnegi       4096 Oct 13 03:03 allso
-rwxrwxrwx 1 asnegi asnegi      29376 Jul 19 19:25 corerun
-rwxrwxrwx 1 asnegi asnegi     118776 Jul 19 19:25 corerun.dbg
-rwxrwxrwx 1 asnegi asnegi      86296 Jul 19 19:25 createdump
-rwxrwxrwx 1 asnegi asnegi     467480 Jul 19 19:25 createdump.dbg
-rwxrwxrwx 1 asnegi asnegi 1015279616 Oct 12 10:21 dotnet.49126.dmp
-rwxrwxrwx 1 asnegi asnegi    1646224 Jul 19 19:25 ilasm
-rwxrwxrwx 1 asnegi asnegi    4556520 Jul 19 19:25 ilasm.dbg
-rwxrwxrwx 1 asnegi asnegi    1606912 Jul 19 19:25 ildasm
-rwxrwxrwx 1 asnegi asnegi    4124184 Jul 19 19:25 ildasm.dbg
-rwxrwxrwx 1 asnegi asnegi   11713248 Jul 19 19:25 libclrjit.so.dbg
-rwxrwxrwx 1 asnegi asnegi   79269552 Jul 19 19:25 libcoreclr.so.dbg
-rwxrwxrwx 1 asnegi asnegi     860440 Jul 19 19:25 libcoreclrtraceptprovider.so.dbg
-rwxrwxrwx 1 asnegi asnegi    2965160 Jul 19 19:25 libdbgshim.so.dbg
-rwxrwxrwx 1 asnegi asnegi   52992576 Jul 19 19:25 libmscordaccore.so.dbg
-rwxrwxrwx 1 asnegi asnegi   11612952 Jul 19 19:25 libmscordbi.so.dbg
-rwxrwxrwx 1 asnegi asnegi     391152 Jul 19 19:25 libsosplugin.so.dbg
-rwxrwxrwx 1 asnegi asnegi    2783600 Jul 19 19:25 libsos.so.dbg
-rwxrwxrwx 1 asnegi asnegi     979464 Jul 19 19:25 libSystem.Globalization.Native.a
-rwxrwxrwx 1 asnegi asnegi      65869 Jul 19 19:25 sosdocsunix.txt
-rwxrwxrwx 1 asnegi asnegi     436024 Jul 19 19:25 System.Globalization.Native.so.dbg
-rwxrwxrwx 1 asnegi asnegi   10742784 Jul 19 19:25 System.Private.CoreLib.dll
-rwxrwxrwx 1 asnegi asnegi     967300 Jul 19 19:25 System.Private.CoreLib.pdb

  1. Dump is also present in same folder : dotnet.49126.dmp

  2. I then start lldb with :

    
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ lldb-3.6 -c dotnet.49126.dmp /usr/share/dotnet/dotnet

(lldb) plugin load allso/libsosplugin.so (lldb) setsostid c049 1 Mapped sos OS tid 0xc049 to lldb thread index 1 (lldb) set ClrStack Ambiguous command 'set'. Possible matches: settings setclrpath setsostid (lldb) sos ClrStack OS Thread Id: 0xc049 (1) Child SP IP Call Site GetFrameContext failed: 1 0000000000000000 0000000000000000 (lldb) target modules list [ 0] DAAB9A3E-C6DC-6F4B-E681-71B4F4A013FB-8B431DF0 /usr/share/dotnet/dotnet [ 1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl-2.23.so [ 2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0 [ 3] 144E588F-94CA-FAFD-BD0B-D1499C74190F-678DAD88 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 [ 4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm-2.23.so [ 5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1 [ 6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc-2.23.so [ 7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2 [ 8] FA3EC34F-FF22-C058-FE6C-120AD8466ACE-A05DCEBB /usr/share/dotnet/host/fxr/2.0.0/libhostfxr.so [ 9] 82FA3DDE-91A5-5710-F6CD-42274FF2701E-5A2E9BE3 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libhostpolicy.so [ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so [ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt-2.23.so [ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8 [ 13] 13BF33CF-CE9A-8E60-360A-390912876AC2-75C8AD93 /lib/x86_64-linux-gnu/libuuid.so.1 [ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8 [ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5 [ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclrtraceptprovider.so /home/asnegi/ryu/ryu/coreclr/runtimes/linux-x64/native/./libcoreclrtraceptprovider.so.dbg [ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 [ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 [ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4 [ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4 [ 21] 92F4379E-B144-FCB9-47CD-A120F46B67A7-A38D39F9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libclrjit.so [ 22] F3FA2CBA-8B15-ABC5-E785-BE2B38B6F620-E4E617A4 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Globalization.Native.so [ 23] 23A28B8C-35CF-747B-C507-EE9900D71790-47415517 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1 [ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55 [ 25] 6A9FD2EF-FD0E-B6D3-EE33-CAC7131B30E8-F6C2C6E1 /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1 [ 26] 7B4BE8CE-1CC5-DCF8-629A-A008500A93F0-E77577A0 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Native.so [ 27] 29F5E5B3-04BE-ED20-55CF-467E97187F20-6D9E3312 /usr/lib/x86_64-linux-gnu/libssh2.so.1 [ 28] 675F454A-D6FD-0B6C-A2E4-1127C7B98079-DA37F7B6 /lib/x86_64-linux-gnu/libssl.so.1.0.0 [ 29] 2DA08A7E-5BF6-1003-0DD3-3B70DB951399-626B7496 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [ 30] 85CA824C-BE57-3813-B1E3-59894F743EBB-100AB034 /usr/lib/x86_64-linux-gnu/libxml2.so.2 [ 31] 523E5F7F-4262-BD15-56B1-57D57F13ED16-722E804E /lib/x86_64-linux-gnu/libbz2.so.1.0 [ 32] 340B7B46-3F98-1B8A-0FB3-451751F881DF-1B0C2F74 /lib/x86_64-linux-gnu/libz.so.1 [ 33] 2EB15945-45CC-A1B5-258D-D030B398DA17-4F4541DA /lib/x86_64-linux-gnu/libgcrypt.so.20 [ 34] 3A325585-F22D-AE6B-A918-CDC3D2B62760-F666FE16 /lib/x86_64-linux-gnu/libgpg-error.so.0 [ 35] CE5C34DF-F6A9-8121-F82E-ED4547861645-5801DB1D /usr/lib/x86_64-linux-gnu/libsnappy.so.1 /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0 [ 36] D5C47E54-D8F2-BA13-26E2-8B7C8BAB45FD-5B537660 /usr/lib/x86_64-linux-gnu/liblz4.so.1 /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1 [ 37] 85F3D1F1-BE82-5AF2-FC27-936CD9F113D4-636B0F61 /lib/x86_64-linux-gnu/libcgroup.so.1


Observation :
As you can see, libcoreclr.so is at location :
`[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so`

It is not picking unstripped binaries from my current folder. I am not able to see any stack.

Expected :
It should load symbols from local folder.

Please tell me the right steps to debug.

Extra details :
dotnet version 2.0.0
lldb version 3.6

Note : I also tried with *.so and *.dbg files in same folder with crash_dump.

---

@jkotas commented on [Fri Oct 13 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-336469425)

cc @lt72 

---

@mikem8361 commented on [Fri Oct 13 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-336516609)

Try adding `-O "settings set target.exec-search-paths $HOME/ryu/ryu/coreclr/runtimes/linux-x64/native" ` to your lldb command line. This is neccessary if the .dbg files are not in the same place as the where they were in the core dump.  If "target modules list -s" still doesn't show the .dbg file for libcoreclr.so then it maybe that *.dbg is the wrong version some how. You can use readelf on the libcoreclr.dbg file to get the build id and see if it matches the one in the target modules command (`FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` ).

The reason clrstack isn't working is because you need to try another thread index.  The setsosid command is a hacky workaround for a lldb 3.6 bug that requires guessing which thread index maps to which OS thread id. 

lldb 3.9 has this problem fixed, but you need an libsosplugin.so that works with 3.9. You can try the one I attached (built on ubuntu 14.04 but should work on 16.xx).

[libsosplugin.so.zip](https://github.com/dotnet/coreclr/files/1383435/libsosplugin.so.zip)

---

@danmosemsft commented on [Thu Oct 26 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-339820817)

@ashishnegi is this resolved for you now? If so, let's close it.

---

@ashishnegi commented on [Sat Oct 28 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340241610)

Hi, I will be able to update on this thread in next 3 days.
Thanks for your support.

---

@ashishnegi commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340350119)

I see that build id is different for me :

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ readelf -n libcoreclr.so.dbg

Displaying notes found at file offset 0x00000200 with length 0x00000024: Owner Data size Description GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: f09424a9ee0addca898872e72b24a51357677af5

Displaying notes found at file offset 0x049e1d5c with length 0x0000001c: Owner Data size Description GNU 0x00000009 NT_GNU_GOLD_VERSION (gold version) Version: gold 1.11 asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $


and in `target modules list` :

[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so


`target modules list -s` does not show any libcoreclr.so.dbg.
I see one `libcoreclrtraceptprovider.so.dbg` in `target moduels list -s`. readelf shows same target id as that of `target modules list`.

so, I think my symbols are not right. 

1. Is this right inference ?

Also,  I think lldb-3.9 will not be able to help here as symbols are not right.

2. Is it possible to find right symbols corresponding to build id :  `FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` of coreclr.so ?

---

@ashishnegi commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340454756)

One question :
In lldb, executing `target modules list` gives me `FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` as build id. This is same as my local installed dotnet's build id. This dotnet has version : 2.0.0.

Is lldb showing build id of dotnet which was crashed on the remote machine Or build id of dotnet present in my local machine ?

---

@mikem8361 commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340534923)

"target modules list -u -s" is giving you the build id of the modules in the core dump not what is on your local machine. Your build id of the libcoreclr.so in the dump doesn't match libcoreclr.so.dbg so lldb doesn't load it. What version of .NET Core was on the remote machine that the core dump was generated? It could be either 1.x or maybe a 2.0.x service release which could be the reason for the mismatch.

---

@mikem8361 commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340600776)

It does look like the symbol zip files don't match the product. I'm looking into a solution. Not sure what time frame they will be fixed.

---

@ashishnegi commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340648585)

@mikem8361 `FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.

So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in [coreclr's download archives of 2.0.0](https://github.com/dotnet/core/blob/master/release-notes/download-archives/2.0.0-download.md) ?

I will try building coreclr locally on linux to get the symbols.
I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.

---

@mikem8361 commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340655778)

We are working on fixing them now.

-------- Original message --------
From: Ashish Negi <notifications@github.com>
Date: 10/30/17 8:21 PM (GMT-08:00)
To: dotnet/coreclr <coreclr@noreply.github.com>
Cc: Mike McLaughlin <mikem@microsoft.com>, Mention <mention@noreply.github.com>
Subject: Re: [dotnet/coreclr] Debugging linux dump (#14490)

@mikem8361<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikem8361&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=Xz3mCMRORKdIKrsAhZ1yDaKAg1uKDVVvvRO0e67MtvA%3D&reserved=0> FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.

So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcore%2Fblob%2Fmaster%2Frelease-notes%2Fdownload-archives%2F2.0.0-download.md&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=hW%2B4p7uMStSDr7NxPAUsM8W9whKD021XlieiXU2Wpls%3D&reserved=0> ?

I will try building coreclr locally on linux to get the symbols.
I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fissues%2F14490%23issuecomment-340648585&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=PCOH6Z%2FFWAn3jbbEy%2Fs3pPzerB2PVFOGC4pBAYJUTNc%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJ-Fa_MRtvKb4Cs-KnDmIpW1IiEypX1pks5sxpJJgaJpZM4P4Rvp&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=RQ3N%2B1u1C3TBuPPwd4Xwtu8gUnOeQyLkl4SwEc3%2B6kM%3D&reserved=0>.

---

@mikem8361 commented on [Tue Oct 31 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340870094)

For now if you get the runtime.linux-x64.Microsoft.NETCore.App.2.0.0.symbols.nupkg package from the core-setup symbol [zip](https://download.microsoft.com/download/D/E/B/DEB55D32-4CC3-46B0-A241-1A5BF2BB533C/core-setup-2.0-symbols.zip), it will match the 2.0.0 binaries in your core dump.

---

@mikem8361 commented on [Wed Nov 01 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-341177505)

The download site symbol link has been fixed for 2.0.0 and 2.0.2. 

---

@ashishnegi commented on [Thu Nov 02 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-341413972)

Hi @mikem8361 Thanks for sosplugin.
I made good progress.
I installed lldb-3.9 on Ubuntu 16.04 and used your libsosplugin.so.
`sos Threads` shows real os thread id.
I switched to it. I can use `sos PrintException` and it works.

I am still not able to get any stack on `sos ClrStack`. Can you help understand why that's happening ?

(lldb) sos PrintException Exception object: 00007f5ec808ce00 Exception type: System.IO.FileNotFoundException Message: Could not load the specified file. InnerException: StackTrace (generated): SP IP Function 00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d 00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c

StackTraceString: HResult: 80070002 (lldb) sos ClrStack OS Thread Id: 0xc049 (57) Child SP IP Call Site GetFrameContext failed: 1 0000000000000000 0000000000000000 


Here is my full debug session :

asnegi@asnegi-vm ~/ryu/dump $ lldb-3.9 -O "settings set target.exec-search-paths /home/asnegi/ryu/dump/" -c dotnet.49126.dmp /usr/share/dotnet/dotnet (lldb) settings set target.exec-search-paths /home/asnegi/ryu/dump/ (lldb) target create "/usr/share/dotnet/dotnet" --core "dotnet.49126.dmp" Core file '/home/asnegi/ryu/dump/dotnet.49126.dmp' (x86_64) was loaded. (lldb) plugin load libsosplugin.so libclrjit.so was compiled with optimization - stepping may behave oddly; variables may not be available. (lldb) (lldb) (lldb) sos Threads ThreadCount: 20 UnstartedThread: 0 BackgroundThread: 19 PendingThread: 0 DeadThread: 0 Hosted Runtime: no Lock ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception 11 1 bfef 000000000178BEC0 21220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Finalizer) 13 2 bfe6 00000000017A26B0 2020020 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 38 3 c01a 0000000001822D10 1021220 Preemptive 00007F5EC8652450:00007F5EC8652F60 0000000001788130 0 Ukn (Threadpool Worker) 46 4 c01c 00007F5EC0004BC0 1021220 Preemptive 00007F5EC8625A98:00007F5EC8626F60 0000000001788130 0 Ukn (Threadpool Worker) 57 5 c049 00007F5E5C00D360 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn System.IO.FileNotFoundException 00007f5ec808ce00 9 6 c045 00007F5E6401CF60 20220 Cooperative 00007F5EC86A5718:00007F5EC86A6F60 0000000001788130 0 Ukn 49 7 c0c2 00007F5EBC0066C0 1021220 Cooperative 00007F5EC865FC20:00007F5EC8660F60 0000000001788130 0 Ukn (Threadpool Worker) 31 8 c10e 00007F5E4C0044F0 1020220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Threadpool Worker) 55 9 c14e 00007F5E1C0009F0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 52 10 c153 00007F5E20000CD0 20220 Preemptive 00007F5EC8646EE0:00007F5EC8646F60 0000000001788130 0 Ukn 24 11 c744 00007F5EC055BEB0 1021220 Preemptive 00007F5EC865C438:00007F5EC865CF60 0000000001788130 0 Ukn (Threadpool Worker) 51 12 c746 00007F5EBC00C780 1021220 Preemptive 00007F5EC860ACE0:00007F5EC860AF60 0000000001788130 0 Ukn (Threadpool Worker) 53 13 c04c 00007F5E183178B0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 47 14 c04a 00007F5E5801B8D0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 37 15 c775 00007F5E480086D0 1021220 Cooperative 00007F5EC867CE60:00007F5EC867CF60 0000000001788130 0 Ukn (Threadpool Worker) 4 16 c776 00007F5DF41909B0 1021220 Cooperative 00007F5EC867D1A0:00007F5EC867EF60 0000000001788130 0 Ukn (Threadpool Worker) 20 17 c777 00007F5EBC011B30 1021220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Threadpool Worker) 21 18 c778 00007F5EC0560820 1021220 Preemptive 00007F5EC85FAEE0:00007F5EC85FAF60 0000000001788130 0 Ukn (Threadpool Worker) 27 19 c7f3 00007F5E6405FC50 21220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 48 20 c953 00007F5DEC0009F0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (lldb) thread select 57 (lldb) * thread #57: tid = 49225, 0x00007f5f68ad8c1d libpthread.so.0__GI___nanosleep + 45, stop reason = signal SIGSEGV frame #0: 0x00007f5f68ad8c1d libpthread.so.0GInanosleep + 45 libpthread.so.0`GInanosleep: -> 0x7f5f68ad8c1d <+45>: addb %al, (%rax) 0x7f5f68ad8c1f <+47>: addb %al, (%rax) 0x7f5f68ad8c21 <+49>: addb %al, (%rax) 0x7f5f68ad8c23 <+51>: addb %al, (%rax) (lldb) sos PrintException Exception object: 00007f5ec808ce00 Exception type: System.IO.FileNotFoundException Message: Could not load the specified file. InnerException: StackTrace (generated): SP IP Function 00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d 00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c

StackTraceString: HResult: 80070002 (lldb) sos ClrStack OS Thread Id: 0xc049 (57) Child SP IP Call Site GetFrameContext failed: 1 0000000000000000 0000000000000000 (lldb) thread select 1 (lldb) * thread #1: tid = 49481, 0x00007f5f4af52fd9 libclrjit.so::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545, name = 'dotnet', stop reason = signal SIGSEGV frame #0: 0x00007f5f4af52fd9 libclrjit.so::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=) + 249 at context.cpp:545 [opt] (lldb) sos PrintException The current thread is unmanaged (lldb) sos ClrStack OS Thread Id: 0xc149 (1) Unable to walk the managed stack. The current thread is likely not a managed thread. You can run !threads to get a list of managed threads in the process Failed to start stack walk: 80070057 (lldb)


---

@lt72 commented on [Tue Nov 07 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-342655242)

@mikem8361 to investigate

---

@mikem8361 commented on [Tue Nov 07 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-342670098)

@ashishnegi: Is there anyway you could attach the core dump? I would need it to debug why clrstack isn't working.

---

@kirpichiki commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353372931)

Hi everyone and sorry for invasion. I have another situation with debugging core dump. I can't determine how to load symbols for my own code.
<h6> `clrstack` works fine for clr libraries, but  when meets the library from my project it marks it as !Unknown</h6>
<details><summary>clrstack of thread 4b (core dump)</summary>
<pre>
OS Thread Id: 0x4b (1)
        Child SP               IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C /app/Hangfire.Mongo.dll!Unknown
00007F78D37FD6F0 00007F7B39A19AEA /app/Hangfire.Core.dll!Unknown
00007F78D37FD750 00007F7B39A19A4D /app/Hangfire.Core.dll!Unknown
00007F78D37FD770 00007F7B39A1964D /app/Hangfire.Core.dll!Unknown
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]
</pre>
</details>
</br>
<h6>When i attaching to process directly all symbols is loaded</h6>
<details><summary>clrstack of thread 4b (process attach)</summary>
<pre>
OS Thread Id: 0x4b (1)
        Child SP               IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C Hangfire.Mongo.Signal.Mongo.MongoSignal.Listen(System.Threading.CancellationToken)
00007F78D37FD6F0 00007F7B39A19AEA Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD750 00007F7B39A19A4D Hangfire.Server.InfiniteLoopProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD770 00007F7B39A1964D Hangfire.Server.ServerProcessExtensions.RunProcess(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]
</pre>
</details>
</br>
<h6>As far as I can gather the difference consists in loading of characters from vdso. After the core dump is loaded, `image list` contains of 65 positions</h6>
<details><summary>image list (core dump)</summary>
<pre>
[  0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A                    /usr/share/dotnet/dotnet
[  1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
[  2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  3] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
[  5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
[  7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  8] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10                    /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[  9] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 10] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 22] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 23] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 628F069D-8645-434A-D88E-502966359B23-213456EB                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 27] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 28] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6                    /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 31] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 32] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6                    /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 33] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57                    /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 34] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF                    /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 35] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 36] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 37] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 38] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832                    /lib/x86_64-linux-gnu/libz.so.1
[ 39] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 40] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF                    /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 41] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54                    /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 42] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620                    /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 43] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 44] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 45] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D                    /lib/x86_64-linux-gnu/libcom_err.so.2
[ 46] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 47] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2                    /lib/x86_64-linux-gnu/libresolv.so.2
[ 48] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 49] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 50] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 51] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 52] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6                    /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 53] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 54] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 55] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1                    /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 56] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 57] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818                    /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 58] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F                    /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 59] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700                    /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 60] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 61] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7                    /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 62] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 63] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9                    /lib/x86_64-linux-gnu/libcrypt.so.1
[ 64] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752                    /lib/x86_64-linux-gnu/libnss_files.so.2
[ 65] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416                    /lib/x86_64-linux-gnu/libnss_dns.so.2
</pre>
</details>
</br>
<h6>After attaching to process there are 66 including <code>[vdso] (0x00007fff2978d000)</code></h6>
<details><summary>image list (core dump)</summary>
<pre>
[  0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A                    /usr/share/dotnet/dotnet
[  1] 41FD7D4C-745E-681C-706B-ED73B4CC46C3-E422C5E6 0x00007fff2978d000 [vdso] (0x00007fff2978d000)
[  2] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
[  3] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  4] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  5] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
[  6] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  7] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
[  8] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  9] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10                    /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[ 10] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 11] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 12] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
[ 13] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 14] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 15] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 16] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 17] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 18] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 19] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 20] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 21] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 22] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 23] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 24] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 25] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 26] 628F069D-8645-434A-D88E-502966359B23-213456EB                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 27] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 28] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 29] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 30] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 31] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6                    /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 32] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 33] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6                    /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 34] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57                    /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 35] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF                    /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 36] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 37] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 38] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 39] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832                    /lib/x86_64-linux-gnu/libz.so.1
[ 40] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 41] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF                    /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 42] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54                    /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 43] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620                    /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 44] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 45] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 46] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D                    /lib/x86_64-linux-gnu/libcom_err.so.2
[ 47] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 48] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2                    /lib/x86_64-linux-gnu/libresolv.so.2
[ 49] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 50] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 51] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 52] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 53] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6                    /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 54] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 55] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 56] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1                    /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 57] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 58] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818                    /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 59] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F                    /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 60] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700                    /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 61] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 62] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7                    /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 63] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 64] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9                    /lib/x86_64-linux-gnu/libcrypt.so.1
[ 65] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752                    /lib/x86_64-linux-gnu/libnss_files.so.2
[ 66] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416                    /lib/x86_64-linux-gnu/libnss_dns.so.2
</pre>
</details>
</br>
If my conclution is right, how can I load vdso while dedugging core dump?</br>
@mikem8361 Won't you mind to gide me how to deal with this situation?

Misc info:
+ All work takes place in `docker` container with `Ubuntu:xenial` base
+ `dotnet` version is `2.0.3`
+ `lldb` version is `4.0` with manually builded `libsosplugin.so`
+ Changing `settings set target.exec-search-paths` to folder with my program dlls seems to have no effect

---

@mikem8361 commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353437315)

I don't think vso has anything to do with your problem.  I think sos is having a problem finding the pdb's for your app or having problem unwinding the managed frames in the core dump.  How did you generate the core dump? sos should load the portable pdbs from the same directory as the your app assemblies/program. What was the exact command used to load it in LLDB? 

Is there any way you could zip up the core dump and all your app binaries and attach them to this issue?

---

@kirpichiki commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353462274)

Thanks for the answer. I am generating coredump with the `createdump` utility using default parameters. Something like:
`/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump 6`
To start debugger I am using this command:
`lldb-4.0 -O "settings set target.exec-search-paths /app/" -o "plugin load /libsosplugin.so" --core /tmp/coredump.6 /usr/share/dotnet/dotnet`
Here `/app` - folder with my asp.net core app (dlls and pdbs), builded with `Release` configuration,
`/libsosplugin.so` - manually builded plugin

I'm afraid sources is pretty confidential, so I cant share it with you =(

---

@mikem8361 commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353471042)

This may be a bug in createdump. Try building the createdump from the coreclr master branch (I can also send it to you if you want) and/or try the “--full” option.

---

@ashishnegi commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353518069)

@mikem8361 I somehow missed earlier notification about dump.

I can confirm that sos ClrStack is working for me for few weeks.
I think I didn't do any thing new to make it work.
Otherwise, even then `sos EEStack -EE` was working for me.

We can close this thread now.

---

@kirpichiki commented on [Fri Dec 22 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353602857)

Yep, I see my symbols only with **full** dump. Thanks for your guidance @mikem8361. It's frustrating I have to load all 24 Gb of virtual memory just to watch stacktrace, but if thats only option I'll find some workaround.

---

@mikem8361 commented on [Fri Dec 22 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353654006)

Did you try the createdump built from the coreclr master branch (with the default options)? This will let me know that the bug/problem is fixed for 2.1. It should work with the 2.0.3 runtime.

[createdump.zip](https://github.com/dotnet/coreclr/files/1583248/createdump.zip)

---

@kirpichiki commented on [Mon Dec 25 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353860447)

@mikem8361 your createdump fails with
`relocation error: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump: symbol PAL_VirtualUnwindOutOfProc, version V1.0 not defined in file libmscordaccore.so with link time reference`

I compiled coreclr  from the `master` and tried to run `createdump` with the default options - same result

OS Thread Id: 0x7f (1) Child SP IP Call Site 00007F85347F7210 00007f8808245360 [GCFrame: 00007f85347f7210] 00007F85347F7300 00007f8808245360 [HelperMethodFrame_1OBJ: 00007f85347f7300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object) 00007F85347F7430 00007F878D45B503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) 00007F85347F74C0 00007F878D41B6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) 00007F85347F7520 00007F878D41B559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) 00007F85347F75E0 00007F878D41B273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken) 00007F85347F7610 00007F87914B4BFC /app/Hangfire.Mongo.dll!Unknown 00007F85347F76F0 00007F8791486F2A /app/Hangfire.Core.dll!Unknown 00007F85347F7750 00007F8791486E8D /app/Hangfire.Core.dll!Unknown 00007F85347F7770 00007F8791486A8D /app/Hangfire.Core.dll!Unknown 00007F85347F77F0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F85347F7840 00007F878D41AB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) 00007F85347F78E0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F85347F7C58 00007f8806b94067 [GCFrame: 00007f85347f7c58] 00007F85347F7D30 00007f8806b94067 [DebuggerU2MCatchHandlerFrame: 00007f85347f7d30]



---

@mikem8361 commented on [Tue Jan 02 2018](https://github.com/dotnet/coreclr/issues/14490#issuecomment-354854110)

Sorry about that.  The standard createdump has some dependencies on libmscordaccore.so.  Try  this createdump: 

[createdump.zip](https://github.com/dotnet/coreclr/files/1598487/createdump.zip)

---

@kirpichiki commented on [Wed Jan 10 2018](https://github.com/dotnet/coreclr/issues/14490#issuecomment-356557152)

No changes. Still getting !Unknown.
mikem8361 commented 6 years ago

Your !Unknown module name issues with createdump is #56. A workaround in createdump has been merged into coreclr "master" (.NET Core 3).