search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.19k stars 4.72k forks source link

Debugging linux dump #9128

Closed ashishnegi closed 4 years ago

ashishnegi commented 7 years ago

Hi I am trying to debug a crash dump on Linux. But I am not able to load any symbols.

Here are the steps I tried :

  1. I downloaded coreclr symbols from : 2.0.0 download which are symbols.

  2. I unzipped .symbols.nupkg into a folder with all .dbgs in one folder :

All *.so files are in folder allso.

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ ls -l
total 1181308
drwxrwxr-x 2 asnegi asnegi       4096 Oct 13 03:03 allso
-rwxrwxrwx 1 asnegi asnegi      29376 Jul 19 19:25 corerun
-rwxrwxrwx 1 asnegi asnegi     118776 Jul 19 19:25 corerun.dbg
-rwxrwxrwx 1 asnegi asnegi      86296 Jul 19 19:25 createdump
-rwxrwxrwx 1 asnegi asnegi     467480 Jul 19 19:25 createdump.dbg
-rwxrwxrwx 1 asnegi asnegi 1015279616 Oct 12 10:21 dotnet.49126.dmp
-rwxrwxrwx 1 asnegi asnegi    1646224 Jul 19 19:25 ilasm
-rwxrwxrwx 1 asnegi asnegi    4556520 Jul 19 19:25 ilasm.dbg
-rwxrwxrwx 1 asnegi asnegi    1606912 Jul 19 19:25 ildasm
-rwxrwxrwx 1 asnegi asnegi    4124184 Jul 19 19:25 ildasm.dbg
-rwxrwxrwx 1 asnegi asnegi   11713248 Jul 19 19:25 libclrjit.so.dbg
-rwxrwxrwx 1 asnegi asnegi   79269552 Jul 19 19:25 libcoreclr.so.dbg
-rwxrwxrwx 1 asnegi asnegi     860440 Jul 19 19:25 libcoreclrtraceptprovider.so.dbg
-rwxrwxrwx 1 asnegi asnegi    2965160 Jul 19 19:25 libdbgshim.so.dbg
-rwxrwxrwx 1 asnegi asnegi   52992576 Jul 19 19:25 libmscordaccore.so.dbg
-rwxrwxrwx 1 asnegi asnegi   11612952 Jul 19 19:25 libmscordbi.so.dbg
-rwxrwxrwx 1 asnegi asnegi     391152 Jul 19 19:25 libsosplugin.so.dbg
-rwxrwxrwx 1 asnegi asnegi    2783600 Jul 19 19:25 libsos.so.dbg
-rwxrwxrwx 1 asnegi asnegi     979464 Jul 19 19:25 libSystem.Globalization.Native.a
-rwxrwxrwx 1 asnegi asnegi      65869 Jul 19 19:25 sosdocsunix.txt
-rwxrwxrwx 1 asnegi asnegi     436024 Jul 19 19:25 System.Globalization.Native.so.dbg
-rwxrwxrwx 1 asnegi asnegi   10742784 Jul 19 19:25 System.Private.CoreLib.dll
-rwxrwxrwx 1 asnegi asnegi     967300 Jul 19 19:25 System.Private.CoreLib.pdb

  1. Dump is also present in same folder : dotnet.49126.dmp

  2. I then start lldb with :

    
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ lldb-3.6 -c dotnet.49126.dmp /usr/share/dotnet/dotnet

(lldb) plugin load allso/libsosplugin.so (lldb) setsostid c049 1 Mapped sos OS tid 0xc049 to lldb thread index 1 (lldb) set ClrStack Ambiguous command 'set'. Possible matches: settings setclrpath setsostid (lldb) sos ClrStack OS Thread Id: 0xc049 (1) Child SP IP Call Site GetFrameContext failed: 1 0000000000000000 0000000000000000 (lldb) target modules list [ 0] DAAB9A3E-C6DC-6F4B-E681-71B4F4A013FB-8B431DF0 /usr/share/dotnet/dotnet [ 1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl-2.23.so [ 2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0 [ 3] 144E588F-94CA-FAFD-BD0B-D1499C74190F-678DAD88 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 [ 4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm-2.23.so [ 5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1 [ 6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc-2.23.so [ 7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2 [ 8] FA3EC34F-FF22-C058-FE6C-120AD8466ACE-A05DCEBB /usr/share/dotnet/host/fxr/2.0.0/libhostfxr.so [ 9] 82FA3DDE-91A5-5710-F6CD-42274FF2701E-5A2E9BE3 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libhostpolicy.so [ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so [ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt-2.23.so [ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8 [ 13] 13BF33CF-CE9A-8E60-360A-390912876AC2-75C8AD93 /lib/x86_64-linux-gnu/libuuid.so.1 [ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8 [ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5 [ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclrtraceptprovider.so /home/asnegi/ryu/ryu/coreclr/runtimes/linux-x64/native/./libcoreclrtraceptprovider.so.dbg [ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 [ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 [ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4 [ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4 [ 21] 92F4379E-B144-FCB9-47CD-A120F46B67A7-A38D39F9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libclrjit.so [ 22] F3FA2CBA-8B15-ABC5-E785-BE2B38B6F620-E4E617A4 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Globalization.Native.so [ 23] 23A28B8C-35CF-747B-C507-EE9900D71790-47415517 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1 [ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55 [ 25] 6A9FD2EF-FD0E-B6D3-EE33-CAC7131B30E8-F6C2C6E1 /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1 [ 26] 7B4BE8CE-1CC5-DCF8-629A-A008500A93F0-E77577A0 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Native.so [ 27] 29F5E5B3-04BE-ED20-55CF-467E97187F20-6D9E3312 /usr/lib/x86_64-linux-gnu/libssh2.so.1 [ 28] 675F454A-D6FD-0B6C-A2E4-1127C7B98079-DA37F7B6 /lib/x86_64-linux-gnu/libssl.so.1.0.0 [ 29] 2DA08A7E-5BF6-1003-0DD3-3B70DB951399-626B7496 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [ 30] 85CA824C-BE57-3813-B1E3-59894F743EBB-100AB034 /usr/lib/x86_64-linux-gnu/libxml2.so.2 [ 31] 523E5F7F-4262-BD15-56B1-57D57F13ED16-722E804E /lib/x86_64-linux-gnu/libbz2.so.1.0 [ 32] 340B7B46-3F98-1B8A-0FB3-451751F881DF-1B0C2F74 /lib/x86_64-linux-gnu/libz.so.1 [ 33] 2EB15945-45CC-A1B5-258D-D030B398DA17-4F4541DA /lib/x86_64-linux-gnu/libgcrypt.so.20 [ 34] 3A325585-F22D-AE6B-A918-CDC3D2B62760-F666FE16 /lib/x86_64-linux-gnu/libgpg-error.so.0 [ 35] CE5C34DF-F6A9-8121-F82E-ED4547861645-5801DB1D /usr/lib/x86_64-linux-gnu/libsnappy.so.1 /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0 [ 36] D5C47E54-D8F2-BA13-26E2-8B7C8BAB45FD-5B537660 /usr/lib/x86_64-linux-gnu/liblz4.so.1 /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1 [ 37] 85F3D1F1-BE82-5AF2-FC27-936CD9F113D4-636B0F61 /lib/x86_64-linux-gnu/libcgroup.so.1



Observation :
As you can see, libcoreclr.so is at location :
`[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so`

It is not picking unstripped binaries from my current folder. I am not able to see any stack.

Expected :
It should load symbols from local folder.

Please tell me the right steps to debug.

Extra details :
dotnet version 2.0.0
lldb version 3.6

Note : I also tried with *.so and *.dbg files in same folder with crash_dump.
jkotas commented 7 years ago

cc @lt72

mikem8361 commented 7 years ago

Try adding -O "settings set target.exec-search-paths $HOME/ryu/ryu/coreclr/runtimes/linux-x64/native" to your lldb command line. This is neccessary if the .dbg files are not in the same place as the where they were in the core dump. If "target modules list -s" still doesn't show the .dbg file for libcoreclr.so then it maybe that *.dbg is the wrong version some how. You can use readelf on the libcoreclr.dbg file to get the build id and see if it matches the one in the target modules command (FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 ).

The reason clrstack isn't working is because you need to try another thread index. The setsosid command is a hacky workaround for a lldb 3.6 bug that requires guessing which thread index maps to which OS thread id.

lldb 3.9 has this problem fixed, but you need an libsosplugin.so that works with 3.9. You can try the one I attached (built on ubuntu 14.04 but should work on 16.xx).

libsosplugin.so.zip

danmoseley commented 7 years ago

@ashishnegi is this resolved for you now? If so, let's close it.

ashishnegi commented 7 years ago

Hi, I will be able to update on this thread in next 3 days. Thanks for your support.

ashishnegi commented 6 years ago

I see that build id is different for me :

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ readelf -n libcoreclr.so.dbg

Displaying notes found at file offset 0x00000200 with length 0x00000024:
  Owner                 Data size       Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: f09424a9ee0addca898872e72b24a51357677af5

Displaying notes found at file offset 0x049e1d5c with length 0x0000001c:
  Owner                 Data size       Description
  GNU                  0x00000009       NT_GNU_GOLD_VERSION (gold version)
    Version: gold 1.11
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $

and in target modules list :

[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so

target modules list -s does not show any libcoreclr.so.dbg. I see one libcoreclrtraceptprovider.so.dbg in target moduels list -s. readelf shows same target id as that of target modules list.

so, I think my symbols are not right.

  1. Is this right inference ?

Also, I think lldb-3.9 will not be able to help here as symbols are not right.

  1. Is it possible to find right symbols corresponding to build id : FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 of coreclr.so ?
ashishnegi commented 6 years ago

One question : In lldb, executing target modules list gives me FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 as build id. This is same as my local installed dotnet's build id. This dotnet has version : 2.0.0.

Is lldb showing build id of dotnet which was crashed on the remote machine Or build id of dotnet present in my local machine ?

mikem8361 commented 6 years ago

"target modules list -u -s" is giving you the build id of the modules in the core dump not what is on your local machine. Your build id of the libcoreclr.so in the dump doesn't match libcoreclr.so.dbg so lldb doesn't load it. What version of .NET Core was on the remote machine that the core dump was generated? It could be either 1.x or maybe a 2.0.x service release which could be the reason for the mismatch.

mikem8361 commented 6 years ago

It does look like the symbol zip files don't match the product. I'm looking into a solution. Not sure what time frame they will be fixed.

ashishnegi commented 6 years ago

@mikem8361 FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0. My local install has 2.0.0 and same build id.

So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0 ?

I will try building coreclr locally on linux to get the symbols. I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.

mikem8361 commented 6 years ago

We are working on fixing them now.

-------- Original message -------- From: Ashish Negi notifications@github.com Date: 10/30/17 8:21 PM (GMT-08:00) To: dotnet/coreclr coreclr@noreply.github.com Cc: Mike McLaughlin mikem@microsoft.com, Mention mention@noreply.github.com Subject: Re: [dotnet/coreclr] Debugging linux dump (#14490)

@mikem8361https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikem8361&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=Xz3mCMRORKdIKrsAhZ1yDaKAg1uKDVVvvRO0e67MtvA%3D&reserved=0 FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0. My local install has 2.0.0 and same build id.

So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcore%2Fblob%2Fmaster%2Frelease-notes%2Fdownload-archives%2F2.0.0-download.md&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=hW%2B4p7uMStSDr7NxPAUsM8W9whKD021XlieiXU2Wpls%3D&reserved=0 ?

I will try building coreclr locally on linux to get the symbols. I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fissues%2F14490%23issuecomment-340648585&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=PCOH6Z%2FFWAn3jbbEy%2Fs3pPzerB2PVFOGC4pBAYJUTNc%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJ-Fa_MRtvKb4Cs-KnDmIpW1IiEypX1pks5sxpJJgaJpZM4P4Rvp&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=RQ3N%2B1u1C3TBuPPwd4Xwtu8gUnOeQyLkl4SwEc3%2B6kM%3D&reserved=0.

mikem8361 commented 6 years ago

For now if you get the runtime.linux-x64.Microsoft.NETCore.App.2.0.0.symbols.nupkg package from the core-setup symbol zip, it will match the 2.0.0 binaries in your core dump.

mikem8361 commented 6 years ago

The download site symbol link has been fixed for 2.0.0 and 2.0.2.

ashishnegi commented 6 years ago

Hi @mikem8361 Thanks for sosplugin. I made good progress. I installed lldb-3.9 on Ubuntu 16.04 and used your libsosplugin.so. sos Threads shows real os thread id. I switched to it. I can use sos PrintException and it works.

I am still not able to get any stack on sos ClrStack. Can you help understand why that's happening ?

(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type:   System.IO.FileNotFoundException
Message:          Could not load the specified file.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
    00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c

StackTraceString: <none>
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
        Child SP               IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>

Here is my full debug session :

asnegi@asnegi-vm ~/ryu/dump $ lldb-3.9 -O "settings set target.exec-search-paths /home/asnegi/ryu/dump/" -c dotnet.49126.dmp /usr/share/dotnet/dotnet
(lldb) settings set target.exec-search-paths /home/asnegi/ryu/dump/
(lldb) target create "/usr/share/dotnet/dotnet" --core "dotnet.49126.dmp"
Core file '/home/asnegi/ryu/dump/dotnet.49126.dmp' (x86_64) was loaded.
(lldb) plugin load libsosplugin.so
libclrjit.so was compiled with optimization - stepping may behave oddly; variables may not be available.
(lldb) (lldb)
(lldb) sos Threads
ThreadCount:      20
UnstartedThread:  0
BackgroundThread: 19
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                                                        Lock
       ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
  11    1 bfef 000000000178BEC0    21220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn (Finalizer)
  13    2 bfe6 00000000017A26B0  2020020 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  38    3 c01a 0000000001822D10  1021220 Preemptive  00007F5EC8652450:00007F5EC8652F60 0000000001788130 0     Ukn (Threadpool Worker)
  46    4 c01c 00007F5EC0004BC0  1021220 Preemptive  00007F5EC8625A98:00007F5EC8626F60 0000000001788130 0     Ukn (Threadpool Worker)
  57    5 c049 00007F5E5C00D360    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn System.IO.FileNotFoundException 00007f5ec808ce00
   9    6 c045 00007F5E6401CF60    20220 Cooperative 00007F5EC86A5718:00007F5EC86A6F60 0000000001788130 0     Ukn
  49    7 c0c2 00007F5EBC0066C0  1021220 Cooperative 00007F5EC865FC20:00007F5EC8660F60 0000000001788130 0     Ukn (Threadpool Worker)
  31    8 c10e 00007F5E4C0044F0  1020220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn (Threadpool Worker)
  55    9 c14e 00007F5E1C0009F0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  52   10 c153 00007F5E20000CD0    20220 Preemptive  00007F5EC8646EE0:00007F5EC8646F60 0000000001788130 0     Ukn
  24   11 c744 00007F5EC055BEB0  1021220 Preemptive  00007F5EC865C438:00007F5EC865CF60 0000000001788130 0     Ukn (Threadpool Worker)
  51   12 c746 00007F5EBC00C780  1021220 Preemptive  00007F5EC860ACE0:00007F5EC860AF60 0000000001788130 0     Ukn (Threadpool Worker)
  53   13 c04c 00007F5E183178B0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  47   14 c04a 00007F5E5801B8D0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  37   15 c775 00007F5E480086D0  1021220 Cooperative 00007F5EC867CE60:00007F5EC867CF60 0000000001788130 0     Ukn (Threadpool Worker)
   4   16 c776 00007F5DF41909B0  1021220 Cooperative 00007F5EC867D1A0:00007F5EC867EF60 0000000001788130 0     Ukn (Threadpool Worker)
  20   17 c777 00007F5EBC011B30  1021220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn (Threadpool Worker)
  21   18 c778 00007F5EC0560820  1021220 Preemptive  00007F5EC85FAEE0:00007F5EC85FAF60 0000000001788130 0     Ukn (Threadpool Worker)
  27   19 c7f3 00007F5E6405FC50    21220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  48   20 c953 00007F5DEC0009F0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
(lldb) thread select 57
(lldb) * thread dotnet/runtime#3878: tid = 49225, 0x00007f5f68ad8c1d libpthread.so.0`__GI___nanosleep + 45, stop reason = signal SIGSEGV
    frame #0: 0x00007f5f68ad8c1d libpthread.so.0`__GI___nanosleep + 45
libpthread.so.0`__GI___nanosleep:
->  0x7f5f68ad8c1d <+45>: addb   %al, (%rax)
    0x7f5f68ad8c1f <+47>: addb   %al, (%rax)
    0x7f5f68ad8c21 <+49>: addb   %al, (%rax)
    0x7f5f68ad8c23 <+51>: addb   %al, (%rax)
(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type:   System.IO.FileNotFoundException
Message:          Could not load the specified file.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
    00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c

StackTraceString: <none>
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
        Child SP               IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>
(lldb) thread select 1
(lldb) * thread dotnet/coreclr#1: tid = 49481, 0x00007f5f4af52fd9 libclrjit.so`::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545, name = 'dotnet', stop reason = signal SIGSEGV
    frame #0: 0x00007f5f4af52fd9 libclrjit.so`::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545 [opt]
(lldb) sos PrintException
The current thread is unmanaged
(lldb) sos ClrStack
OS Thread Id: 0xc149 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
Failed to start stack walk: 80070057
(lldb)
lt72 commented 6 years ago

@mikem8361 to investigate

mikem8361 commented 6 years ago

@ashishnegi: Is there anyway you could attach the core dump? I would need it to debug why clrstack isn't working.

kirpichiki commented 6 years ago

Hi everyone and sorry for invasion. I have another situation with debugging core dump. I can't determine how to load symbols for my own code.

`clrstack` works fine for clr libraries, but when meets the library from my project it marks it as !Unknown
clrstack of thread 4b (core dump) 
OS Thread Id: 0x4b (1)
        Child SP               IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C /app/Hangfire.Mongo.dll!Unknown
00007F78D37FD6F0 00007F7B39A19AEA /app/Hangfire.Core.dll!Unknown
00007F78D37FD750 00007F7B39A19A4D /app/Hangfire.Core.dll!Unknown
00007F78D37FD770 00007F7B39A1964D /app/Hangfire.Core.dll!Unknown
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]


When i attaching to process directly all symbols is loaded
clrstack of thread 4b (process attach) 
OS Thread Id: 0x4b (1)
        Child SP               IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C Hangfire.Mongo.Signal.Mongo.MongoSignal.Listen(System.Threading.CancellationToken)
00007F78D37FD6F0 00007F7B39A19AEA Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD750 00007F7B39A19A4D Hangfire.Server.InfiniteLoopProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD770 00007F7B39A1964D Hangfire.Server.ServerProcessExtensions.RunProcess(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]


As far as I can gather the difference consists in loading of characters from vdso. After the core dump is loaded, `image list` contains of 65 positions
image list (core dump) 
[  0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A                    /usr/share/dotnet/dotnet
[  1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
[  2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  3] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
[  5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
[  7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  8] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10                    /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[  9] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 10] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 22] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 23] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 628F069D-8645-434A-D88E-502966359B23-213456EB                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 27] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 28] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6                    /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 31] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 32] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6                    /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 33] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57                    /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 34] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF                    /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 35] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 36] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 37] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 38] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832                    /lib/x86_64-linux-gnu/libz.so.1
[ 39] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 40] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF                    /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 41] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54                    /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 42] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620                    /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 43] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 44] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 45] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D                    /lib/x86_64-linux-gnu/libcom_err.so.2
[ 46] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 47] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2                    /lib/x86_64-linux-gnu/libresolv.so.2
[ 48] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 49] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 50] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 51] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 52] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6                    /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 53] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 54] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 55] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1                    /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 56] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 57] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818                    /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 58] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F                    /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 59] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700                    /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 60] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 61] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7                    /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 62] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 63] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9                    /lib/x86_64-linux-gnu/libcrypt.so.1
[ 64] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752                    /lib/x86_64-linux-gnu/libnss_files.so.2
[ 65] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416                    /lib/x86_64-linux-gnu/libnss_dns.so.2


After attaching to process there are 66 including [vdso] (0x00007fff2978d000)
image list (core dump) 
[  0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A                    /usr/share/dotnet/dotnet
[  1] 41FD7D4C-745E-681C-706B-ED73B4CC46C3-E422C5E6 0x00007fff2978d000 [vdso] (0x00007fff2978d000)
[  2] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
[  3] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  4] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  5] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
[  6] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  7] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
[  8] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  9] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10                    /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[ 10] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 11] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 12] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
[ 13] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 14] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 15] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 16] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 17] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 18] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 19] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 20] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 21] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 22] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 23] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 24] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 25] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 26] 628F069D-8645-434A-D88E-502966359B23-213456EB                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 27] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 28] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 29] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 30] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 31] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6                    /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 32] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 33] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6                    /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 34] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57                    /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 35] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF                    /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 36] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 37] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 38] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 39] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832                    /lib/x86_64-linux-gnu/libz.so.1
[ 40] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 41] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF                    /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 42] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54                    /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 43] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620                    /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 44] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 45] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 46] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D                    /lib/x86_64-linux-gnu/libcom_err.so.2
[ 47] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 48] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2                    /lib/x86_64-linux-gnu/libresolv.so.2
[ 49] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 50] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 51] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 52] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 53] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6                    /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 54] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 55] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 56] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1                    /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 57] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 58] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818                    /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 59] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F                    /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 60] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700                    /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 61] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 62] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7                    /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 63] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 64] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9                    /lib/x86_64-linux-gnu/libcrypt.so.1
[ 65] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752                    /lib/x86_64-linux-gnu/libnss_files.so.2
[ 66] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416                    /lib/x86_64-linux-gnu/libnss_dns.so.2


If my conclution is right, how can I load vdso while dedugging core dump?
@mikem8361 Won't you mind to gide me how to deal with this situation?

Misc info:

mikem8361 commented 6 years ago

I don't think vso has anything to do with your problem. I think sos is having a problem finding the pdb's for your app or having problem unwinding the managed frames in the core dump. How did you generate the core dump? sos should load the portable pdbs from the same directory as the your app assemblies/program. What was the exact command used to load it in LLDB?

Is there any way you could zip up the core dump and all your app binaries and attach them to this issue?

kirpichiki commented 6 years ago

Thanks for the answer. I am generating coredump with the createdump utility using default parameters. Something like: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump 6 To start debugger I am using this command: lldb-4.0 -O "settings set target.exec-search-paths /app/" -o "plugin load /libsosplugin.so" --core /tmp/coredump.6 /usr/share/dotnet/dotnet Here /app - folder with my asp.net core app (dlls and pdbs), builded with Release configuration, /libsosplugin.so - manually builded plugin

I'm afraid sources is pretty confidential, so I cant share it with you =(

mikem8361 commented 6 years ago

This may be a bug in createdump. Try building the createdump from the coreclr master branch (I can also send it to you if you want) and/or try the “--full” option.

ashishnegi commented 6 years ago

@mikem8361 I somehow missed earlier notification about dump.

I can confirm that sos ClrStack is working for me for few weeks. I think I didn't do any thing new to make it work. Otherwise, even then sos EEStack -EE was working for me.

We can close this thread now.

kirpichiki commented 6 years ago

Yep, I see my symbols only with full dump. Thanks for your guidance @mikem8361. It's frustrating I have to load all 24 Gb of virtual memory just to watch stacktrace, but if thats only option I'll find some workaround.

mikem8361 commented 6 years ago

Did you try the createdump built from the coreclr master branch (with the default options)? This will let me know that the bug/problem is fixed for 2.1. It should work with the 2.0.3 runtime.

createdump.zip

kirpichiki commented 6 years ago

@mikem8361 your createdump fails with relocation error: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump: symbol PAL_VirtualUnwindOutOfProc, version V1.0 not defined in file libmscordaccore.so with link time reference

I compiled coreclr from the master and tried to run createdump with the default options - same result

OS Thread Id: 0x7f (1)
        Child SP               IP Call Site
00007F85347F7210 00007f8808245360 [GCFrame: 00007f85347f7210]
00007F85347F7300 00007f8808245360 [HelperMethodFrame_1OBJ: 00007f85347f7300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F85347F7430 00007F878D45B503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F85347F74C0 00007F878D41B6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F85347F7520 00007F878D41B559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F85347F75E0 00007F878D41B273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F85347F7610 00007F87914B4BFC /app/Hangfire.Mongo.dll!Unknown
00007F85347F76F0 00007F8791486F2A /app/Hangfire.Core.dll!Unknown
00007F85347F7750 00007F8791486E8D /app/Hangfire.Core.dll!Unknown
00007F85347F7770 00007F8791486A8D /app/Hangfire.Core.dll!Unknown
00007F85347F77F0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F85347F7840 00007F878D41AB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F85347F78E0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F85347F7C58 00007f8806b94067 [GCFrame: 00007f85347f7c58]
00007F85347F7D30 00007f8806b94067 [DebuggerU2MCatchHandlerFrame: 00007f85347f7d30]
mikem8361 commented 6 years ago

Sorry about that. The standard createdump has some dependencies on libmscordaccore.so. Try this createdump:

createdump.zip

kirpichiki commented 6 years ago

No changes. Still getting !Unknown.

mikem8361 commented 6 years ago

Issue moved to dotnet/diagnostics dotnet/coreclr#48 via ZenHub