Observation :
As you can see, libcoreclr.so is at location :
`[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so`
It is not picking unstripped binaries from my current folder. I am not able to see any stack.
Expected :
It should load symbols from local folder.
Please tell me the right steps to debug.
Extra details :
dotnet version 2.0.0
lldb version 3.6
Note : I also tried with *.so and *.dbg files in same folder with crash_dump.
---
@jkotas commented on [Fri Oct 13 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-336469425)
cc @lt72
---
@mikem8361 commented on [Fri Oct 13 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-336516609)
Try adding `-O "settings set target.exec-search-paths $HOME/ryu/ryu/coreclr/runtimes/linux-x64/native" ` to your lldb command line. This is neccessary if the .dbg files are not in the same place as the where they were in the core dump. If "target modules list -s" still doesn't show the .dbg file for libcoreclr.so then it maybe that *.dbg is the wrong version some how. You can use readelf on the libcoreclr.dbg file to get the build id and see if it matches the one in the target modules command (`FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` ).
The reason clrstack isn't working is because you need to try another thread index. The setsosid command is a hacky workaround for a lldb 3.6 bug that requires guessing which thread index maps to which OS thread id.
lldb 3.9 has this problem fixed, but you need an libsosplugin.so that works with 3.9. You can try the one I attached (built on ubuntu 14.04 but should work on 16.xx).
[libsosplugin.so.zip](https://github.com/dotnet/coreclr/files/1383435/libsosplugin.so.zip)
---
@danmosemsft commented on [Thu Oct 26 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-339820817)
@ashishnegi is this resolved for you now? If so, let's close it.
---
@ashishnegi commented on [Sat Oct 28 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340241610)
Hi, I will be able to update on this thread in next 3 days.
Thanks for your support.
---
@ashishnegi commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340350119)
I see that build id is different for me :
Displaying notes found at file offset 0x00000200 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: f09424a9ee0addca898872e72b24a51357677af5
Displaying notes found at file offset 0x049e1d5c with length 0x0000001c:
Owner Data size Description
GNU 0x00000009 NT_GNU_GOLD_VERSION (gold version)
Version: gold 1.11
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $
`target modules list -s` does not show any libcoreclr.so.dbg.
I see one `libcoreclrtraceptprovider.so.dbg` in `target moduels list -s`. readelf shows same target id as that of `target modules list`.
so, I think my symbols are not right.
1. Is this right inference ?
Also, I think lldb-3.9 will not be able to help here as symbols are not right.
2. Is it possible to find right symbols corresponding to build id : `FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` of coreclr.so ?
---
@ashishnegi commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340454756)
One question :
In lldb, executing `target modules list` gives me `FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` as build id. This is same as my local installed dotnet's build id. This dotnet has version : 2.0.0.
Is lldb showing build id of dotnet which was crashed on the remote machine Or build id of dotnet present in my local machine ?
---
@mikem8361 commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340534923)
"target modules list -u -s" is giving you the build id of the modules in the core dump not what is on your local machine. Your build id of the libcoreclr.so in the dump doesn't match libcoreclr.so.dbg so lldb doesn't load it. What version of .NET Core was on the remote machine that the core dump was generated? It could be either 1.x or maybe a 2.0.x service release which could be the reason for the mismatch.
---
@mikem8361 commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340600776)
It does look like the symbol zip files don't match the product. I'm looking into a solution. Not sure what time frame they will be fixed.
---
@ashishnegi commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340648585)
@mikem8361 `FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119` is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.
So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in [coreclr's download archives of 2.0.0](https://github.com/dotnet/core/blob/master/release-notes/download-archives/2.0.0-download.md) ?
I will try building coreclr locally on linux to get the symbols.
I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.
---
@mikem8361 commented on [Mon Oct 30 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340655778)
We are working on fixing them now.
-------- Original message --------
From: Ashish Negi <notifications@github.com>
Date: 10/30/17 8:21 PM (GMT-08:00)
To: dotnet/coreclr <coreclr@noreply.github.com>
Cc: Mike McLaughlin <mikem@microsoft.com>, Mention <mention@noreply.github.com>
Subject: Re: [dotnet/coreclr] Debugging linux dump (#14490)
@mikem8361<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikem8361&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=Xz3mCMRORKdIKrsAhZ1yDaKAg1uKDVVvvRO0e67MtvA%3D&reserved=0> FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.
So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcore%2Fblob%2Fmaster%2Frelease-notes%2Fdownload-archives%2F2.0.0-download.md&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=hW%2B4p7uMStSDr7NxPAUsM8W9whKD021XlieiXU2Wpls%3D&reserved=0> ?
I will try building coreclr locally on linux to get the symbols.
I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fissues%2F14490%23issuecomment-340648585&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=PCOH6Z%2FFWAn3jbbEy%2Fs3pPzerB2PVFOGC4pBAYJUTNc%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJ-Fa_MRtvKb4Cs-KnDmIpW1IiEypX1pks5sxpJJgaJpZM4P4Rvp&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=RQ3N%2B1u1C3TBuPPwd4Xwtu8gUnOeQyLkl4SwEc3%2B6kM%3D&reserved=0>.
---
@mikem8361 commented on [Tue Oct 31 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-340870094)
For now if you get the runtime.linux-x64.Microsoft.NETCore.App.2.0.0.symbols.nupkg package from the core-setup symbol [zip](https://download.microsoft.com/download/D/E/B/DEB55D32-4CC3-46B0-A241-1A5BF2BB533C/core-setup-2.0-symbols.zip), it will match the 2.0.0 binaries in your core dump.
---
@mikem8361 commented on [Wed Nov 01 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-341177505)
The download site symbol link has been fixed for 2.0.0 and 2.0.2.
---
@ashishnegi commented on [Thu Nov 02 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-341413972)
Hi @mikem8361 Thanks for sosplugin.
I made good progress.
I installed lldb-3.9 on Ubuntu 16.04 and used your libsosplugin.so.
`sos Threads` shows real os thread id.
I switched to it. I can use `sos PrintException` and it works.
I am still not able to get any stack on `sos ClrStack`. Can you help understand why that's happening ?
(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type: System.IO.FileNotFoundException
Message: Could not load the specified file.
InnerException:
StackTrace (generated):
SP IP Function
00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c
StackTraceString:
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
StackTraceString:
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
(lldb) thread select 1
(lldb) * thread #1: tid = 49481, 0x00007f5f4af52fd9 libclrjit.so::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545, name = 'dotnet', stop reason = signal SIGSEGV frame #0: 0x00007f5f4af52fd9 libclrjit.so::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=) + 249 at context.cpp:545 [opt]
(lldb) sos PrintException
The current thread is unmanaged
(lldb) sos ClrStack
OS Thread Id: 0xc149 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
Failed to start stack walk: 80070057
(lldb)
---
@lt72 commented on [Tue Nov 07 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-342655242)
@mikem8361 to investigate
---
@mikem8361 commented on [Tue Nov 07 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-342670098)
@ashishnegi: Is there anyway you could attach the core dump? I would need it to debug why clrstack isn't working.
---
@kirpichiki commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353372931)
Hi everyone and sorry for invasion. I have another situation with debugging core dump. I can't determine how to load symbols for my own code.
<h6> `clrstack` works fine for clr libraries, but when meets the library from my project it marks it as !Unknown</h6>
<details><summary>clrstack of thread 4b (core dump)</summary>
<pre>
OS Thread Id: 0x4b (1)
Child SP IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C /app/Hangfire.Mongo.dll!Unknown
00007F78D37FD6F0 00007F7B39A19AEA /app/Hangfire.Core.dll!Unknown
00007F78D37FD750 00007F7B39A19A4D /app/Hangfire.Core.dll!Unknown
00007F78D37FD770 00007F7B39A1964D /app/Hangfire.Core.dll!Unknown
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]
</pre>
</details>
</br>
<h6>When i attaching to process directly all symbols is loaded</h6>
<details><summary>clrstack of thread 4b (process attach)</summary>
<pre>
OS Thread Id: 0x4b (1)
Child SP IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C Hangfire.Mongo.Signal.Mongo.MongoSignal.Listen(System.Threading.CancellationToken)
00007F78D37FD6F0 00007F7B39A19AEA Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD750 00007F7B39A19A4D Hangfire.Server.InfiniteLoopProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD770 00007F7B39A1964D Hangfire.Server.ServerProcessExtensions.RunProcess(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]
</pre>
</details>
</br>
<h6>As far as I can gather the difference consists in loading of characters from vdso. After the core dump is loaded, `image list` contains of 65 positions</h6>
<details><summary>image list (core dump)</summary>
<pre>
[ 0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A /usr/share/dotnet/dotnet
[ 1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2
[ 2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0
[ 3] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[ 4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6
[ 5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1
[ 6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6
[ 7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2
[ 8] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10 /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[ 9] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 10] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 22] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 23] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 628F069D-8645-434A-D88E-502966359B23-213456EB /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 27] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 28] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6 /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 31] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 32] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6 /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 33] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57 /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 34] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 35] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 36] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 37] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 38] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832 /lib/x86_64-linux-gnu/libz.so.1
[ 39] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 40] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 41] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54 /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 42] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620 /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 43] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 44] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 45] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D /lib/x86_64-linux-gnu/libcom_err.so.2
[ 46] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 47] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2 /lib/x86_64-linux-gnu/libresolv.so.2
[ 48] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 49] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3 /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 50] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 51] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3 /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 52] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6 /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 53] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 54] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357 /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 55] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1 /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 56] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 57] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818 /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 58] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 59] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700 /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 60] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91 /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 61] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7 /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 62] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 63] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9 /lib/x86_64-linux-gnu/libcrypt.so.1
[ 64] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752 /lib/x86_64-linux-gnu/libnss_files.so.2
[ 65] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416 /lib/x86_64-linux-gnu/libnss_dns.so.2
</pre>
</details>
</br>
<h6>After attaching to process there are 66 including <code>[vdso] (0x00007fff2978d000)</code></h6>
<details><summary>image list (core dump)</summary>
<pre>
[ 0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A /usr/share/dotnet/dotnet
[ 1] 41FD7D4C-745E-681C-706B-ED73B4CC46C3-E422C5E6 0x00007fff2978d000 [vdso] (0x00007fff2978d000)
[ 2] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2
[ 3] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0
[ 4] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[ 5] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6
[ 6] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1
[ 7] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6
[ 8] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2
[ 9] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10 /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[ 10] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 11] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 12] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1
[ 13] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 14] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A /lib/x86_64-linux-gnu/libuuid.so.1
[ 15] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 16] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5
[ 17] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 18] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 19] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 20] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 21] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 22] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 23] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 24] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 25] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 26] 628F069D-8645-434A-D88E-502966359B23-213456EB /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 27] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 28] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 29] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 30] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 31] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6 /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 32] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 33] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6 /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 34] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57 /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 35] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 36] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 37] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 38] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 39] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832 /lib/x86_64-linux-gnu/libz.so.1
[ 40] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 41] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 42] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54 /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 43] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620 /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 44] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 45] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 46] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D /lib/x86_64-linux-gnu/libcom_err.so.2
[ 47] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 48] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2 /lib/x86_64-linux-gnu/libresolv.so.2
[ 49] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 50] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3 /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 51] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 52] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3 /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 53] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6 /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 54] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 55] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357 /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 56] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1 /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 57] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 58] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818 /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 59] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 60] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700 /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 61] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91 /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 62] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7 /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 63] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 64] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9 /lib/x86_64-linux-gnu/libcrypt.so.1
[ 65] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752 /lib/x86_64-linux-gnu/libnss_files.so.2
[ 66] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416 /lib/x86_64-linux-gnu/libnss_dns.so.2
</pre>
</details>
</br>
If my conclution is right, how can I load vdso while dedugging core dump?</br>
@mikem8361 Won't you mind to gide me how to deal with this situation?
Misc info:
+ All work takes place in `docker` container with `Ubuntu:xenial` base
+ `dotnet` version is `2.0.3`
+ `lldb` version is `4.0` with manually builded `libsosplugin.so`
+ Changing `settings set target.exec-search-paths` to folder with my program dlls seems to have no effect
---
@mikem8361 commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353437315)
I don't think vso has anything to do with your problem. I think sos is having a problem finding the pdb's for your app or having problem unwinding the managed frames in the core dump. How did you generate the core dump? sos should load the portable pdbs from the same directory as the your app assemblies/program. What was the exact command used to load it in LLDB?
Is there any way you could zip up the core dump and all your app binaries and attach them to this issue?
---
@kirpichiki commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353462274)
Thanks for the answer. I am generating coredump with the `createdump` utility using default parameters. Something like:
`/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump 6`
To start debugger I am using this command:
`lldb-4.0 -O "settings set target.exec-search-paths /app/" -o "plugin load /libsosplugin.so" --core /tmp/coredump.6 /usr/share/dotnet/dotnet`
Here `/app` - folder with my asp.net core app (dlls and pdbs), builded with `Release` configuration,
`/libsosplugin.so` - manually builded plugin
I'm afraid sources is pretty confidential, so I cant share it with you =(
---
@mikem8361 commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353471042)
This may be a bug in createdump. Try building the createdump from the coreclr master branch (I can also send it to you if you want) and/or try the “--full” option.
---
@ashishnegi commented on [Thu Dec 21 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353518069)
@mikem8361 I somehow missed earlier notification about dump.
I can confirm that sos ClrStack is working for me for few weeks.
I think I didn't do any thing new to make it work.
Otherwise, even then `sos EEStack -EE` was working for me.
We can close this thread now.
---
@kirpichiki commented on [Fri Dec 22 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353602857)
Yep, I see my symbols only with **full** dump. Thanks for your guidance @mikem8361. It's frustrating I have to load all 24 Gb of virtual memory just to watch stacktrace, but if thats only option I'll find some workaround.
---
@mikem8361 commented on [Fri Dec 22 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353654006)
Did you try the createdump built from the coreclr master branch (with the default options)? This will let me know that the bug/problem is fixed for 2.1. It should work with the 2.0.3 runtime.
[createdump.zip](https://github.com/dotnet/coreclr/files/1583248/createdump.zip)
---
@kirpichiki commented on [Mon Dec 25 2017](https://github.com/dotnet/coreclr/issues/14490#issuecomment-353860447)
@mikem8361 your createdump fails with
`relocation error: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump: symbol PAL_VirtualUnwindOutOfProc, version V1.0 not defined in file libmscordaccore.so with link time reference`
I compiled coreclr from the `master` and tried to run `createdump` with the default options - same result
---
@mikem8361 commented on [Tue Jan 02 2018](https://github.com/dotnet/coreclr/issues/14490#issuecomment-354854110)
Sorry about that. The standard createdump has some dependencies on libmscordaccore.so. Try this createdump:
[createdump.zip](https://github.com/dotnet/coreclr/files/1598487/createdump.zip)
---
@kirpichiki commented on [Wed Jan 10 2018](https://github.com/dotnet/coreclr/issues/14490#issuecomment-356557152)
No changes. Still getting !Unknown.
@ashishnegi commented on Fri Oct 13 2017
Hi I am trying to debug a crash dump on Linux. But I am not able to load any symbols.
Here are the steps I tried :
I downloaded coreclr symbols from : 2.0.0 download which are symbols.
I unzipped .symbols.nupkg into a folder with all .dbgs in one folder :
All *.so files are in folder
allso
.Dump is also present in same folder : dotnet.49126.dmp
I then start
lldb
with :(lldb) plugin load allso/libsosplugin.so (lldb) setsostid c049 1 Mapped sos OS tid 0xc049 to lldb thread index 1 (lldb) set ClrStack Ambiguous command 'set'. Possible matches: settings setclrpath setsostid (lldb) sos ClrStack OS Thread Id: 0xc049 (1) Child SP IP Call Site GetFrameContext failed: 1 0000000000000000 0000000000000000
(lldb) target modules list
[ 0] DAAB9A3E-C6DC-6F4B-E681-71B4F4A013FB-8B431DF0 /usr/share/dotnet/dotnet
[ 1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libdl-2.23.so
[ 2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0
[ 3] 144E588F-94CA-FAFD-BD0B-D1499C74190F-678DAD88 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
[ 4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libm-2.23.so
[ 5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1
[ 6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc-2.23.so
[ 7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2
[ 8] FA3EC34F-FF22-C058-FE6C-120AD8466ACE-A05DCEBB /usr/share/dotnet/host/fxr/2.0.0/libhostfxr.so
[ 9] 82FA3DDE-91A5-5710-F6CD-42274FF2701E-5A2E9BE3 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libhostpolicy.so
[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1
/lib/x86_64-linux-gnu/librt-2.23.so
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 13BF33CF-CE9A-8E60-360A-390912876AC2-75C8AD93 /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclrtraceptprovider.so
/home/asnegi/ryu/ryu/coreclr/runtimes/linux-x64/native/./libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 92F4379E-B144-FCB9-47CD-A120F46B67A7-A38D39F9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libclrjit.so
[ 22] F3FA2CBA-8B15-ABC5-E785-BE2B38B6F620-E4E617A4 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Globalization.Native.so
[ 23] 23A28B8C-35CF-747B-C507-EE9900D71790-47415517 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 6A9FD2EF-FD0E-B6D3-EE33-CAC7131B30E8-F6C2C6E1 /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 7B4BE8CE-1CC5-DCF8-629A-A008500A93F0-E77577A0 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Native.so
[ 27] 29F5E5B3-04BE-ED20-55CF-467E97187F20-6D9E3312 /usr/lib/x86_64-linux-gnu/libssh2.so.1
[ 28] 675F454A-D6FD-0B6C-A2E4-1127C7B98079-DA37F7B6 /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] 2DA08A7E-5BF6-1003-0DD3-3B70DB951399-626B7496 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 85CA824C-BE57-3813-B1E3-59894F743EBB-100AB034 /usr/lib/x86_64-linux-gnu/libxml2.so.2
[ 31] 523E5F7F-4262-BD15-56B1-57D57F13ED16-722E804E /lib/x86_64-linux-gnu/libbz2.so.1.0
[ 32] 340B7B46-3F98-1B8A-0FB3-451751F881DF-1B0C2F74 /lib/x86_64-linux-gnu/libz.so.1
[ 33] 2EB15945-45CC-A1B5-258D-D030B398DA17-4F4541DA /lib/x86_64-linux-gnu/libgcrypt.so.20
[ 34] 3A325585-F22D-AE6B-A918-CDC3D2B62760-F666FE16 /lib/x86_64-linux-gnu/libgpg-error.so.0
[ 35] CE5C34DF-F6A9-8121-F82E-ED4547861645-5801DB1D /usr/lib/x86_64-linux-gnu/libsnappy.so.1
/usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
[ 36] D5C47E54-D8F2-BA13-26E2-8B7C8BAB45FD-5B537660 /usr/lib/x86_64-linux-gnu/liblz4.so.1
/usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
[ 37] 85F3D1F1-BE82-5AF2-FC27-936CD9F113D4-636B0F61 /lib/x86_64-linux-gnu/libcgroup.so.1
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ readelf -n libcoreclr.so.dbg
Displaying notes found at file offset 0x00000200 with length 0x00000024: Owner Data size Description GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: f09424a9ee0addca898872e72b24a51357677af5
Displaying notes found at file offset 0x049e1d5c with length 0x0000001c: Owner Data size Description GNU 0x00000009 NT_GNU_GOLD_VERSION (gold version) Version: gold 1.11 asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $
[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so
(lldb) sos PrintException Exception object: 00007f5ec808ce00 Exception type: System.IO.FileNotFoundException Message: Could not load the specified file. InnerException:
StackTrace (generated):
SP IP Function
00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c
StackTraceString:
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
asnegi@asnegi-vm ~/ryu/dump $ lldb-3.9 -O "settings set target.exec-search-paths /home/asnegi/ryu/dump/" -c dotnet.49126.dmp /usr/share/dotnet/dotnet (lldb) settings set target.exec-search-paths /home/asnegi/ryu/dump/ (lldb) target create "/usr/share/dotnet/dotnet" --core "dotnet.49126.dmp" Core file '/home/asnegi/ryu/dump/dotnet.49126.dmp' (x86_64) was loaded. (lldb) plugin load libsosplugin.so libclrjit.so was compiled with optimization - stepping may behave oddly; variables may not be available. (lldb) (lldb) (lldb) sos Threads ThreadCount: 20 UnstartedThread: 0 BackgroundThread: 19 PendingThread: 0 DeadThread: 0 Hosted Runtime: no Lock ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception 11 1 bfef 000000000178BEC0 21220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Finalizer) 13 2 bfe6 00000000017A26B0 2020020 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 38 3 c01a 0000000001822D10 1021220 Preemptive 00007F5EC8652450:00007F5EC8652F60 0000000001788130 0 Ukn (Threadpool Worker) 46 4 c01c 00007F5EC0004BC0 1021220 Preemptive 00007F5EC8625A98:00007F5EC8626F60 0000000001788130 0 Ukn (Threadpool Worker) 57 5 c049 00007F5E5C00D360 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn System.IO.FileNotFoundException 00007f5ec808ce00 9 6 c045 00007F5E6401CF60 20220 Cooperative 00007F5EC86A5718:00007F5EC86A6F60 0000000001788130 0 Ukn 49 7 c0c2 00007F5EBC0066C0 1021220 Cooperative 00007F5EC865FC20:00007F5EC8660F60 0000000001788130 0 Ukn (Threadpool Worker) 31 8 c10e 00007F5E4C0044F0 1020220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Threadpool Worker) 55 9 c14e 00007F5E1C0009F0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 52 10 c153 00007F5E20000CD0 20220 Preemptive 00007F5EC8646EE0:00007F5EC8646F60 0000000001788130 0 Ukn 24 11 c744 00007F5EC055BEB0 1021220 Preemptive 00007F5EC865C438:00007F5EC865CF60 0000000001788130 0 Ukn (Threadpool Worker) 51 12 c746 00007F5EBC00C780 1021220 Preemptive 00007F5EC860ACE0:00007F5EC860AF60 0000000001788130 0 Ukn (Threadpool Worker) 53 13 c04c 00007F5E183178B0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 47 14 c04a 00007F5E5801B8D0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 37 15 c775 00007F5E480086D0 1021220 Cooperative 00007F5EC867CE60:00007F5EC867CF60 0000000001788130 0 Ukn (Threadpool Worker) 4 16 c776 00007F5DF41909B0 1021220 Cooperative 00007F5EC867D1A0:00007F5EC867EF60 0000000001788130 0 Ukn (Threadpool Worker) 20 17 c777 00007F5EBC011B30 1021220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Threadpool Worker) 21 18 c778 00007F5EC0560820 1021220 Preemptive 00007F5EC85FAEE0:00007F5EC85FAF60 0000000001788130 0 Ukn (Threadpool Worker) 27 19 c7f3 00007F5E6405FC50 21220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn 48 20 c953 00007F5DEC0009F0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (lldb) thread select 57 (lldb) * thread #57: tid = 49225, 0x00007f5f68ad8c1d libpthread.so.0
StackTrace (generated):
SP IP Function
00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c
__GI___nanosleep + 45, stop reason = signal SIGSEGV frame #0: 0x00007f5f68ad8c1d libpthread.so.0
GInanosleep + 45 libpthread.so.0`GInanosleep: -> 0x7f5f68ad8c1d <+45>: addb %al, (%rax) 0x7f5f68ad8c1f <+47>: addb %al, (%rax) 0x7f5f68ad8c21 <+49>: addb %al, (%rax) 0x7f5f68ad8c23 <+51>: addb %al, (%rax) (lldb) sos PrintException Exception object: 00007f5ec808ce00 Exception type: System.IO.FileNotFoundException Message: Could not load the specified file. InnerException:StackTraceString:
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
(lldb) thread select 1
(lldb) * thread #1: tid = 49481, 0x00007f5f4af52fd9 libclrjit.so) + 249 at context.cpp:545 [opt]
(lldb) sos PrintException
The current thread is unmanaged
(lldb) sos ClrStack
OS Thread Id: 0xc149 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
Failed to start stack walk: 80070057
(lldb)
::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545, name = 'dotnet', stop reason = signal SIGSEGV frame #0: 0x00007f5f4af52fd9 libclrjit.so
::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=OS Thread Id: 0x7f (1) Child SP IP Call Site 00007F85347F7210 00007f8808245360 [GCFrame: 00007f85347f7210] 00007F85347F7300 00007f8808245360 [HelperMethodFrame_1OBJ: 00007f85347f7300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object) 00007F85347F7430 00007F878D45B503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) 00007F85347F74C0 00007F878D41B6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) 00007F85347F7520 00007F878D41B559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) 00007F85347F75E0 00007F878D41B273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken) 00007F85347F7610 00007F87914B4BFC /app/Hangfire.Mongo.dll!Unknown 00007F85347F76F0 00007F8791486F2A /app/Hangfire.Core.dll!Unknown 00007F85347F7750 00007F8791486E8D /app/Hangfire.Core.dll!Unknown 00007F85347F7770 00007F8791486A8D /app/Hangfire.Core.dll!Unknown 00007F85347F77F0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F85347F7840 00007F878D41AB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) 00007F85347F78E0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F85347F7C58 00007f8806b94067 [GCFrame: 00007f85347f7c58] 00007F85347F7D30 00007f8806b94067 [DebuggerU2MCatchHandlerFrame: 00007f85347f7d30]