Closed ashishnegi closed 4 years ago
cc @lt72
Try adding -O "settings set target.exec-search-paths $HOME/ryu/ryu/coreclr/runtimes/linux-x64/native"
to your lldb command line. This is neccessary if the .dbg files are not in the same place as the where they were in the core dump. If "target modules list -s" still doesn't show the .dbg file for libcoreclr.so then it maybe that *.dbg is the wrong version some how. You can use readelf on the libcoreclr.dbg file to get the build id and see if it matches the one in the target modules command (FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119
).
The reason clrstack isn't working is because you need to try another thread index. The setsosid command is a hacky workaround for a lldb 3.6 bug that requires guessing which thread index maps to which OS thread id.
lldb 3.9 has this problem fixed, but you need an libsosplugin.so that works with 3.9. You can try the one I attached (built on ubuntu 14.04 but should work on 16.xx).
@ashishnegi is this resolved for you now? If so, let's close it.
Hi, I will be able to update on this thread in next 3 days. Thanks for your support.
I see that build id is different for me :
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ readelf -n libcoreclr.so.dbg
Displaying notes found at file offset 0x00000200 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: f09424a9ee0addca898872e72b24a51357677af5
Displaying notes found at file offset 0x049e1d5c with length 0x0000001c:
Owner Data size Description
GNU 0x00000009 NT_GNU_GOLD_VERSION (gold version)
Version: gold 1.11
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $
and in target modules list
:
[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so
target modules list -s
does not show any libcoreclr.so.dbg.
I see one libcoreclrtraceptprovider.so.dbg
in target moduels list -s
. readelf shows same target id as that of target modules list
.
so, I think my symbols are not right.
Also, I think lldb-3.9 will not be able to help here as symbols are not right.
FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119
of coreclr.so ?One question :
In lldb, executing target modules list
gives me FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119
as build id. This is same as my local installed dotnet's build id. This dotnet has version : 2.0.0.
Is lldb showing build id of dotnet which was crashed on the remote machine Or build id of dotnet present in my local machine ?
"target modules list -u -s" is giving you the build id of the modules in the core dump not what is on your local machine. Your build id of the libcoreclr.so in the dump doesn't match libcoreclr.so.dbg so lldb doesn't load it. What version of .NET Core was on the remote machine that the core dump was generated? It could be either 1.x or maybe a 2.0.x service release which could be the reason for the mismatch.
It does look like the symbol zip files don't match the product. I'm looking into a solution. Not sure what time frame they will be fixed.
@mikem8361 FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119
is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.
So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0 ?
I will try building coreclr locally on linux to get the symbols. I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.
We are working on fixing them now.
-------- Original message -------- From: Ashish Negi notifications@github.com Date: 10/30/17 8:21 PM (GMT-08:00) To: dotnet/coreclr coreclr@noreply.github.com Cc: Mike McLaughlin mikem@microsoft.com, Mention mention@noreply.github.com Subject: Re: [dotnet/coreclr] Debugging linux dump (#14490)
@mikem8361https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikem8361&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=Xz3mCMRORKdIKrsAhZ1yDaKAg1uKDVVvvRO0e67MtvA%3D&reserved=0 FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0. My local install has 2.0.0 and same build id.
So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcore%2Fblob%2Fmaster%2Frelease-notes%2Fdownload-archives%2F2.0.0-download.md&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=hW%2B4p7uMStSDr7NxPAUsM8W9whKD021XlieiXU2Wpls%3D&reserved=0 ?
I will try building coreclr locally on linux to get the symbols. I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fissues%2F14490%23issuecomment-340648585&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=PCOH6Z%2FFWAn3jbbEy%2Fs3pPzerB2PVFOGC4pBAYJUTNc%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJ-Fa_MRtvKb4Cs-KnDmIpW1IiEypX1pks5sxpJJgaJpZM4P4Rvp&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=RQ3N%2B1u1C3TBuPPwd4Xwtu8gUnOeQyLkl4SwEc3%2B6kM%3D&reserved=0.
For now if you get the runtime.linux-x64.Microsoft.NETCore.App.2.0.0.symbols.nupkg package from the core-setup symbol zip, it will match the 2.0.0 binaries in your core dump.
The download site symbol link has been fixed for 2.0.0 and 2.0.2.
Hi @mikem8361 Thanks for sosplugin.
I made good progress.
I installed lldb-3.9 on Ubuntu 16.04 and used your libsosplugin.so.
sos Threads
shows real os thread id.
I switched to it. I can use sos PrintException
and it works.
I am still not able to get any stack on sos ClrStack
. Can you help understand why that's happening ?
(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type: System.IO.FileNotFoundException
Message: Could not load the specified file.
InnerException: <none>
StackTrace (generated):
SP IP Function
00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c
StackTraceString: <none>
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>
Here is my full debug session :
asnegi@asnegi-vm ~/ryu/dump $ lldb-3.9 -O "settings set target.exec-search-paths /home/asnegi/ryu/dump/" -c dotnet.49126.dmp /usr/share/dotnet/dotnet
(lldb) settings set target.exec-search-paths /home/asnegi/ryu/dump/
(lldb) target create "/usr/share/dotnet/dotnet" --core "dotnet.49126.dmp"
Core file '/home/asnegi/ryu/dump/dotnet.49126.dmp' (x86_64) was loaded.
(lldb) plugin load libsosplugin.so
libclrjit.so was compiled with optimization - stepping may behave oddly; variables may not be available.
(lldb) (lldb)
(lldb) sos Threads
ThreadCount: 20
UnstartedThread: 0
BackgroundThread: 19
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
Lock
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
11 1 bfef 000000000178BEC0 21220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Finalizer)
13 2 bfe6 00000000017A26B0 2020020 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn
38 3 c01a 0000000001822D10 1021220 Preemptive 00007F5EC8652450:00007F5EC8652F60 0000000001788130 0 Ukn (Threadpool Worker)
46 4 c01c 00007F5EC0004BC0 1021220 Preemptive 00007F5EC8625A98:00007F5EC8626F60 0000000001788130 0 Ukn (Threadpool Worker)
57 5 c049 00007F5E5C00D360 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn System.IO.FileNotFoundException 00007f5ec808ce00
9 6 c045 00007F5E6401CF60 20220 Cooperative 00007F5EC86A5718:00007F5EC86A6F60 0000000001788130 0 Ukn
49 7 c0c2 00007F5EBC0066C0 1021220 Cooperative 00007F5EC865FC20:00007F5EC8660F60 0000000001788130 0 Ukn (Threadpool Worker)
31 8 c10e 00007F5E4C0044F0 1020220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Threadpool Worker)
55 9 c14e 00007F5E1C0009F0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn
52 10 c153 00007F5E20000CD0 20220 Preemptive 00007F5EC8646EE0:00007F5EC8646F60 0000000001788130 0 Ukn
24 11 c744 00007F5EC055BEB0 1021220 Preemptive 00007F5EC865C438:00007F5EC865CF60 0000000001788130 0 Ukn (Threadpool Worker)
51 12 c746 00007F5EBC00C780 1021220 Preemptive 00007F5EC860ACE0:00007F5EC860AF60 0000000001788130 0 Ukn (Threadpool Worker)
53 13 c04c 00007F5E183178B0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn
47 14 c04a 00007F5E5801B8D0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn
37 15 c775 00007F5E480086D0 1021220 Cooperative 00007F5EC867CE60:00007F5EC867CF60 0000000001788130 0 Ukn (Threadpool Worker)
4 16 c776 00007F5DF41909B0 1021220 Cooperative 00007F5EC867D1A0:00007F5EC867EF60 0000000001788130 0 Ukn (Threadpool Worker)
20 17 c777 00007F5EBC011B30 1021220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn (Threadpool Worker)
21 18 c778 00007F5EC0560820 1021220 Preemptive 00007F5EC85FAEE0:00007F5EC85FAF60 0000000001788130 0 Ukn (Threadpool Worker)
27 19 c7f3 00007F5E6405FC50 21220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn
48 20 c953 00007F5DEC0009F0 20220 Preemptive 0000000000000000:0000000000000000 0000000001788130 0 Ukn
(lldb) thread select 57
(lldb) * thread dotnet/runtime#3878: tid = 49225, 0x00007f5f68ad8c1d libpthread.so.0`__GI___nanosleep + 45, stop reason = signal SIGSEGV
frame #0: 0x00007f5f68ad8c1d libpthread.so.0`__GI___nanosleep + 45
libpthread.so.0`__GI___nanosleep:
-> 0x7f5f68ad8c1d <+45>: addb %al, (%rax)
0x7f5f68ad8c1f <+47>: addb %al, (%rax)
0x7f5f68ad8c21 <+49>: addb %al, (%rax)
0x7f5f68ad8c23 <+51>: addb %al, (%rax)
(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type: System.IO.FileNotFoundException
Message: Could not load the specified file.
InnerException: <none>
StackTrace (generated):
SP IP Function
00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c
StackTraceString: <none>
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>
(lldb) thread select 1
(lldb) * thread dotnet/coreclr#1: tid = 49481, 0x00007f5f4af52fd9 libclrjit.so`::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545, name = 'dotnet', stop reason = signal SIGSEGV
frame #0: 0x00007f5f4af52fd9 libclrjit.so`::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545 [opt]
(lldb) sos PrintException
The current thread is unmanaged
(lldb) sos ClrStack
OS Thread Id: 0xc149 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
Failed to start stack walk: 80070057
(lldb)
@mikem8361 to investigate
@ashishnegi: Is there anyway you could attach the core dump? I would need it to debug why clrstack isn't working.
Hi everyone and sorry for invasion. I have another situation with debugging core dump. I can't determine how to load symbols for my own code.
OS Thread Id: 0x4b (1) Child SP IP Call Site 00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210] 00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object) 00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) 00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) 00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) 00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken) 00007F78D37FD610 00007F7B39A4725C /app/Hangfire.Mongo.dll!Unknown 00007F78D37FD6F0 00007F7B39A19AEA /app/Hangfire.Core.dll!Unknown 00007F78D37FD750 00007F7B39A19A4D /app/Hangfire.Core.dll!Unknown 00007F78D37FD770 00007F7B39A1964D /app/Hangfire.Core.dll!Unknown 00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) 00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58] 00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]
OS Thread Id: 0x4b (1) Child SP IP Call Site 00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210] 00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object) 00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) 00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) 00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) 00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken) 00007F78D37FD610 00007F7B39A4725C Hangfire.Mongo.Signal.Mongo.MongoSignal.Listen(System.Threading.CancellationToken) 00007F78D37FD6F0 00007F7B39A19AEA Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext) 00007F78D37FD750 00007F7B39A19A4D Hangfire.Server.InfiniteLoopProcess.Execute(Hangfire.Server.BackgroundProcessContext) 00007F78D37FD770 00007F7B39A1964D Hangfire.Server.ServerProcessExtensions.RunProcess(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext) 00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) 00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58] 00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]
[ 0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A /usr/share/dotnet/dotnet [ 1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2 [ 2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0 [ 3] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 [ 4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6 [ 5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1 [ 6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6 [ 7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2 [ 8] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10 /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so [ 9] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so [ 10] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg [ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1 [ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8 [ 13] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A /lib/x86_64-linux-gnu/libuuid.so.1 [ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8 [ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5 [ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg [ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 [ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 [ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4 [ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4 [ 21] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so [ 22] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg [ 23] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1 [ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55 [ 25] 628F069D-8645-434A-D88E-502966359B23-213456EB /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1 [ 26] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so [ 27] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so [ 28] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F /lib/x86_64-linux-gnu/libssl.so.1.0.0 [ 29] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [ 30] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6 /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so [ 31] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so [ 32] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6 /usr/lib/x86_64-linux-gnu/libcurl.so.4 [ 33] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57 /usr/lib/x86_64-linux-gnu/libidn.so.11 [ 34] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF /usr/lib/x86_64-linux-gnu/librtmp.so.1 [ 35] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 [ 36] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 [ 37] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [ 38] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832 /lib/x86_64-linux-gnu/libz.so.1 [ 39] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [ 40] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF /usr/lib/x86_64-linux-gnu/libhogweed.so.4 [ 41] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54 /usr/lib/x86_64-linux-gnu/libnettle.so.6 [ 42] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620 /usr/lib/x86_64-linux-gnu/libgmp.so.10 [ 43] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F /usr/lib/x86_64-linux-gnu/libkrb5.so.3 [ 44] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 [ 45] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D /lib/x86_64-linux-gnu/libcom_err.so.2 [ 46] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 [ 47] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2 /lib/x86_64-linux-gnu/libresolv.so.2 [ 48] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC /usr/lib/x86_64-linux-gnu/libsasl2.so.2 [ 49] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3 /usr/lib/x86_64-linux-gnu/libgssapi.so.3 [ 50] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 [ 51] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3 /usr/lib/x86_64-linux-gnu/libtasn1.so.6 [ 52] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6 /lib/x86_64-linux-gnu/libkeyutils.so.1 [ 53] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 [ 54] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357 /usr/lib/x86_64-linux-gnu/libkrb5.so.26 [ 55] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1 /usr/lib/x86_64-linux-gnu/libasn1.so.8 [ 56] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 [ 57] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818 /usr/lib/x86_64-linux-gnu/libroken.so.18 [ 58] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F /usr/lib/x86_64-linux-gnu/libffi.so.6 [ 59] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700 /usr/lib/x86_64-linux-gnu/libwind.so.0 [ 60] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91 /usr/lib/x86_64-linux-gnu/libheimbase.so.1 [ 61] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7 /usr/lib/x86_64-linux-gnu/libhx509.so.5 [ 62] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 [ 63] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9 /lib/x86_64-linux-gnu/libcrypt.so.1 [ 64] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752 /lib/x86_64-linux-gnu/libnss_files.so.2 [ 65] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416 /lib/x86_64-linux-gnu/libnss_dns.so.2
[vdso] (0x00007fff2978d000)
[ 0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A /usr/share/dotnet/dotnet [ 1] 41FD7D4C-745E-681C-706B-ED73B4CC46C3-E422C5E6 0x00007fff2978d000 [vdso] (0x00007fff2978d000) [ 2] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2 [ 3] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0 [ 4] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 [ 5] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6 [ 6] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1 [ 7] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6 [ 8] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2 [ 9] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10 /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so [ 10] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so [ 11] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg [ 12] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1 [ 13] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8 [ 14] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A /lib/x86_64-linux-gnu/libuuid.so.1 [ 15] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8 [ 16] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5 [ 17] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg [ 18] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 [ 19] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 [ 20] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4 [ 21] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4 [ 22] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so [ 23] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg [ 24] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1 [ 25] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55 [ 26] 628F069D-8645-434A-D88E-502966359B23-213456EB /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1 [ 27] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so [ 28] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so [ 29] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F /lib/x86_64-linux-gnu/libssl.so.1.0.0 [ 30] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [ 31] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6 /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so [ 32] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so [ 33] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6 /usr/lib/x86_64-linux-gnu/libcurl.so.4 [ 34] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57 /usr/lib/x86_64-linux-gnu/libidn.so.11 [ 35] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF /usr/lib/x86_64-linux-gnu/librtmp.so.1 [ 36] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 [ 37] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 [ 38] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [ 39] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832 /lib/x86_64-linux-gnu/libz.so.1 [ 40] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [ 41] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF /usr/lib/x86_64-linux-gnu/libhogweed.so.4 [ 42] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54 /usr/lib/x86_64-linux-gnu/libnettle.so.6 [ 43] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620 /usr/lib/x86_64-linux-gnu/libgmp.so.10 [ 44] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F /usr/lib/x86_64-linux-gnu/libkrb5.so.3 [ 45] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 [ 46] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D /lib/x86_64-linux-gnu/libcom_err.so.2 [ 47] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 [ 48] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2 /lib/x86_64-linux-gnu/libresolv.so.2 [ 49] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC /usr/lib/x86_64-linux-gnu/libsasl2.so.2 [ 50] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3 /usr/lib/x86_64-linux-gnu/libgssapi.so.3 [ 51] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 [ 52] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3 /usr/lib/x86_64-linux-gnu/libtasn1.so.6 [ 53] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6 /lib/x86_64-linux-gnu/libkeyutils.so.1 [ 54] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 [ 55] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357 /usr/lib/x86_64-linux-gnu/libkrb5.so.26 [ 56] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1 /usr/lib/x86_64-linux-gnu/libasn1.so.8 [ 57] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 [ 58] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818 /usr/lib/x86_64-linux-gnu/libroken.so.18 [ 59] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F /usr/lib/x86_64-linux-gnu/libffi.so.6 [ 60] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700 /usr/lib/x86_64-linux-gnu/libwind.so.0 [ 61] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91 /usr/lib/x86_64-linux-gnu/libheimbase.so.1 [ 62] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7 /usr/lib/x86_64-linux-gnu/libhx509.so.5 [ 63] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 [ 64] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9 /lib/x86_64-linux-gnu/libcrypt.so.1 [ 65] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752 /lib/x86_64-linux-gnu/libnss_files.so.2 [ 66] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416 /lib/x86_64-linux-gnu/libnss_dns.so.2
If my conclution is right, how can I load vdso while dedugging core dump? @mikem8361 Won't you mind to gide me how to deal with this situation?
Misc info:
docker
container with Ubuntu:xenial
basedotnet
version is 2.0.3
lldb
version is 4.0
with manually builded libsosplugin.so
settings set target.exec-search-paths
to folder with my program dlls seems to have no effectI don't think vso has anything to do with your problem. I think sos is having a problem finding the pdb's for your app or having problem unwinding the managed frames in the core dump. How did you generate the core dump? sos should load the portable pdbs from the same directory as the your app assemblies/program. What was the exact command used to load it in LLDB?
Is there any way you could zip up the core dump and all your app binaries and attach them to this issue?
Thanks for the answer. I am generating coredump with the createdump
utility using default parameters. Something like:
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump 6
To start debugger I am using this command:
lldb-4.0 -O "settings set target.exec-search-paths /app/" -o "plugin load /libsosplugin.so" --core /tmp/coredump.6 /usr/share/dotnet/dotnet
Here /app
- folder with my asp.net core app (dlls and pdbs), builded with Release
configuration,
/libsosplugin.so
- manually builded plugin
I'm afraid sources is pretty confidential, so I cant share it with you =(
This may be a bug in createdump. Try building the createdump from the coreclr master branch (I can also send it to you if you want) and/or try the “--full” option.
@mikem8361 I somehow missed earlier notification about dump.
I can confirm that sos ClrStack is working for me for few weeks.
I think I didn't do any thing new to make it work.
Otherwise, even then sos EEStack -EE
was working for me.
We can close this thread now.
Yep, I see my symbols only with full dump. Thanks for your guidance @mikem8361. It's frustrating I have to load all 24 Gb of virtual memory just to watch stacktrace, but if thats only option I'll find some workaround.
Did you try the createdump built from the coreclr master branch (with the default options)? This will let me know that the bug/problem is fixed for 2.1. It should work with the 2.0.3 runtime.
@mikem8361 your createdump fails with
relocation error: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump: symbol PAL_VirtualUnwindOutOfProc, version V1.0 not defined in file libmscordaccore.so with link time reference
I compiled coreclr from the master
and tried to run createdump
with the default options - same result
OS Thread Id: 0x7f (1)
Child SP IP Call Site
00007F85347F7210 00007f8808245360 [GCFrame: 00007f85347f7210]
00007F85347F7300 00007f8808245360 [HelperMethodFrame_1OBJ: 00007f85347f7300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F85347F7430 00007F878D45B503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F85347F74C0 00007F878D41B6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F85347F7520 00007F878D41B559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F85347F75E0 00007F878D41B273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F85347F7610 00007F87914B4BFC /app/Hangfire.Mongo.dll!Unknown
00007F85347F76F0 00007F8791486F2A /app/Hangfire.Core.dll!Unknown
00007F85347F7750 00007F8791486E8D /app/Hangfire.Core.dll!Unknown
00007F85347F7770 00007F8791486A8D /app/Hangfire.Core.dll!Unknown
00007F85347F77F0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F85347F7840 00007F878D41AB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F85347F78E0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F85347F7C58 00007f8806b94067 [GCFrame: 00007f85347f7c58]
00007F85347F7D30 00007f8806b94067 [DebuggerU2MCatchHandlerFrame: 00007f85347f7d30]
Sorry about that. The standard createdump has some dependencies on libmscordaccore.so. Try this createdump:
No changes. Still getting !Unknown.
Issue moved to dotnet/diagnostics dotnet/coreclr#48 via ZenHub
Hi I am trying to debug a crash dump on Linux. But I am not able to load any symbols.
Here are the steps I tried :
I downloaded coreclr symbols from : 2.0.0 download which are symbols.
I unzipped .symbols.nupkg into a folder with all .dbgs in one folder :
All *.so files are in folder
allso
.Dump is also present in same folder : dotnet.49126.dmp
I then start
lldb
with :(lldb) plugin load allso/libsosplugin.so (lldb) setsostid c049 1 Mapped sos OS tid 0xc049 to lldb thread index 1 (lldb) set ClrStack Ambiguous command 'set'. Possible matches: settings setclrpath setsostid (lldb) sos ClrStack OS Thread Id: 0xc049 (1) Child SP IP Call Site GetFrameContext failed: 1 0000000000000000 0000000000000000
(lldb) target modules list
[ 0] DAAB9A3E-C6DC-6F4B-E681-71B4F4A013FB-8B431DF0 /usr/share/dotnet/dotnet
[ 1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032 /lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libdl-2.23.so
[ 2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2 /lib/x86_64-linux-gnu/libpthread.so.0
[ 3] 144E588F-94CA-FAFD-BD0B-D1499C74190F-678DAD88 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
[ 4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8 /lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libm-2.23.so
[ 5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434 /lib/x86_64-linux-gnu/libgcc_s.so.1
[ 6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518 /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc-2.23.so
[ 7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA /lib64/ld-linux-x86-64.so.2
[ 8] FA3EC34F-FF22-C058-FE6C-120AD8466ACE-A05DCEBB /usr/share/dotnet/host/fxr/2.0.0/libhostfxr.so
[ 9] 82FA3DDE-91A5-5710-F6CD-42274FF2701E-5A2E9BE3 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libhostpolicy.so
[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9 /lib/x86_64-linux-gnu/librt.so.1
/lib/x86_64-linux-gnu/librt-2.23.so
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1 /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 13BF33CF-CE9A-8E60-360A-390912876AC2-75C8AD93 /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260 /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclrtraceptprovider.so
/home/asnegi/ryu/ryu/coreclr/runtimes/linux-x64/native/./libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 92F4379E-B144-FCB9-47CD-A120F46B67A7-A38D39F9 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libclrjit.so
[ 22] F3FA2CBA-8B15-ABC5-E785-BE2B38B6F620-E4E617A4 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Globalization.Native.so
[ 23] 23A28B8C-35CF-747B-C507-EE9900D71790-47415517 /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 6A9FD2EF-FD0E-B6D3-EE33-CAC7131B30E8-F6C2C6E1 /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 7B4BE8CE-1CC5-DCF8-629A-A008500A93F0-E77577A0 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Native.so
[ 27] 29F5E5B3-04BE-ED20-55CF-467E97187F20-6D9E3312 /usr/lib/x86_64-linux-gnu/libssh2.so.1
[ 28] 675F454A-D6FD-0B6C-A2E4-1127C7B98079-DA37F7B6 /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] 2DA08A7E-5BF6-1003-0DD3-3B70DB951399-626B7496 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 85CA824C-BE57-3813-B1E3-59894F743EBB-100AB034 /usr/lib/x86_64-linux-gnu/libxml2.so.2
[ 31] 523E5F7F-4262-BD15-56B1-57D57F13ED16-722E804E /lib/x86_64-linux-gnu/libbz2.so.1.0
[ 32] 340B7B46-3F98-1B8A-0FB3-451751F881DF-1B0C2F74 /lib/x86_64-linux-gnu/libz.so.1
[ 33] 2EB15945-45CC-A1B5-258D-D030B398DA17-4F4541DA /lib/x86_64-linux-gnu/libgcrypt.so.20
[ 34] 3A325585-F22D-AE6B-A918-CDC3D2B62760-F666FE16 /lib/x86_64-linux-gnu/libgpg-error.so.0
[ 35] CE5C34DF-F6A9-8121-F82E-ED4547861645-5801DB1D /usr/lib/x86_64-linux-gnu/libsnappy.so.1
/usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
[ 36] D5C47E54-D8F2-BA13-26E2-8B7C8BAB45FD-5B537660 /usr/lib/x86_64-linux-gnu/liblz4.so.1
/usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
[ 37] 85F3D1F1-BE82-5AF2-FC27-936CD9F113D4-636B0F61 /lib/x86_64-linux-gnu/libcgroup.so.1