search

openresty / openresty-gdb-utils

GDB Utilities for OpenResty (including Nginx, ngx_lua, LuaJIT, and more)
336 stars 89 forks source link

I got "No global L located (tried globalL and ngx_cycle)" When I run lgcstat,any ngx-gdb-cmds #8

Closed idevz closed 8 years ago

idevz commented 8 years ago

I got fail like infos below, When I run any ngx-gdb-cmd in my Mac OSX:

 /Users/zj-git/idevz.org/ [master*] gdb -p 32957
GNU gdb (GDB) 7.9
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin15.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 32957
Reading symbols from /usr/local/nginx_x/nginx/sbin/nginx...done.
0x00007fff8d81dfc6 in kevent () from /usr/lib/system/libsystem_kernel.dylib
(gdb) lvmst
No global L located (tried globalL and ngx_cycle)
(gdb)

my OSX is 10.11.2 (15C50).

I installed OpenResty with those options:

./configure --with-cc-opt='-I/usr/local/Cellar/pcre/8.37/include/' --with-ld-opt='-L/usr/local/Cellar/pcre/8.37/lib' --with-openssl=/Users/zhoujing/Downloads/openssl-1.0.2e  --with-debug --with-dtrace=/usr/sbin/dtrace --prefix=/usr/local/nginx_x

and make like:

make CCDEBUG=-g
make CCDEBUG=-g install

even though i renew the CCDEBUG option in luajit source code /Users/zhoujing/Downloads/ngx_openresty-1.9.3.2/bundle/LuaJIT-2.1-20151028/src/Makefile, I still got fail.

here are many info I run in gdb. what should I do to solve this No global L located (tried globalL and ngx_cycle)mistake? Thanks every much.

(gdb) bt

(gdb) bt
#0  0x00007fff8d81dfc6 in kevent () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00000001088e5c3a in ngx_kqueue_process_events (cycle=0x7fa441007c50, timer=18446744073709551615, flags=1) at src/event/modules/ngx_kqueue_module.c:536
#2  0x00000001088dc363 in ngx_process_events_and_timers (cycle=0xffffffffffffffff) at src/event/ngx_event.c:242
#3  0x00000001088e470e in ngx_worker_process_cycle (cycle=0x0, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:769
#4  0x00000001088e2980 in ngx_spawn_process (cycle=0x1, proc=0x1088e4600 <ngx_worker_process_cycle>, data=0x0, name=0x108afc60c <ngx_crypt_to64.itoa64+5228> "worker process",
    respawn=2) at src/os/unix/ngx_process.c:198
#5  0x00000001088e3f5c in ngx_start_worker_processes (cycle=0xffffffffffffffff, n=2, type=1) at src/os/unix/ngx_process_cycle.c:358
#6  0x00000001088e351f in ngx_master_process_cycle (cycle=0x1) at src/os/unix/ngx_process_cycle.c:130
#7  0x00000001088c19be in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:415

(gdb) info shared

(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x000080000d8d29a0  0x000080000d96e968  Yes         /usr/local/nginx_x/luajit/lib/libluajit-5.1.2.dylib
0x00007fff89cd1989  0x00007fff89cd22c0  Yes (*)     /usr/lib/libSystem.B.dylib
0x0000000108d4f590  0x0000000108db42ec  Yes (*)     /usr/local/lib/libpcre.1.dylib
0x00007fff8cf1fae0  0x00007fff8cf31200  Yes (*)     /usr/lib/libz.1.dylib
0x00007fff939128c0  0x00007fff93917160  Yes (*)     /usr/lib/system/libcache.dylib
0x00007fff84d7b164  0x00007fff84d87640  Yes (*)     /usr/lib/system/libcommonCrypto.dylib
0x00007fff80dd8d24  0x00007fff80de10c0  Yes (*)     /usr/lib/system/libcompiler_rt.dylib
0x00007fff878054a8  0x00007fff8780d6b0  Yes (*)     /usr/lib/system/libcopyfile.dylib
0x00007fff81fa46c0  0x00007fff82022e10  Yes (*)     /usr/lib/system/libcorecrypto.dylib
0x00007fff8160ab30  0x00007fff8164b614  Yes (*)     /usr/lib/system/libdispatch.dylib
0x00007fff80f910e8  0x00007fff80f944b0  Yes (*)     /usr/lib/system/libdyld.dylib
0x00007fff821dba19  0x00007fff821dc0b8  Yes (*)     /usr/lib/system/libkeymgr.dylib
0x00007fff910bbfc0  0x00007fff910bbff8  Yes (*)     /usr/lib/system/liblaunch.dylib
0x00007fff8e86f11c  0x00007fff8e874790  Yes (*)     /usr/lib/system/libmacho.dylib
0x00007fff87c5b210  0x00007fff87c5d2b0  Yes (*)     /usr/lib/system/libquarantine.dylib
0x00007fff910b9924  0x00007fff910bb160  Yes (*)     /usr/lib/system/libremovefile.dylib
0x00007fff8497a190  0x00007fff84992224  Yes (*)     /usr/lib/system/libsystem_asl.dylib
0x00007fff87cc0340  0x00007fff87cc16a2  Yes (*)     /usr/lib/system/libsystem_blocks.dylib
0x00007fff87fad020  0x00007fff88043c10  Yes (*)     /usr/lib/system/libsystem_c.dylib
0x00007fff93048df0  0x00007fff9304b4e9  Yes (*)     /usr/lib/system/libsystem_configuration.dylib
0x00007fff93886b04  0x00007fff93889738  Yes (*)     /usr/lib/system/libsystem_coreservices.dylib
0x00007fff948e3a38  0x00007fff948fb121  Yes (*)     /usr/lib/system/libsystem_coretls.dylib
0x00007fff90e825c8  0x00007fff90e8a36c  Yes (*)     /usr/lib/system/libsystem_dnssd.dylib
0x00007fff908b9490  0x00007fff908e4658  Yes (*)     /usr/lib/system/libsystem_info.dylib
0x00007fff8d806a20  0x00007fff8d827928  Yes (*)     /usr/lib/system/libsystem_kernel.dylib
0x00007fff928237c0  0x00007fff9285324c  Yes (*)     /usr/lib/system/libsystem_m.dylib
0x00007fff93565ef0  0x00007fff9358270d  Yes (*)     /usr/lib/system/libsystem_malloc.dylib
0x00007fff90e8b490  0x00007fff90ef26c0  Yes (*)     /usr/lib/system/libsystem_network.dylib
0x00007fff821fdd91  0x00007fff82207048  Yes (*)     /usr/lib/system/libsystem_networkextension.dylib
0x00007fff8d83587b  0x00007fff8d83f440  Yes (*)     /usr/lib/system/libsystem_notify.dylib
0x00007fff87837920  0x00007fff878404dc  Yes (*)     /usr/lib/system/libsystem_platform.dylib
0x00007fff8371df28  0x00007fff8372c120  Yes (*)     /usr/lib/system/libsystem_pthread.dylib
0x00007fff81f9d7d4  0x00007fff81fa062c  Yes (*)     /usr/lib/system/libsystem_sandbox.dylib
0x00007fff81607ac7  0x00007fff816091f0  Yes (*)     /usr/lib/system/libsystem_secinit.dylib
0x00007fff94877ab4  0x00007fff94889bd4  Yes (*)     /usr/lib/system/libsystem_trace.dylib
0x00007fff8f782fc0  0x00007fff8f782ff8  Yes (*)     /usr/lib/system/libunc.dylib
0x00007fff89443918  0x00007fff89449a9a  Yes (*)     /usr/lib/system/libunwind.dylib
0x00007fff8785fd44  0x00007fff8788dd58  Yes (*)     /usr/lib/system/libxpc.dylib
0x00007fff849ad000  0x00007fff84dd9fa8  Yes (*)     /usr/lib/libobjc.A.dylib
0x00007fff9177fda0  0x00007fff917c7090  Yes (*)     /usr/lib/libauto.dylib
0x00007fff83a1ee80  0x00007fff83a49408  Yes (*)     /usr/lib/libc++abi.dylib
0x00007fff8e794f90  0x00007fff8e7efc31  Yes (*)     /usr/lib/libc++.1.dylib
0x00007fff9299812f  0x00007fff929991e0  Yes (*)     /usr/lib/libDiagnosticMessagesClient.dylib

(gdb) sharedlibrary lib

(gdb) sharedlibrary lib Symbols already loaded for /usr/local/nginx_x/luajit/lib/libluajit-5.1.2.dylib Symbols already loaded for /usr/lib/libSystem.B.dylib Symbols already loaded for /usr/local/lib/libpcre.1.dylib Symbols already loaded for /usr/lib/libz.1.dylib Symbols already loaded for /usr/lib/system/libcache.dylib Symbols already loaded for /usr/lib/system/libcommonCrypto.dylib Symbols already loaded for /usr/lib/system/libcompiler_rt.dylib Symbols already loaded for /usr/lib/system/libcopyfile.dylib Symbols already loaded for /usr/lib/system/libcorecrypto.dylib Symbols already loaded for /usr/lib/system/libdispatch.dylib Symbols already loaded for /usr/lib/system/libdyld.dylib Symbols already loaded for /usr/lib/system/libkeymgr.dylib Symbols already loaded for /usr/lib/system/liblaunch.dylib Symbols already loaded for /usr/lib/system/libmacho.dylib Symbols already loaded for /usr/lib/system/libquarantine.dylib Symbols already loaded for /usr/lib/system/libremovefile.dylib Symbols already loaded for /usr/lib/system/libsystem_asl.dylib Symbols already loaded for /usr/lib/system/libsystem_blocks.dylib Symbols already loaded for /usr/lib/system/libsystem_c.dylib Symbols already loaded for /usr/lib/system/libsystem_configuration.dylib Symbols already loaded for /usr/lib/system/libsystem_coreservices.dylib Symbols already loaded for /usr/lib/system/libsystem_coretls.dylib Symbols already loaded for /usr/lib/system/libsystem_dnssd.dylib Symbols already loaded for /usr/lib/system/libsystem_info.dylib Symbols already loaded for /usr/lib/system/libsystem_kernel.dylib Symbols already loaded for /usr/lib/system/libsystem_m.dylib Symbols already loaded for /usr/lib/system/libsystem_malloc.dylib Symbols already loaded for /usr/lib/system/libsystem_network.dylib Symbols already loaded for /usr/lib/system/libsystem_networkextension.dylib Symbols already loaded for /usr/lib/system/libsystem_notify.dylib Symbols already loaded for /usr/lib/system/libsystem_platform.dylib Symbols already loaded for /usr/lib/system/libsystem_pthread.dylib Symbols already loaded for /usr/lib/system/libsystem_sandbox.dylib Symbols already loaded for /usr/lib/system/libsystem_secinit.dylib Symbols already loaded for /usr/lib/system/libsystem_trace.dylib Symbols already loaded for /usr/lib/system/libunc.dylib Symbols already loaded for /usr/lib/system/libunwind.dylib Symbols already loaded for /usr/lib/system/libxpc.dylib Symbols already loaded for /usr/lib/libobjc.A.dylib Symbols already loaded for /usr/lib/libauto.dylib Symbols already loaded for /usr/lib/libc++abi.dylib Symbols already loaded for /usr/lib/libc++.1.dylib Symbols already loaded for /usr/lib/libDiagnosticMessagesClient.dylib

(gdb) info shared

(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x000080000d8d29a0  0x000080000d96e968  Yes         /usr/local/nginx_x/luajit/lib/libluajit-5.1.2.dylib
0x00007fff89cd1989  0x00007fff89cd22c0  Yes (*)     /usr/lib/libSystem.B.dylib
0x0000000108d4f590  0x0000000108db42ec  Yes (*)     /usr/local/lib/libpcre.1.dylib
0x00007fff8cf1fae0  0x00007fff8cf31200  Yes (*)     /usr/lib/libz.1.dylib
0x00007fff939128c0  0x00007fff93917160  Yes (*)     /usr/lib/system/libcache.dylib
0x00007fff84d7b164  0x00007fff84d87640  Yes (*)     /usr/lib/system/libcommonCrypto.dylib
0x00007fff80dd8d24  0x00007fff80de10c0  Yes (*)     /usr/lib/system/libcompiler_rt.dylib
0x00007fff878054a8  0x00007fff8780d6b0  Yes (*)     /usr/lib/system/libcopyfile.dylib
0x00007fff81fa46c0  0x00007fff82022e10  Yes (*)     /usr/lib/system/libcorecrypto.dylib
0x00007fff8160ab30  0x00007fff8164b614  Yes (*)     /usr/lib/system/libdispatch.dylib
0x00007fff80f910e8  0x00007fff80f944b0  Yes (*)     /usr/lib/system/libdyld.dylib
0x00007fff821dba19  0x00007fff821dc0b8  Yes (*)     /usr/lib/system/libkeymgr.dylib
0x00007fff910bbfc0  0x00007fff910bbff8  Yes (*)     /usr/lib/system/liblaunch.dylib
0x00007fff8e86f11c  0x00007fff8e874790  Yes (*)     /usr/lib/system/libmacho.dylib
0x00007fff87c5b210  0x00007fff87c5d2b0  Yes (*)     /usr/lib/system/libquarantine.dylib
0x00007fff910b9924  0x00007fff910bb160  Yes (*)     /usr/lib/system/libremovefile.dylib
0x00007fff8497a190  0x00007fff84992224  Yes (*)     /usr/lib/system/libsystem_asl.dylib
0x00007fff87cc0340  0x00007fff87cc16a2  Yes (*)     /usr/lib/system/libsystem_blocks.dylib
0x00007fff87fad020  0x00007fff88043c10  Yes (*)     /usr/lib/system/libsystem_c.dylib
0x00007fff93048df0  0x00007fff9304b4e9  Yes (*)     /usr/lib/system/libsystem_configuration.dylib
0x00007fff93886b04  0x00007fff93889738  Yes (*)     /usr/lib/system/libsystem_coreservices.dylib
0x00007fff948e3a38  0x00007fff948fb121  Yes (*)     /usr/lib/system/libsystem_coretls.dylib
0x00007fff90e825c8  0x00007fff90e8a36c  Yes (*)     /usr/lib/system/libsystem_dnssd.dylib
0x00007fff908b9490  0x00007fff908e4658  Yes (*)     /usr/lib/system/libsystem_info.dylib
0x00007fff8d806a20  0x00007fff8d827928  Yes (*)     /usr/lib/system/libsystem_kernel.dylib
0x00007fff928237c0  0x00007fff9285324c  Yes (*)     /usr/lib/system/libsystem_m.dylib
0x00007fff93565ef0  0x00007fff9358270d  Yes (*)     /usr/lib/system/libsystem_malloc.dylib
0x00007fff90e8b490  0x00007fff90ef26c0  Yes (*)     /usr/lib/system/libsystem_network.dylib
0x00007fff821fdd91  0x00007fff82207048  Yes (*)     /usr/lib/system/libsystem_networkextension.dylib
0x00007fff8d83587b  0x00007fff8d83f440  Yes (*)     /usr/lib/system/libsystem_notify.dylib
0x00007fff87837920  0x00007fff878404dc  Yes (*)     /usr/lib/system/libsystem_platform.dylib
0x00007fff8371df28  0x00007fff8372c120  Yes (*)     /usr/lib/system/libsystem_pthread.dylib
0x00007fff81f9d7d4  0x00007fff81fa062c  Yes (*)     /usr/lib/system/libsystem_sandbox.dylib
0x00007fff81607ac7  0x00007fff816091f0  Yes (*)     /usr/lib/system/libsystem_secinit.dylib
0x00007fff94877ab4  0x00007fff94889bd4  Yes (*)     /usr/lib/system/libsystem_trace.dylib
0x00007fff8f782fc0  0x00007fff8f782ff8  Yes (*)     /usr/lib/system/libunc.dylib
0x00007fff89443918  0x00007fff89449a9a  Yes (*)     /usr/lib/system/libunwind.dylib
0x00007fff8785fd44  0x00007fff8788dd58  Yes (*)     /usr/lib/system/libxpc.dylib
0x00007fff849ad000  0x00007fff84dd9fa8  Yes (*)     /usr/lib/libobjc.A.dylib
0x00007fff9177fda0  0x00007fff917c7090  Yes (*)     /usr/lib/libauto.dylib
0x00007fff83a1ee80  0x00007fff83a49408  Yes (*)     /usr/lib/libc++abi.dylib
0x00007fff8e794f90  0x00007fff8e7efc31  Yes (*)     /usr/lib/libc++.1.dylib
0x00007fff9299812f  0x00007fff929991e0  Yes (*)     /usr/lib/libDiagnosticMessagesClient.dylib
agentzh commented 8 years ago

@idevz OpenResty enables debuginfo in luajit (and all other components) by default, no special treatment needed (not even --with-debug). And our gdb tools should work on OpenResty without those special ./configure options as well.

The No global L located (tried globalL and ngx_cycle) error is usually a sign of lack of Lua related configurations in your nginx server instance. Are you 100% sure that your nginx worker process you're analyzed is indeed running Lua code?

agentzh commented 8 years ago

@idevz Okay, I can reproduce the issue on Mac OS X with the gdb 7.10 installed via homebrew. It works well, however, with gdb 7.9.1 on Fedora Linux with OpenResty 1.9.3.2, according to my test today.

agentzh commented 8 years ago

@idevz Okay, I've digged it up and found that there might be problem in this gdb on Mac OS X or the debuginfo in the nginx binary generated by clang there since gdb.lookup_global_symbol("ngx_cycle") returns None while the global C variable ngx_cycle is definitely there:

(gdb) p/x ngx_cycle
$2 = 0x81005250

(gdb) py print(gdb.lookup_global_symbol("ngx_cycle"))
None

For comparsion, on Linux with gdb 7.9.1, we have the following expected:

(gdb) p/x ngx_cycle
$1 = 0x22731b0
(gdb) py print(gdb.lookup_global_symbol("ngx_cycle"))
ngx_cycle

So it looks like a gdb issue. Will you try on Linux or some other OSes instead?

idevz commented 8 years ago

@agentzh thank you very much, I try again with gdb 7.10.1, and it still fail, I will build a Centos VM for testing. thanks again. :+1:

idevz commented 8 years ago

@agentzh thanks for your help, I build up this on my centos vm with gdb-7.10.1 and openresty-1.9.3.2, It's works well.

[root@zc1 src]# gdb -p 17540
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 17540
Reading symbols from /usr/local/openresty-1.9.3.2/nginx/sbin/nginx...done.
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/local/openresty-1.9.3.2/luajit/lib/libluajit-5.1.so.2...done.
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
0x0000003c766e9133 in __epoll_wait_nocancel () from /lib64/libc.so.6
(gdb) lvmst
current VM state: C code (from interpreted Lua code)
(gdb) lgcstat
 430 str        objects: max=940, avg = 27, min=18, sum=11783
  15 upval      objects: max=24, avg = 24, min=24, sum=360
   1 thread     objects: max=496, avg = 496, min=496, sum=496
  13 proto      objects: max=231, avg = 139, min=80, sum=1815
 306 func       objects: max=144, avg = 32, min=20, sum=9836
   0 trace      objects: max=0, avg = 0, min=0, sum=0
   0 cdata      objects: max=0, avg = 0, min=0, sum=0
  65 tab        objects: max=3104, avg = 296, min=32, sum=19288
   4 udata      objects: max=64, avg = 46, min=40, sum=184

 sizeof strhash 2048
 sizeof g->tmpbuf 128
 sizeof ctype_state 0
 sizeof jit_state 0

total sz 51170
g->strnum 430, g->gc.total 51170
elapsed: 0.120000 sec
(gdb) p/x ngx_cycle
$1 = 0x223e3b0
(gdb)

:+1:

agentzh commented 8 years ago

@idevz Okay, then it looks like a Mac OS X thing.

idevz commented 8 years ago

@agentzh Yes，Thank you very much. :+1: :100:

ssdr commented 8 years ago

same issue on centos 6.5 with gdb 7.11 😢

agentzh commented 8 years ago

@ssdr Are you using the latest version of OpenResty releases? See

http://openresty.org/en/download.html

Since you provide little details about your gdb output, I guess it's either you are using LuaJIT 2.0 instead of LuaJIT 2.1 (only the latter is supported) and/or your LuaJIT build lacks debug symbols. Better use the OpenResty bundle to avoid such problems.

ssdr commented 8 years ago

@agentzh yes, I am using LuaJIT2.0, I'll try LuaJIT2.1, thank you for your reply 😃