kvspb / nginx-auth-ldap

LDAP authentication module for nginx
BSD 2-Clause "Simplified" License
731 stars 251 forks source link

segfaulting on nginx shutdown #168

Closed gbjtv closed 6 years ago

gbjtv commented 7 years ago

We're running nginx 1.10.1 + nginx-auth-ldap (master) running on ubuntu 16.04 and are seeing segfaults. Pasting the backtrace from the core file. I also tried nginx 1.10.3 with the same segfault result.

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from nginx...done.
[New LWP 23531]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process is shu'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_getenv (name=name@entry=0x7fcaadc384fe "GNUTLS_NO_EXPLICIT_INIT") at getenv.c:75
75  getenv.c: No such file or directory.
(gdb) where
#0  __GI_getenv (name=name@entry=0x7fcaadc384fe "GNUTLS_NO_EXPLICIT_INIT") at getenv.c:75
#1  0x00007fcaadb69cae in lib_deinit () at gnutls_global.c:497
#2  0x00007fcaafeb4c17 in _dl_fini () at dl-fini.c:235
#3  0x00007fcaae524ff8 in __run_exit_handlers (status=status@entry=0, listp=0x7fcaae8ae5f8 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#4  0x00007fcaae525045 in __GI_exit (status=status@entry=0) at exit.c:104
#5  0x0000000000431fa7 in ngx_worker_process_exit (cycle=cycle@entry=0xee4820) at src/os/unix/ngx_process_cycle.c:1003
#6  0x0000000000432020 in ngx_worker_process_cycle (cycle=cycle@entry=0xee4820, data=data@entry=0x5)
    at src/os/unix/ngx_process_cycle.c:747
#7  0x0000000000430ac0 in ngx_spawn_process (cycle=cycle@entry=0xee4820, 
    proc=proc@entry=0x431fa7 <ngx_worker_process_cycle>, data=data@entry=0x5, name=name@entry=0x4a69b7 "worker process", 
    respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198
#8  0x0000000000432190 in ngx_start_worker_processes (cycle=cycle@entry=0xee4820, n=40, type=type@entry=-4)
    at src/os/unix/ngx_process_cycle.c:358
#9  0x0000000000433086 in ngx_master_process_cycle (cycle=0xee4820, cycle@entry=0xde7650)
    at src/os/unix/ngx_process_cycle.c:243
#10 0x00000000004120c0 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:367
gbjtv commented 7 years ago

here is a segfault with 1.10.3

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from nginx...done.
[New LWP 13367]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process is shutting down                      '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_getenv (name=name@entry=0x7f3e408744fe "GNUTLS_NO_EXPLICIT_INIT") at getenv.c:75
75  getenv.c: No such file or directory.
(gdb) bt full
#0  __GI_getenv (name=name@entry=0x7f3e408744fe "GNUTLS_NO_EXPLICIT_INIT") at getenv.c:75
        len = 23
        ep = <optimized out>
        name_start = 20039
#1  0x00007f3e407a5cae in lib_deinit () at gnutls_global.c:497
        e = <optimized out>
#2  0x00007f3e42df9c17 in _dl_fini () at dl-fini.c:235
        array = 0x7f3e40aa2dc8
        i = 1
        l = 0x7f3e42ffe000
        maps = 0x7ffffa115ce0
        i = 10
        l = <optimized out>
        nmaps = 37
        nloaded = <optimized out>
        ns = 0
        do_audit = 0
        __PRETTY_FUNCTION__ = "_dl_fini"
#3  0x00007f3e41160ff8 in __run_exit_handlers (status=status@entry=0, listp=0x7f3e414ea5f8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
        atfct = <optimized out>
        onfct = <optimized out>
        cxafct = <optimized out>
#4  0x00007f3e41161045 in __GI_exit (status=status@entry=0) at exit.c:104
No locals.
#5  0x000000000043259d in ngx_worker_process_exit (cycle=cycle@entry=0x1594310) at src/os/unix/ngx_process_cycle.c:1003
        i = <optimized out>
        c = <optimized out>
#6  0x0000000000432616 in ngx_worker_process_cycle (cycle=cycle@entry=0x1594310, data=data@entry=0xa) at src/os/unix/ngx_process_cycle.c:747
        worker = 10
#7  0x00000000004310b6 in ngx_spawn_process (cycle=cycle@entry=0x1594310, proc=proc@entry=0x43259d <ngx_worker_process_cycle>, data=data@entry=0xa, name=name@entry=0x4e9cd7 "worker process", 
    respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 50
#8  0x0000000000432786 in ngx_start_worker_processes (cycle=cycle@entry=0x1594310, n=40, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:358
        i = 10
        ch = {command = 1, pid = 13366, slot = 49, fd = 146}
#9  0x000000000043367c in ngx_master_process_cycle (cycle=0x1594310, cycle@entry=0xfcee10) at src/os/unix/ngx_process_cycle.c:243
        title = <optimized out>
        p = <optimized out>
        size = <optimized out>
        i = <optimized out>
        n = <optimized out>
        sigio = 0
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 49, tv_usec = 18}, it_value = {tv_sec = 16583881, tv_usec = 0}}
        live = 1
        delay = 0
        ls = <optimized out>
        ccf = 0x1595350
#10 0x00000000004126a0 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:367
        b = <optimized out>
        log = 0x794ae0 <ngx_log>
        i = <optimized out>
        cycle = 0xfcee10
        init_cycle = {conf_ctx = 0x0, pool = 0xfce810, log = 0x794ae0 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, 
            action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, 
            next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, 
            nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, 
              next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 32, 
            data = 0xfce860 ""}, conf_param = {len = 29, data = 0x7ffffa117f6b "ng down"}, conf_prefix = {len = 22, data = 0xfce860 ""}, prefix = {len = 17, data = 0x4e591d "/usr/local/nginx/"}, lock_file = {
            len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = <optimized out>
        ccf = <optimized out>
(gdb) 
rengers commented 7 years ago

I'm seeing the same behavior on Ubuntu 16.04 with nginx 1.10.1

What triggers it for us seems to be calling nginx -s reload twice. The first time works, but then the second causes a segfault.

sudo /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.1
built by gcc 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2.1)
built with OpenSSL 1.0.2g-fips  1 Mar 2016 (running with OpenSSL 1.0.2g  1 Mar 2016)
TLS SNI support enabled
configure arguments: --with-http_ssl_module --with-http_v2_module --with-http_flv_module --with-http_dav_module --with-file-aio --with-http_gzip_static_module --with-http_mp4_module --with-http_stub_status_module --with-http_sub_module --add-module=../ngx_http_bytes_filter_module-57365655ee44 --add-module=../nginx_upload_module-2.2.0 --add-module=../nginx-auth-ldap --add-module=../ngx_http_consistent_hash --add-module=../graphite-nginx-module

GDB backtrace:

(gdb) bt full
#0  __GI_getenv (name=0x7fc908c374fe "GNUTLS_NO_EXPLICIT_INIT") at getenv.c:75
        len = 23
        ep = <optimized out>
        name_start = 20039
#1  0x00007fc908b68cae in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
No symbol table info available.
#2  0x00007fc90aeb3c17 in _dl_fini () at dl-fini.c:235
        array = 0x7fc908e65dc8
        i = 1
        l = 0x7fc90b0b99f8
        maps = 0x7ffe13436550
        i = 9
        l = <optimized out>
        nmaps = 36
        nloaded = <optimized out>
        ns = 0
        do_audit = 0
        __PRETTY_FUNCTION__ = "_dl_fini"
#3  0x00007fc909523ff8 in __run_exit_handlers (status=0, listp=0x7fc9098ad5f8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
        atfct = <optimized out>
        onfct = <optimized out>
        cxafct = <optimized out>
#4  0x00007fc909524045 in __GI_exit (status=<optimized out>) at exit.c:104
No locals.
#5  0x0000000000431fa7 in ?? ()
No symbol table info available.
#6  0x0000000000432020 in ?? ()
No symbol table info available.
#7  0x0000000000430ac0 in ngx_spawn_process ()
No symbol table info available.
#8  0x0000000000432190 in ?? ()
No symbol table info available.
#9  0x0000000000433086 in ngx_master_process_cycle ()
No symbol table info available.
#10 0x00000000004120c0 in main ()
No symbol table info available.
gbjtv commented 6 years ago

Went with this for ldap auth instead https://www.nginx.com/blog/nginx-plus-authenticate-users/