search

wandenberg / nginx-push-stream-module

A pure stream http push technology for your Nginx setup. Comet made easy and really scalable.
Other
2.21k stars 295 forks source link

Segfault error #263

Closed materkov closed 6 years ago

materkov commented 6 years ago

Hi, @wandenberg. I've found some random strange segfaults (probably this is because of events channel). Could you please check this core dump:

GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/nginx-debug...Reading symbols from /usr/lib/debug/.build-id/bf/c2bb73eb1b9af699fdebbac45d6a9dd4dc9f5f.debug...done.
done.
[New LWP 81565]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process               '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ngx_http_push_stream_add_msg_to_channel (mcf=mcf@entry=0x7f6d69ad6220, log=log@entry=0x7f6d69a80808, channel=0x0, 
    text=text@entry=0x7f6d69a84810 "{\"type\": \"channel_destroyed\", \"channel\": \"session.w81apqcgzv9bzzq04qd7cnbfuprcwj6t7ro9oldubutwnwzwcdnoeok5imf5otrn\"}", len=116, 
    event_id=event_id@entry=0x0, event_type=event_type@entry=0x7f6d67493160 <NGX_HTTP_PUSH_STREAM_EVENT_TYPE_CHANNEL_DESTROYED>, store_messages=store_messages@entry=1, 
    temp_pool=temp_pool@entry=0x7f6d69a847b0) at ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c:385
385     ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c: No such file or directory.
(gdb) bt full
#0  ngx_http_push_stream_add_msg_to_channel (mcf=mcf@entry=0x7f6d69ad6220, log=log@entry=0x7f6d69a80808, channel=0x0, 
    text=text@entry=0x7f6d69a84810 "{\"type\": \"channel_destroyed\", \"channel\": \"session.w81apqcgzv9bzzq04qd7cnbfuprcwj6t7ro9oldubutwnwzwcdnoeok5imf5otrn\"}", len=116, 
    event_id=event_id@entry=0x0, event_type=event_type@entry=0x7f6d67493160 <NGX_HTTP_PUSH_STREAM_EVENT_TYPE_CHANNEL_DESTROYED>, store_messages=store_messages@entry=1, 
    temp_pool=temp_pool@entry=0x7f6d69a847b0) at ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c:385
        data = 0x7f6d46d3d000
        msg = <optimized out>
        qtd_removed = <optimized out>
        id = <optimized out>
        time = <optimized out>
        tag = <optimized out>
#1  0x00007f6d672869ea in ngx_http_push_stream_send_event (mcf=0x7f6d69ad6220, log=0x7f6d69a80808, channel=<optimized out>, 
    event_type=0x7f6d67493160 <NGX_HTTP_PUSH_STREAM_EVENT_TYPE_CHANNEL_DESTROYED>, received_temp_pool=0x7f6d69a847b0)
    at ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c:461
        len = <optimized out>
        event = 0x7f6d69a84800
        temp_pool = 0x7f6d69a847b0
        channel = <optimized out>
        received_temp_pool = 0x7f6d69a847b0
        event_type = 0x7f6d67493160 <NGX_HTTP_PUSH_STREAM_EVENT_TYPE_CHANNEL_DESTROYED>
        log = 0x7f6d69a80808
        mcf = 0x7f6d69ad6220
        temp_pool = <optimized out>
#2  0x00007f6d672884c3 in ngx_http_push_stream_collect_expired_messages_and_empty_channels_data (data=data@entry=0x7f6d46d3d000, force=force@entry=0)
    at ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1035
        mcf = 0x7f6d69ad6220
        channel = 0x7f6d46d4d300
        q = 0x7f6d46d4d428
        temp_pool = 0x7f6d69a847b0
#3  0x00007f6d672885de in ngx_http_push_stream_memory_cleanup () at ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1123
        data = 0x7f6d46d3d000
        global_data = 0x7f6d68ec4000
        q = 0x7f6d46d4b1f0
#4  ngx_http_push_stream_memory_cleanup_timer_wake_handler (ev=<optimized out>) at ../ngx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1328
No locals.
#5  0x00007f6d68beae5e in ngx_event_expire_timers () at src/event/ngx_event_timer.c:94
        ev = 0x7f6d67493ea0 <ngx_http_push_stream_memory_cleanup_event>
        root = 0x7f6d67495f28 <ngx_http_push_stream_buffer_cleanup_event+40>
        sentinel = 0x7f6d68eeab20 <ngx_event_timer_sentinel>
#6  0x00007f6d68beaa8d in ngx_process_events_and_timers (cycle=cycle@entry=0x7f6d69a807f0) at src/event/ngx_event.c:256
        flags = <optimized out>
        timer = <optimized out>
        delta = 1232
#7  0x00007f6d68bf2eb5 in ngx_worker_process_cycle (cycle=cycle@entry=0x7f6d69a807f0, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:749
        worker = 0
#8  0x00007f6d68bf189a in ngx_spawn_process (cycle=cycle@entry=0x7f6d69a807f0, proc=proc@entry=0x7f6d68bf2e60 <ngx_worker_process_cycle>, data=data@entry=0x0, 
    name=name@entry=0x7f6d68c97097 "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198
        on = 1
---Type <return> to continue, or q <return> to quit---
        pid = 0
        s = 2
#9  0x00007f6d68bf3190 in ngx_start_worker_processes (cycle=cycle@entry=0x7f6d69a807f0, n=2, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:358
        i = 0
        ch = {command = 1, pid = 0, slot = 0, fd = 0}
#10 0x00007f6d68bf422d in ngx_master_process_cycle (cycle=0x7f6d69a807f0) at src/os/unix/ngx_process_cycle.c:243
        title = <optimized out>
        p = <optimized out>
        size = <optimized out>
        i = <optimized out>
        n = <optimized out>
        sigio = 0
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
        live = <optimized out>
        delay = 0
        ls = <optimized out>
        ccf = 0x7f6d69a81c48
#11 0x00007f6d68bc90b5 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
        b = <optimized out>
        log = <optimized out>
        i = <optimized out>
        cycle = 0x7f6d69a7c7e0
        init_cycle = {conf_ctx = 0x0, pool = 0x7f6d69a7bc60, log = 0x7f6d68ee83e0 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, handler = 0x0, 
            data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, 
          modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, 
          paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, 
            sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {
              elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, 
            pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x7ffd9dbe6951 "rocess"}, 
          conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x7ffd9dbe6951 "rocess"}, prefix = {len = 11, data = 0x7f6d68c921e9 "/etc/nginx/"}, lock_file = {len = 0, 
            data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = <optimized out>
        ccf = 0x7f6d69a7e598
(gdb)

Please, let me know if you need additional info. Thanks!

PS This could be somehow related to this https://github.com/wandenberg/nginx-push-stream-module/issues/260 because this line (ngx_http_push_stream_module_utils.c:385) was added here

materkov commented 6 years ago

Any updates?

wandenberg commented 6 years ago

Hi @materkov sorry, I missed the notification of this issue. I will take a look. Can you please confirm the Nginx version and the exact commit of the module you are using?

materkov commented 6 years ago

Nginx Push Stream module was built from latest commit in master: 16e6b4f7645dc6645ac81998af7649cd59328360

Nginx info:

$ nginx -V
nginx version: nginx/1.12.0
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
materkov commented 6 years ago

Is there any news about this issue?

wandenberg commented 6 years ago

Hi @materkov can you confirm if this issue happened after a reload? Looking at the code this seems to be the only way to have this problem happening. Just want to confirm before I change something.

materkov commented 6 years ago

This happens at random period of time. It can happen in 1 minute after start, or 1 hour after start, or 1 day after start.

It also crashes on previous version 8ff981d6d51e998fb48fc7592245ab8ec38f9e73 with events channel enabled. With disabled events channel, everything is working normal.

wandenberg commented 6 years ago

Can you check if there is something reloading your Nginx? Maybe a cron, or even another segfault happening before this one?

wandenberg commented 6 years ago

Can you test the code in this branch fix_issue_263?

If the problem persists, please send me a new backtrace.

materkov commented 6 years ago

I've tried some tests today with reload command. And yes, it looks like that service nginx reload was the root of the problem. At least core dump today was the same as the original comment (futhermore, no messages were sent in events channel after reloading, and then SEGFAULT happened after a minute after reload)

Branch fix_issue_263 fixed this issue. At least, it's not reproducing with reload now. Thanks! :)