agelun commented 11 years ago

Maybe, there is problem in nginx core (ngx_rbtree_delete function), but this problem doesn't appears without push-stream-module (or with module but when it wasn't used in configuration)

Nginx 1.3.8, latest nginx-http-push-stream-module (commit 73e2760c0a9f1a4776478ba023d384e150513ae6)

Backtrace:

0 0x000000000040f6bc in ngx_rbtree_delete (tree=0x7f56ba655000, node=) at src/core/ngx_rbtree.c:300

1 0x000000000047bb3f in ngx_http_push_stream_collect_expired_messages_and_empty_channels (data=0x7f56ba655000, shpool=0x7f56ba4d7000,

force=<value optimized out>) at /usr/src/redhat/SOURCES/nginx/nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:733

2 0x000000000047bc34 in ngx_http_push_stream_memory_cleanup (ev=)

at /usr/src/redhat/SOURCES/nginx/nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:810

3 ngx_http_push_stream_memory_cleanup_timer_wake_handler (ev=)

at /usr/src/redhat/SOURCES/nginx/nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:980

4 0x000000000041cb8c in ngx_event_expire_timers () at src/event/ngx_event_timer.c:149

5 0x000000000041c97f in ngx_process_events_and_timers (cycle=0xeeb330) at src/event/ngx_event.c:263

6 0x0000000000422b58 in ngx_worker_process_cycle (cycle=0xeeb330, data=) at src/os/unix/ngx_process_cycle.c:810

7 0x0000000000421147 in ngx_spawn_process (cycle=0xeeb330, proc=0x422a90 , data=0x0, name=0x47ffb9 "worker process",

respawn=-4) at src/os/unix/ngx_process.c:198

8 0x0000000000422112 in ngx_start_worker_processes (cycle=0xeeb330, n=4, type=-4) at src/os/unix/ngx_process_cycle.c:365

9 0x000000000042331d in ngx_master_process_cycle (cycle=0xeeb330) at src/os/unix/ngx_process_cycle.c:250

10 0x000000000040787b in main (argc=, argv=) at src/core/nginx.c:412

(gdb) list 295 ngx_rbt_red(temp->parent); 296 ngx_rbtree_right_rotate(root, sentinel, temp->parent); 297 w = temp->parent->left; 298 } 299 300 if (ngx_rbt_is_black(w->left) && ngx_rbt_is_black(w->right)) { 301 ngx_rbt_red(w); 302 temp = temp->parent; 303 304 } else {

(gdb) p *w $1 = {key = 0, left = 0x0, right = 0x0, parent = 0x7f56bb2aa200, color = 0 '\000', data = 0 '\000'}

(gdb) p *sentinel $5 = {key = 0, left = 0x0, right = 0x0, parent = 0x7f56bb2aa200, color = 0 '\000', data = 0 '\000'}

(gdb) p w==sentinel $27 = 1

ngx_rbt_is_black is macros:

define ngx_rbt_is_black(node) (!ngx_rbt_is_red(node))

define ngx_rbt_copy_color(n1, n2) (n1->color = n2->color)

and we are trying to send null pointer to ngx_rbt_copy_color (w->left == 0 and w->right == 0)

wandenberg commented 11 years ago

Hi,

just to help me debug the problem, are you using the feature of do a DELETE in one channel? Can you send me your configuration file too?

agelun commented 11 years ago

No, we are not using DELETE.

nginx version: nginx/1.3.8 TLS SNI support enabled configure arguments: --prefix=/opt/smartlabs/nginx --conf-path=/opt/smartlabs/nginx/conf/nginx.conf --sbin-path=/opt/smartlabs/nginx/sbin/nginx --error-log-path=/var/log/smartlabs/nginx/error.log --http-log-path=/var/log/smartlabs/nginx/access.log --http-client-body-temp-path=/opt/smartlabs/nginx/tmp/client --http-proxy-temp-path=/opt/smartlabs/nginx/tmp/proxy --http-fastcgi-temp-path=/opt/smartlabs/nginx/tmp/fastcgi --pid-path=/var/run/nginx.pid --user=smartcat --group=smartcat --with-cc-opt='-I /usr/include/pcre/' --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_image_filter_module --add-module=/usr/src/redhat/SOURCES/nginx/nginx-push-stream-module

Configuration related to push_stream_module:

push_stream_shared_memory_size 256M; push_stream_shared_memory_cleanup_objects_ttl 30s; push_stream_max_number_of_channels 150000;

server {

Server for pushing long-polling requests.

    listen                  X.X.X.X:PPPP;

    location = / {
            access_log      off;
            deny        all;
    }

    location /publish {
            push_stream_publisher           admin;
            set $push_stream_channel_id     $arg_channel;
            push_stream_store_messages      off;
            push_stream_content_type        text/plain;
            push_stream_keepalive           off;
            default_type                    text/plain;
    }
    location /push-stat {
            push_stream_channels_statistics;
            set $push_stream_channel_id             $arg_id;
    }

}