Nginx hangs

[boniek83]

Threads stacktraces:

14893

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14894

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14895

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14896

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14897

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f30a9 in ngx_http_push_stream_throw_the_message_away ()

3 0x00000000004f61e8 in ngx_http_push_stream_ensure_qtd_of_messages.part.18 ()

4 0x00000000004f821f in ngx_http_push_stream_add_msg_to_channel ()

5 0x00000000004fb37e in ngx_http_push_stream_publisher_body_handler ()

6 0x000000000048edb0 in ngx_http_read_client_request_body ()

7 0x00000000004fcc6e in ngx_http_push_stream_publisher_handler ()

8 0x000000000048277e in ngx_http_core_content_phase ()

9 0x000000000047cd85 in ngx_http_core_run_phases ()

10 0x0000000000487e1f in ngx_http_process_request ()

11 0x000000000048868e in ngx_http_process_request_line ()

12 0x0000000000469c00 in ngx_event_process_posted ()

13 0x000000000046fef1 in ngx_worker_process_cycle ()

14 0x000000000046e909 in ngx_spawn_process ()

15 0x00000000004700c4 in ngx_start_worker_processes ()

16 0x0000000000470b64 in ngx_master_process_cycle ()

17 0x000000000044cbb7 in main ()

14898

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f30a9 in ngx_http_push_stream_throw_the_message_away ()

3 0x00000000004f61e8 in ngx_http_push_stream_ensure_qtd_of_messages.part.18 ()

4 0x00000000004f821f in ngx_http_push_stream_add_msg_to_channel ()

5 0x00000000004fb37e in ngx_http_push_stream_publisher_body_handler ()

6 0x000000000048edb0 in ngx_http_read_client_request_body ()

7 0x00000000004fcc6e in ngx_http_push_stream_publisher_handler ()

8 0x000000000048277e in ngx_http_core_content_phase ()

9 0x000000000047cd85 in ngx_http_core_run_phases ()

10 0x0000000000487e1f in ngx_http_process_request ()

11 0x000000000048868e in ngx_http_process_request_line ()

12 0x0000000000471ddc in ngx_epoll_process_events ()

13 0x0000000000469783 in ngx_process_events_and_timers ()

14 0x000000000046fef1 in ngx_worker_process_cycle ()

15 0x000000000046e909 in ngx_spawn_process ()

16 0x00000000004700c4 in ngx_start_worker_processes ()

17 0x0000000000470b64 in ngx_master_process_cycle ()

18 0x000000000044cbb7 in main ()

14899

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14900

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14901

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14902

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f30a9 in ngx_http_push_stream_throw_the_message_away ()

3 0x00000000004f61e8 in ngx_http_push_stream_ensure_qtd_of_messages.part.18 ()

4 0x00000000004f821f in ngx_http_push_stream_add_msg_to_channel ()

5 0x00000000004fb37e in ngx_http_push_stream_publisher_body_handler ()

6 0x000000000048edb0 in ngx_http_read_client_request_body ()

7 0x00000000004fcc6e in ngx_http_push_stream_publisher_handler ()

8 0x000000000048277e in ngx_http_core_content_phase ()

9 0x000000000047cd85 in ngx_http_core_run_phases ()

10 0x0000000000487e1f in ngx_http_process_request ()

11 0x000000000048868e in ngx_http_process_request_line ()

12 0x0000000000471ddc in ngx_epoll_process_events ()

13 0x0000000000469783 in ngx_process_events_and_timers ()

14 0x000000000046fef1 in ngx_worker_process_cycle ()

15 0x000000000046e909 in ngx_spawn_process ()

16 0x00000000004700c4 in ngx_start_worker_processes ()

17 0x0000000000470b64 in ngx_master_process_cycle ()

18 0x000000000044cbb7 in main ()

14903

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14904

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14905

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14906

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14907

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14908

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14909

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004fbac8 in ngx_http_push_stream_free_memory_of_expired_messages_and_channels_data ()

3 0x00000000004fbc98 in ngx_http_push_stream_memory_cleanup_timer_wake_handler ()

4 0x0000000000469b52 in ngx_event_expire_timers ()

5 0x00000000004697ab in ngx_process_events_and_timers ()

6 0x000000000046fef1 in ngx_worker_process_cycle ()

7 0x000000000046e909 in ngx_spawn_process ()

8 0x00000000004700c4 in ngx_start_worker_processes ()

9 0x0000000000470b64 in ngx_master_process_cycle ()

10 0x000000000044cbb7 in main ()

14910

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14911

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14912

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14913

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14914

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14915

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14916

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26 ()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

After some time nginx hangs when using pushstream module. Nginx version 1.10.1 pushstream module version 0.5.2.

[wandenberg]

Check if any of your workers is dying. If so, create a core dump to be possible verify what is happening. On Aug 29, 2016 06:20, "boniek" notifications@github.com wrote:

Threads stacktraces: 14893

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14894

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14895

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14896

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14897

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f30a9 in ngx_http_push_stream_throw_the_message_away ()

3 0x00000000004f61e8 in ngx_http_push_stream_ensure_qtd_of_messages.part.18

()

4 0x00000000004f821f in ngx_http_push_stream_add_msg_to_channel ()

5 0x00000000004fb37e in ngx_http_push_stream_publisher_body_handler ()

6 0x000000000048edb0 in ngx_http_read_client_request_body ()

7 0x00000000004fcc6e in ngx_http_push_stream_publisher_handler ()

8 0x000000000048277e in ngx_http_core_content_phase ()

9 0x000000000047cd85 in ngx_http_core_run_phases ()

10 0x0000000000487e1f in ngx_http_process_request ()

11 0x000000000048868e in ngx_http_process_request_line ()

12 0x0000000000469c00 in ngx_event_process_posted ()

13 0x000000000046fef1 in ngx_worker_process_cycle ()

14 0x000000000046e909 in ngx_spawn_process ()

15 0x00000000004700c4 in ngx_start_worker_processes ()

16 0x0000000000470b64 in ngx_master_process_cycle ()

17 0x000000000044cbb7 in main ()

14898

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f30a9 in ngx_http_push_stream_throw_the_message_away ()

3 0x00000000004f61e8 in ngx_http_push_stream_ensure_qtd_of_messages.part.18

()

4 0x00000000004f821f in ngx_http_push_stream_add_msg_to_channel ()

5 0x00000000004fb37e in ngx_http_push_stream_publisher_body_handler ()

6 0x000000000048edb0 in ngx_http_read_client_request_body ()

7 0x00000000004fcc6e in ngx_http_push_stream_publisher_handler ()

8 0x000000000048277e in ngx_http_core_content_phase ()

9 0x000000000047cd85 in ngx_http_core_run_phases ()

10 0x0000000000487e1f in ngx_http_process_request ()

11 0x000000000048868e in ngx_http_process_request_line ()

12 0x0000000000471ddc in ngx_epoll_process_events ()

13 0x0000000000469783 in ngx_process_events_and_timers ()

14 0x000000000046fef1 in ngx_worker_process_cycle ()

15 0x000000000046e909 in ngx_spawn_process ()

16 0x00000000004700c4 in ngx_start_worker_processes ()

17 0x0000000000470b64 in ngx_master_process_cycle ()

18 0x000000000044cbb7 in main ()

14899

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14900

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14901

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14902

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f30a9 in ngx_http_push_stream_throw_the_message_away ()

3 0x00000000004f61e8 in ngx_http_push_stream_ensure_qtd_of_messages.part.18

()

4 0x00000000004f821f in ngx_http_push_stream_add_msg_to_channel ()

5 0x00000000004fb37e in ngx_http_push_stream_publisher_body_handler ()

6 0x000000000048edb0 in ngx_http_read_client_request_body ()

7 0x00000000004fcc6e in ngx_http_push_stream_publisher_handler ()

8 0x000000000048277e in ngx_http_core_content_phase ()

9 0x000000000047cd85 in ngx_http_core_run_phases ()

10 0x0000000000487e1f in ngx_http_process_request ()

11 0x000000000048868e in ngx_http_process_request_line ()

12 0x0000000000471ddc in ngx_epoll_process_events ()

13 0x0000000000469783 in ngx_process_events_and_timers ()

14 0x000000000046fef1 in ngx_worker_process_cycle ()

15 0x000000000046e909 in ngx_spawn_process ()

16 0x00000000004700c4 in ngx_start_worker_processes ()

17 0x0000000000470b64 in ngx_master_process_cycle ()

18 0x000000000044cbb7 in main ()

14903

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14904

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14905

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14906

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14907

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14908

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14909

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004fbac8 in ngx_http_push_streamfree

memory_of_expired_messages_and_channels_data ()

3 0x00000000004fbc98 in ngx_http_push_stream_memory_cleanup_timer_wake_handler

()

4 0x0000000000469b52 in ngx_event_expire_timers ()

5 0x00000000004697ab in ngx_process_events_and_timers ()

6 0x000000000046fef1 in ngx_worker_process_cycle ()

7 0x000000000046e909 in ngx_spawn_process ()

8 0x00000000004700c4 in ngx_start_worker_processes ()

9 0x0000000000470b64 in ngx_master_process_cycle ()

10 0x000000000044cbb7 in main ()

14910

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14911

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14912

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14913

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f8514 in ngx_http_push_stream_cleanup_request_context ()

3 0x00000000004fde3b in ngx_http_push_stream_websocket_reading ()

4 0x0000000000485f0f in ngx_http_request_handler ()

5 0x0000000000471ddc in ngx_epoll_process_events ()

6 0x0000000000469783 in ngx_process_events_and_timers ()

7 0x000000000046fef1 in ngx_worker_process_cycle ()

8 0x000000000046e909 in ngx_spawn_process ()

9 0x00000000004700c4 in ngx_start_worker_processes ()

10 0x0000000000470b64 in ngx_master_process_cycle ()

11 0x000000000044cbb7 in main ()

14914

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14915

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

14916

0 0x00007f31b01ad790 in sem_wait () from /lib64/libpthread.so.0

1 0x000000000045a1b8 in ngx_shmtx_lock ()

2 0x00000000004f9c09 in ngx_http_push_stream_send_old_messages ()

3 0x00000000004f9eab in ngx_http_push_stream_subscriber_assign_channel.isra.26

()

4 0x00000000004fda07 in ngx_http_push_stream_websocket_handler ()

5 0x000000000048277e in ngx_http_core_content_phase ()

6 0x000000000047cd85 in ngx_http_core_run_phases ()

7 0x0000000000487e1f in ngx_http_process_request ()

8 0x000000000048868e in ngx_http_process_request_line ()

9 0x0000000000469c00 in ngx_event_process_posted ()

10 0x000000000046fef1 in ngx_worker_process_cycle ()

11 0x000000000046e909 in ngx_spawn_process ()

12 0x00000000004700c4 in ngx_start_worker_processes ()

13 0x0000000000470b64 in ngx_master_process_cycle ()

14 0x000000000044cbb7 in main ()

After some time nginx hangs when using pushstream module. Nginx version 1.10.1 pushstream module version 0.5.2.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/wandenberg/nginx-push-stream-module/issues/245, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFfNTCI6kSmoP7cGmgAgzBNg7omsgFiks5qkrJVgaJpZM4JvVu- .

[boniek83]

They are not dying - all are just hanging.

[mkonecny]

Some sort of thread deadlock?

[kvendi]

have the same problem in nginx 1.10.1 and nginx 1.8.0

[brooke14]

problem exists in nginx-extras 1.10.1-1~dotdeb+7.3 from dotdeb.com

[kvendi]

Debug log before hanging

`2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map: "/publish" "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "1"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script copy: ":"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script copy: ":"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map: "1:0:0" "0"

2016/09/29 16:24:31 [warn] 14711#14711: *2036615 using uninitialized "backendhostname" variable while logging request, client: XX.XX.XX.XX, server: _, request: "POST /publish?id=XXX HTTP/1.1", host: "XXXXXXX:92"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http pinba handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 run cleanup: 00007F68BF5A6710

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F67DAAAA970, unused: 4016

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF5A5730, unused: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F67DAACC180, unused: 3

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BFDB17A0, unused: 3372

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF1DD4B0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 hc free: 0000000000000000 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 hc busy: 0000000000000000 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 reusable connection: 1

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 event timer add: 435: 75000:1475155546867

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 post event 00007F67DB21A6B0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 delete posted event 00007F67DB21A6B0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http keepalive handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 malloc: 00007F68BF1DD4B0:1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 recv: fd:435 -1 of 1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 recv() not ready (11: Resource temporarily unavailable)

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF1DD4B0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http keepalive handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 malloc: 00007F67DAA10260:1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 recv: fd:435 257 of 1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 reusable connection: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BF5A5730:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BF1D4380:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 event timer del: 435: 1475155546867

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http process request line

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http request line: "POST /publish?id=XXX HTTP/1.1"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http uri: "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http args: "id=XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http exten: ""

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http process request header line

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header: "Host: XXXXXXX:92"

2016/09/29 16:24:31 [debug] 14711#14711: 2036615 http header: "Accept: */"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header: "Content-Length: 96"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header: "Content-Type: application/x-www-form-urlencoded"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header done

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 2

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 test location: "/eu/mypublish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 test location: "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 using configuration "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http cl:96 max:1048576

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 4

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id no header found

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id no header found, args found

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id no id found

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 trackuploads no id found in POST upload req

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 5

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 6

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 post rewrite phase: 7

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 8

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 9

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 10

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 11

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 12

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 13

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 14

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 15

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 post access phase: 16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 pagespeed phase: 17

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 try files phase: 18

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 add cleanup: 00007F68BF5A6710

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BF1AC520:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http client request body preread 96

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http request body content length filter

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http body new buf t:1 f:0 00007F67DAA10301, pos 00007F67DAA10301, size: 96 file: 0, size: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 uploadprogress error-tracker error: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 xslt filter header

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BF15EF10:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 HTTP/1.1 200 OK

Server: nginx

Date: Thu, 29 Sep 2016 13:24:31 GMT

Content-Type: application/json

Content-Length: 101

Connection: keep-alive

Expires: Thu, 01 Jan 1970 00:00:01 GMT

Cache-Control: no-cache, no-store, must-revalidate

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 write new buf t:1 f:0 00007F68BF15EF30, pos 00007F68BF15EF30, size: 240 file: 0, size: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http write filter: l:0 f:0 s:240

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http output filter "/publish?id=XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http copy filter: "/publish?id=XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 image filter

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 xslt filter body

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http postpone filter "/publish?id=XXX" 00007F68BF1D5350

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 write old buf t:1 f:0 00007F68BF15EF30, pos 00007F68BF15EF30, size: 240 file: 0, size: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 write new buf t:0 f:0 00007F68BF1D5240, pos 00007F68BF1D5240, size: 101 file: 0, size: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http write filter: l:1 f:1 s:341

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http write filter limit 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 writev: 341 of 341

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http write filter 0000000000000000

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http copy filter: 0 "/publish?id=XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http finalize request: 0, "/publish?id=XXX" a:1, c:2

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http request count:2 blk:0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http finalize request: -4, "/publish?id=XXX" a:1, c:1

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 set http keepalive handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http close request

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http log handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map started

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map started

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map: "" "1"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map started

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map: "" "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map started

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map: "/publish" "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "1"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script copy: ":"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script copy: ":"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "0"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http map: "1:0:0" "0"

2016/09/29 16:24:31 [warn] 14711#14711: *2036615 using uninitialized "backendhostname" variable while logging request, client: XX.XX.XX.XX, server: _, request: "POST /publish?id=XXX HTTP/1.1", host: "XXXXXXX:92"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http pinba handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 run cleanup: 00007F68BF5A6710

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF1AC520, unused: 4016

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF5A5730, unused: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF1D4380, unused: 3

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F68BF15EF10, unused: 3372

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F67DAA10260

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 hc free: 0000000000000000 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 hc busy: 0000000000000000 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 reusable connection: 1

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 event timer add: 435: 75000:1475155546913

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 post event 00007F67DB21A6B0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 delete posted event 00007F67DB21A6B0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http keepalive handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 malloc: 00007F67DAA10260:1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 recv: fd:435 -1 of 1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 recv() not ready (11: Resource temporarily unavailable)

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 free: 00007F67DAA10260

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http keepalive handler

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 malloc: 00007F67DAA10260:1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 recv: fd:435 257 of 1024

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 reusable connection: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BEF49340:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BF15EF10:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 event timer del: 435: 1475155546913

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http process request line

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http request line: "POST /publish?id=XXX HTTP/1.1"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http uri: "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http args: "id=XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http exten: ""

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http process request header line

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header: "Host: XXXXXXX:92"

2016/09/29 16:24:31 [debug] 14711#14711: 2036615 http header: "Accept: */"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header: "Content-Length: 96"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header: "Content-Type: application/x-www-form-urlencoded"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http header done

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 0

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 2

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 test location: "/eu/mypublish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 test location: "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 using configuration "/publish"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http cl:96 max:1048576

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 4

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id no header found

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id no header found, args found

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 upload-progress: get_tracking_id no id found

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 trackuploads no id found in POST upload req

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 5

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 rewrite phase: 6

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 post rewrite phase: 7

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 8

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 9

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 generic phase: 10

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 11

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 12

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 13

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 14

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 access phase: 15

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 post access phase: 16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 pagespeed phase: 17

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 try files phase: 18

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 add cleanup: 00007F68BEF4A320

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 posix_memalign: 00007F68BF1D4380:4096 @16

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http script var: "XXX"

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http client request body preread 96

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http request body content length filter

2016/09/29 16:24:31 [debug] 14711#14711: *2036615 http body new buf t:1 f:0 00007F67DAA10301, pos 00007F67DAA10301, size: 96 file: 0, size: 0

2016/09/29 16:24:31 [debug] 14958#14958: epoll add event: fd:326 op:1 ev:00002001

2016/09/29 16:24:32 [debug] 14673#14673: epoll add event: fd:326 op:1 ev:00002001

2016/09/29 16:24:32 [debug] 14958#14958: epoll del event: fd:326 op:2 ev:00000000

2016/09/29 16:24:32 [debug] 14958#14958: epoll add event: fd:326 op:1 ev:00002001

2016/09/29 16:24:32 [debug] 14673#14673: epoll del event: fd:326 op:2 ev:00000000

2016/09/29 16:24:32 [debug] 14958#14958: post event 00007F67DB216AB0

2016/09/29 16:24:32 [debug] 14958#14958: delete posted event 00007F67DB216AB0

2016/09/29 16:24:32 [debug] 14958#14958: accept on 0.0.0.0:92, ready: 1

2016/09/29 16:24:32 [debug] 14958#14958: posix_memalign: 00007F68BFCE4D10:512 @16

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 accept: XXX.XXX.XXX.XXX:51834 fd:460

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 event timer add: 460: 60000:1475155532870

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 reusable connection: 1

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 epoll add event: fd:460 op:1 ev:80002001

2016/09/29 16:24:32 [debug] 14958#14958: accept() not ready (11: Resource temporarily unavailable)

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 post event 00007F67DB218E50

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 delete posted event 00007F67DB218E50

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http wait request handler

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF3E8F10:1024

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv: fd:460 201 of 1024

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 reusable connection: 0

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 posix_memalign: 00007F68C042B600:4096 @16

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 posix_memalign: 00007F68C08130B0:4096 @16

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http process request line

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http request line: "POST /publish?id=c7f6abda9d1a8b6d6aa9deca54dad3e9 HTTP/1.1"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http uri: "/publish"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http args: "id=c7f6abda9d1a8b6d6aa9deca54dad3e9"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http exten: ""

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http process request header line

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http header: "Host: XXXXXXX:92"

2016/09/29 16:24:32 [debug] 14958#14958: 2037138 http header: "Accept: */"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http header: "Content-type: application/json; charset=utf-8"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http header: "Content-Length: 9101"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http header: "Expect: 100-continue"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http header done

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 event timer del: 460: 1475155532870

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 generic phase: 0

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 rewrite phase: 2

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 test location: "/eu/mypublish"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 test location: "/publish"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 using configuration "/publish"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http cl:9101 max:1048576

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 rewrite phase: 4

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 upload-progress: get_tracking_id

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 upload-progress: get_tracking_id no header found

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 upload-progress: get_tracking_id no header found, args found

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 upload-progress: get_tracking_id no id found

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 trackuploads no id found in POST upload req

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 rewrite phase: 5

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 rewrite phase: 6

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 post rewrite phase: 7

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 generic phase: 8

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 generic phase: 9

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 generic phase: 10

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 access phase: 11

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 access phase: 12

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 access phase: 13

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 access phase: 14

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 access phase: 15

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 post access phase: 16

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 pagespeed phase: 17

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 try files phase: 18

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 add cleanup: 00007F68C042C5E8

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 posix_memalign: 00007F68C08140C0:4096 @16

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http script var: "c7f6abda9d1a8b6d6aa9deca54dad3e9"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 send 100 Continue

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 send: fd:460 25 of 25

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http request body content length filter

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F67DB3CEB20:9101

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http read client request body

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv: fd:460 -1 of 9101

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv() not ready (11: Resource temporarily unavailable)

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body recv -2

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body rest 9101

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 event timer add: 460: 60000:1475155532870

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http finalize request: -4, "/publish?id=c7f6abda9d1a8b6d6aa9deca54dad3e9" a:1, c:2

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http request count:2 blk:0

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 post event 00007F67DB218E50

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 delete posted event 00007F67DB218E50

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http run request: "/publish?id=c7f6abda9d1a8b6d6aa9deca54dad3e9"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http read client request body

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv: fd:460 1248 of 9101

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body recv 1248

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body rest 9101

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv: fd:460 -1 of 7853

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv() not ready (11: Resource temporarily unavailable)

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body recv -2

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body rest 9101

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 event timer: 460, old: 1475155532870, new: 1475155532920

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 post event 00007F67DB218E50

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 delete posted event 00007F67DB218E50

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http run request: "/publish?id=c7f6abda9d1a8b6d6aa9deca54dad3e9"

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http read client request body

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 recv: fd:460 7853 of 7853

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body recv 7853

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http body new buf t:1 f:0 00007F67DB3CEB20, pos 00007F67DB3CEB20, size: 9101 file: 0, size: 0

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 http client request body rest 0

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 event timer del: 460: 1475155532870

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F67DA25D090:9102

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF5BB6A0:9118

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF1A1960:9198

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF89A600:9342

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF145440:9118

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF44EDF0:9262

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F68BF591220:9233

2016/09/29 16:24:32 [debug] 14958#14958: *2037138 malloc: 00007F67DB3C61C0:9377

2016/09/29 16:24:32 [debug] 14673#14673: epoll add event: fd:326 op:1 ev:00002001

2016/09/29 16:24:33 [debug] 15022#15022: epoll add event: fd:326 op:1 ev:00002001

2016/09/29 16:24:33 [debug] 14846#14846: epoll add event: fd:326 op:1 ev:00002001`

[wandenberg]

I'm looking for the dead lock. Seems to be an unpredicted race condition. Does somebody have an environment where the problem always happens? Would be great if I have access to the environment, or have a coredump of workers, or if someone could be able reproduce the problem.

[brooke14]

coredump here https://drive.google.com/open?id=0B6T334uR7lw3YjRrZWFiZ1JCYkE

[wandenberg]

@brooke14 can you share your binary and if possible your configuration also? And the "nginx -v" output? (you can send in private to me if you prefer)

[brooke14]

nginx version: nginx/1.10.1 nginx.gz

nginx.conf :

        push_stream_message_ttl 6m;
        push_stream_subscriber_connection_ttl 60s;
        push_stream_max_messages_stored_per_channel 1;
        push_stream_shared_memory_size 3072M;

comet.conf:

server {
        listen       92;
        server_name  _;

    recursive_error_pages off;

    access_log /home/site/logs/nginx/informers.comet.access.log main;
    error_log  /home/site/logs/nginx/informers.comet.error.log;

        ## http://wiki.nginx.org/HttpPushStreamModule

    location /publish {
            push_stream_publisher; 
        push_stream_channels_path $arg_id;
        push_stream_store_messages  on;
        }

        location /activity {
            push_stream_subscriber long-polling;
        push_stream_channels_path $arg_id;
            default_type  text/plain;
        add_header XDomainRequestAllowed 1;

        }
        location /mypublish {
            push_stream_publisher;
        push_stream_channels_path my$arg_client_id;
            push_stream_store_messages  on;
        }
        location /eu/mypublish {
            push_stream_publisher;
            push_stream_channels_path eumy$arg_client_id;
            push_stream_store_messages  on;
        }

    location /channels-stats {
            push_stream_channels_statistics;
        push_stream_channels_path             $arg_id;
    }

}

websockets configs on sites-enable/somehost.conf

       location /activityws {
                access_log /home/site/logs/nginx/****_activityws_access.log;
                error_log  /home/site/logs/nginx/****_activityws_error.log;

                push_stream_subscriber websocket;
                push_stream_channels_path     $arg_id;
                default_type  text/plain;

                push_stream_subscriber_connection_ttl       360s;
                push_stream_ping_message_interval           10s;

                add_header Acecss-Control Allow;
                add_header Access-Control-Allow-Headers *;
                add_header Access-Control-Allow-Origin *;
                add_header XDomainRequestAllowed 1;
        }

[wandenberg]

@brooke14, were you able to open the core dump? I'm receiving a message like this "./nginx": not in executable format: File format not recognized "./nginx.core.685" is not a core dump: File format not recognized

[kvendi]

I checked this files - all is ok.

testuser@testuser:~/temp/123$ ls -l
итого 58284
-rw-rw-r-- 1 testuser testuser 54628389 окт  4 13:13 nginx.core.685.gz
-rw-rw-r-- 1 testuser testuser  5029987 окт  4 13:12 nginx.gz
testuser@testuser:~/temp/123$ gunzip ./nginx.core.685.gz 
testuser@testuser:~/temp/123$ gunzip ./nginx.gz 
testuser@testuser:~/temp/123$ gdb ./nginx ./nginx.core.685
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./nginx...(no debugging symbols found)...done.
[New LWP 685]

warning: Unexpected size of section `.reg-xstate/685' in core file.
Core was generated by `/usr/sbin/nginx'.

warning: Unexpected size of section `.reg-xstate/685' in core file.
#0  0x00007fa5c1209490 in ?? ()
(gdb) bt
#0  0x00007fa5c1209490 in ?? ()
#1  0x00007fa4f4340238 in ?? ()
#2  0x0000000000440b10 in ?? ()
#3  0x000000000047d67f in ?? ()
#4  0x00000000004c7d4a in ?? ()
#5  0xfffffffffffffffc in ?? ()
#6  0x0000000004e14ae0 in ?? ()
#7  0x0000000004e14ae0 in ?? ()
#8  0x0000000000d8b661 in ?? ()
#9  0x0000000000455bff in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb) 

[wandenberg]

I wasn't able to open on macOS but as you bt output shows, there isn't useful information :(

[kvendi]

core depends on libraries and environment. May be you wil tell us which info from core dump you need , and we give it to you ?

[wandenberg]

I need to know the lines of the module where calls to ngx_shmtx_lock are done on each process and the name of channel or queue that is associated with the lock in the call.

[kvendi]

static void
ngx_http_push_stream_worker_subscriber_cleanup(ngx_http_push_stream_subscriber_t *worker_subscriber)
{
    ngx_http_push_stream_main_conf_t        *mcf = ngx_http_get_module_main_conf(worker_subscriber->request, ngx_http_push_stream_module);
    ngx_http_push_stream_shm_data_t         *data = mcf->shm_data;
    ngx_slab_pool_t                         *shpool = mcf->shpool;
    ngx_queue_t                             *cur;

    while (!ngx_queue_empty(&worker_subscriber->subscriptions)) {
        cur = ngx_queue_head(&worker_subscriber->subscriptions);
        ngx_http_push_stream_subscription_t *subscription = ngx_queue_data(cur, ngx_http_push_stream_subscription_t, queue);
        **ngx_shmtx_lock(subscription->channel->mutex);**
        NGX_HTTP_PUSH_STREAM_DECREMENT_COUNTER(subscription->channel->subscribers);
        NGX_HTTP_PUSH_STREAM_DECREMENT_COUNTER(subscription->channel_worker_sentinel->subscribers);
        ngx_queue_remove(&subscription->channel_worker_queue);
        ngx_queue_remove(&subscription->queue);
        ngx_shmtx_unlock(subscription->channel->mutex);

        ngx_http_push_stream_send_event(mcf, ngx_cycle->log, subscription->channel, &NGX_HTTP_PUSH_STREAM_EVENT_TYPE_CLIENT_UNSUBSCRIBED, worker_subscriber->request->pool);
    }

    ngx_shmtx_lock(&shpool->mutex);
    ngx_queue_remove(&worker_subscriber->worker_queue);
    NGX_HTTP_PUSH_STREAM_DECREMENT_COUNTER(data->subscribers);
    NGX_HTTP_PUSH_STREAM_DECREMENT_COUNTER(data->ipc[ngx_process_slot].subscribers);
    ngx_shmtx_unlock(&shpool->mutex);
} 

[wandenberg]

I need the same view in all workers and the name of the object, as example in the line ngx_shmtx_lock(subscription->channel->mutex); what was the subscription->channel->id value ?!

[kvendi]

In attach backtrace of each worker and channel objects cache.txt master.txt worker1.txt worker2.txt worker3.txt worker4.txt worker5.txt worker6.txt worker7.txt worker8.txt worker9.txt worker10.txt worker11.txt worker12.txt

[wandenberg]

@kvendi can you test the code on issue245 branch? There was a nginx version that you haven't had the problem? (just to check if something has changed on nginx core that caused the problem on the module)

[kvendi]

we tested several versions (1.8,1.10) - the problem was in both versions. Ok , we will try this branch

[wandenberg]

@kvendi how are going the tests until now?

[kvendi]

We built nginx with this patсh and ran it - now we are looking for results.

[wandenberg]

Hi @kvendi any news? Can we set this issue as solved with the code I asked you to test? I could not reproduce the problem here but following the code with the coredump you provided, the dead lock occurred because a wrong lock was used while doing some operations on message trash list, but locked the channels trash list instead.

[kvendi]

Very strange, but after upgrade with your patch - there was only one problem, still can not catch the problem anymore.

[wandenberg]

You mean the same dead lock or another problem? Did you restart you server while upgrading the last time or just did the binary upgrade commands?

[kvendi]

yes , i have restarted server after upgrade. Reason of problem is unknown because it was restarted very fast

[wandenberg]

After the last restart what is the uptime without issues? Is that greater than the old version?

[kvendi]

Yes, for 3 days there was no problem

[kvendi]

in debug log when it hanging now:

2016/10/21 09:27:50 [debug] 7140#7140: epoll add event: fd:332 op:1 ev:00002001 2016/10/21 09:27:50 [debug] 7140#7140: epoll del event: fd:332 op:2 ev:00000000 2016/10/21 09:27:50 [debug] 7144#7144: epoll add event: fd:332 op:1 ev:00002001 2016/10/21 09:27:50 [debug] 7144#7144: epoll del event: fd:332 op:2 ev:00000000 2016/10/21 09:27:50 [debug] 7148#7148: 2688705 http keepalive handler 2016/10/21 09:27:50 [debug] 7148#7148: 2688705 close http connection: 358 2016/10/21 09:27:50 [debug] 7148#7148: 2688705 event timer del: 358: 1477031344567 2016/10/21 09:27:50 [debug] 7148#7148: 2688705 reusable connection: 0 2016/10/21 09:27:50 [debug] 7148#7148: 2688705 free: 0000000000000000 2016/10/21 09:27:50 [debug] 7148#7148: 2688705 free: 00007F5DC78A3C20, unused: 128

[wandenberg]

the does not help. I need the coredump like you did last time to check where can be a deadlock, or if it is really stuck on module scope now.

[kvendi]

ok, i`l try to catch that moment

[wandenberg]

Any news?

[kvendi]

new gdb out during problem cache.txt master.txt worker1.txt worker2.txt worker3.txt worker4.txt worker5.txt worker6.txt worker7.txt worker8.txt worker9.txt worker10.txt worker11.txt worker12.txt

[wandenberg]

HI @kvendi the gdb output is pretty much the same as the previous one. Can you restart your server and check if the message ngx_http_push_stream_module debugging #245 v1. is on your error log?

[kvendi]

very strange, but I do not see these lines. I rebuild the server, and will update again.

[kvendi]

I`ve recompiled nginx, turned on global debug and now saw ./error.log:2016/10/31 12:50:17 [notice] 12984#12984: ngx_http_push_stream_module debugging #245 v1. Now I am waiting for the problem.

[wandenberg]

Cool @kvendi . Probably the problem will not happens again with the fixed version ;)

[mkonecny]

@wandenberg good news to hear if true! Is this a corner case issue, or something all users of this module should rush to apply? Do you have any idea what environments cause this bug to trigger, as I haven't seen it yet on nginx 1.4.6 on the Ubuntu 14.04 platform

[wandenberg]

@kvendi any news?

@mkonecny anyone using any version with the 9b9dee58bf118aae25c36046a1295462db3e5009 commit can face this bug. This commit introduced the usage of multiple locks instead of one giant lock. This means as example that messages to two different channels can be published exactly on same time. There is a corner case that make the bug appear, one of the cleanup routines should happens at the same time that a message is published on one of the channels that are been cleaned. Is quite impossible to reproduce outside a production environment and since it depends on a alignment of conditions, may be never happens. The bug happens because instead of locking the messages_trash_mutex lock while handling the messages_trash queue I have locked the channels_trash_mutex lock.

[kvendi]

there are no problems , since rebuild server

[boniek83]

Will we get new release with this bug fixed?

[wandenberg]

@boniek83 I merged the fix on master. Will prepare a release soon, let me just update the readme/changelog and some other small things.

[kvendi]

In 1.12.0 still have this problem. When it will be released ?

[wandenberg]

@kvendi How are you compiling the module? Do you have a coredump / stacktrace?

[kvendi]

Now i get 1.12.1 from dotdeb And there is no fix (https://github.com/wandenberg/nginx-push-stream-module/commit/4d7bd27a82652f4d9c3932045238831a6583deaf) in src. I`ve compiled this patch manualy and now it is ok. But there is no patch in repository still

[wandenberg]

I don't know who is the maintainer of dotdeb package. The only thing I can do for help is to release a new tag later to not use "master". Sorry for not have it done for a long time.

[shadeware]

I'm still having this problem even with latest version from master (nginx version is 1.13.3). Nginx lives for less than 1 minute, but then all its workers hang.

Stacktraces from 16-worker setup:

4982
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa582ca4f76 in do_sigsuspend (set=0x7ffd4af97198) at ../sysdeps/unix/sysv/linux/sigsuspend.c:31
31  ../sysdeps/unix/sysv/linux/sigsuspend.c: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4982)):
#0  0x00007fa582ca4f76 in do_sigsuspend (set=0x7ffd4af97198) at ../sysdeps/unix/sysv/linux/sigsuspend.c:31
#1  __GI___sigsuspend (set=0x7ffd4af97198) at ../sysdeps/unix/sysv/linux/sigsuspend.c:41
#2  0x00007fa584830d47 in ngx_master_process_cycle (cycle=0x7fa5858da250) at src/os/unix/ngx_process_cycle.c:163
#3  0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4982) detached]
4983
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4983)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585a4f6e8, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a1091 in ngx_http_push_stream_publisher_handler (r=0x7fa585a4e790) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:110
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585a4e790, ph=0x7fa585a4ebf8) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585a4e790) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585a4e790) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac8ff0) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x0, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4983) detached]
4985
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4985)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585a4f478, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a1091 in ngx_http_push_stream_publisher_handler (r=0x7fa585a4e520) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:110
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585a4e520, ph=0x7fa585a4e988) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585a4e520) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585a4e520) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac9110) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x2, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4985) detached]
4987
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4987)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa584898b75 in ngx_http_push_stream_get_channel (id=0x7fa585b84420, log=0x7fa585a59770, mcf=0x7fa5858e54b0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:109
#3  0x00007fa5848a1073 in ngx_http_push_stream_publisher_handler (r=0x7fa585b834b0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:100
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585b834b0, ph=0x7fa585b83918) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585b834b0) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585b834b0) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aacb510) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x4, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4987) detached]
4989
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4989)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585b36688, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a1091 in ngx_http_push_stream_publisher_handler (r=0x7fa585b35730) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:110
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585b35730, ph=0x7fa585b35b98) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585b35730) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585b35730) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac86f0) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x5, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4989) detached]
4990
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4990)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585bf63f8, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a1091 in ngx_http_push_stream_publisher_handler (r=0x7fa585bf54a0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:110
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585bf54a0, ph=0x7fa585bf5908) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585bf54a0) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585bf54a0) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac7790) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x6, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4990) detached]
4991
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4991)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa586345aa0, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a40d5 in ngx_http_push_stream_validate_channels (r=0x7fa585ddd8b0, requested_channels=0x7fa586345a50, status_code=0x7ffd4af96ef0, explain_error_message=0x7ffd4af96ee8) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:318
#4  0x00007fa5848a3085 in ngx_http_push_stream_subscriber_handler (r=0x7fa585ddd8b0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:94
#5  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585ddd8b0, ph=0x7fa585dddd18) at src/http/ngx_http_core_module.c:1388
#6  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#7  ngx_http_handler (r=0x7fa585ddd8b0) at src/http/ngx_http_core_module.c:843
#8  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585ddd8b0) at src/http/ngx_http_request.c:1948
#9  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac7370) at src/http/ngx_http_request.c:1048
#10 0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#11 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#12 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#13 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x7, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#14 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#15 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#16 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4991) detached]
4992
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4992)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa584898b75 in ngx_http_push_stream_get_channel (id=0x7fa585f13560, log=0x7fa585c68f70, mcf=0x7fa5858e54b0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:109
#3  0x00007fa5848a1073 in ngx_http_push_stream_publisher_handler (r=0x7fa585f125f0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:100
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585f125f0, ph=0x7fa585f12a58) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585f125f0) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585f125f0) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac83f0) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x8, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4992) detached]
4993
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4993)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa3835ad368) at src/core/ngx_shmtx.c:111
#2  0x00007fa58489d705 in ngx_http_push_stream_worker_subscriber_cleanup (worker_subscriber=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1554
#3  ngx_http_push_stream_cleanup_request_context (r=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1528
#4  0x00007fa584899f4e in ngx_http_push_stream_run_cleanup_pool_handler (p=<optimized out>, handler=0xffffffffffffffff) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:863
#5  ngx_http_push_stream_send_response_finalize (r=0x7fa585b9c6f0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:878
#6  0x00007fa5848298e0 in ngx_event_expire_timers () at src/event/ngx_event_timer.c:94
#7  0x00007fa584828bef in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:256
#8  0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#9  0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x9, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#10 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#11 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#12 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4993) detached]
4994
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4994)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585bf1e28, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a1091 in ngx_http_push_stream_publisher_handler (r=0x7fa585bf0ed0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:110
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585bf0ed0, ph=0x7fa585bf1338) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585bf0ed0) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585bf0ed0) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aacd790) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0xa, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4994) detached]
4995
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4995)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f170) at src/core/ngx_shmtx.c:111
#2  0x00007fa58489920e in ngx_http_push_stream_delete_channels_data (data=0x7fa38359f000) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:92
#3  0x00007fa5848a5c58 in ngx_http_push_stream_memory_cleanup () at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1121
#4  ngx_http_push_stream_memory_cleanup_timer_wake_handler (ev=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1329
#5  0x00007fa5848298e0 in ngx_event_expire_timers () at src/event/ngx_event_timer.c:94
#6  0x00007fa584828bef in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:256
#7  0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#8  0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0xb, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#9  0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#10 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#11 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4995) detached]
4996
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4996)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585e027c8, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a1091 in ngx_http_push_stream_publisher_handler (r=0x7fa585e01870) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_publisher.c:110
#4  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585e01870, ph=0x7fa585e01cd8) at src/http/ngx_http_core_module.c:1388
#5  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#6  ngx_http_handler (r=0x7fa585e01870) at src/http/ngx_http_core_module.c:843
#7  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585e01870) at src/http/ngx_http_request.c:1948
#8  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aad19f0) at src/http/ngx_http_request.c:1048
#9  0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#10 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#11 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#12 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0xc, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#13 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#14 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#15 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4996) detached]
4997
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4997)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa3835ad368) at src/core/ngx_shmtx.c:111
#2  0x00007fa584899091 in ngx_http_push_stream_ensure_qtd_of_messages (data=<optimized out>, channel=<optimized out>, max_messages=0, expired=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:50
#3  0x00007fa58489b6f7 in ngx_http_push_stream_collect_expired_messages_data (data=0x7fa38359f000, force=0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1059
#4  ngx_http_push_stream_buffer_cleanup () at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1144
#5  ngx_http_push_stream_buffer_timer_wake_handler (ev=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1336
#6  0x00007fa5848298e0 in ngx_event_expire_timers () at src/event/ngx_event_timer.c:94
#7  0x00007fa584828bef in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:256
#8  0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#9  0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0xd, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#10 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#11 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#12 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4997) detached]
4998
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4998)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa3835ad368) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a594d in ngx_http_push_stream_assing_subscription_to_channel (shpool=0x7fa3805e6000, channel=0x7fa3835b9100, subscription=0x7fa586ae0b68, subscriptions=0x7fa586ae06e0, log=0x7fa585c68b70) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:626
#3  0x00007fa5848a3aad in ngx_http_push_stream_subscriber_assign_channel (mcf=<optimized out>, r=<optimized out>, requested_channel=0x7fa586025080, if_modified_since=<optimized out>, tag=<optimized out>, last_event_id=<optimized out>, subscriber=<optimized out>, cf=<optimized out>, temp_pool=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:278
#4  ngx_http_push_stream_subscriber_handler (r=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:137
#5  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa58605fb90, ph=0x7fa58605fff8) at src/http/ngx_http_core_module.c:1388
#6  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#7  ngx_http_handler (r=0x7fa58605fb90) at src/http/ngx_http_core_module.c:843
#8  0x00007fa5848492db in ngx_http_process_request (r=0x7fa58605fb90) at src/http/ngx_http_request.c:1948
#9  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aaca370) at src/http/ngx_http_request.c:1048
#10 0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#11 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#12 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#13 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0xe, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#14 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#15 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#16 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4998) detached]
4999
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 4999)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa5867bff10, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a40d5 in ngx_http_push_stream_validate_channels (r=0x7fa585e5bc90, requested_channels=0x7fa5867bfec0, status_code=0x7ffd4af96ef0, explain_error_message=0x7ffd4af96ee8) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:318
#4  0x00007fa5848a3085 in ngx_http_push_stream_subscriber_handler (r=0x7fa585e5bc90) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:94
#5  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585e5bc90, ph=0x7fa585e5c0f8) at src/http/ngx_http_core_module.c:1388
#6  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#7  ngx_http_handler (r=0x7fa585e5bc90) at src/http/ngx_http_core_module.c:843
#8  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585e5bc90) at src/http/ngx_http_request.c:1948
#9  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aacdd30) at src/http/ngx_http_request.c:1048
#10 0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#11 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#12 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#13 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0xf, name=0x7fa5848ac0d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#14 0x00007fa584830b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#15 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#16 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 4999) detached]
9944
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 9944)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa3835ad368) at src/core/ngx_shmtx.c:111
#2  0x00007fa58489d705 in ngx_http_push_stream_worker_subscriber_cleanup (worker_subscriber=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1554
#3  ngx_http_push_stream_cleanup_request_context (r=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1528
#4  0x00007fa584899f4e in ngx_http_push_stream_run_cleanup_pool_handler (p=<optimized out>, handler=0xffffffffffffffff) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:863
#5  ngx_http_push_stream_send_response_finalize (r=0x7fa585dece70) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:878
#6  0x00007fa584899485 in ngx_http_push_stream_delete_channels_data (data=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:131
#7  0x00007fa5848a66bc in ngx_http_push_stream_delete_channels () at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:78
#8  ngx_http_push_stream_delete_worker_channel () at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:194
#9  ngx_http_push_stream_channel_handler (ev=0x7fa37aac71f0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_ipc.c:284
#10 0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#11 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#12 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#13 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x1, name=0x7fa5848ac0d9 "worker process", respawn=1) at src/os/unix/ngx_process.c:198
#14 0x00007fa584831142 in ngx_reap_children (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:621
#15 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:174
#16 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 9944) detached]
9952
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7fa584abe740 (LWP 9952)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007fa584817938 in ngx_shmtx_lock (mtx=0x7fa38359f0b0) at src/core/ngx_shmtx.c:111
#2  0x00007fa5848a0059 in ngx_http_push_stream_find_channel (id=0x7fa585aefcb8, log=0x0, mcf=0x0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_rbtree_util.c:87
#3  0x00007fa5848a40d5 in ngx_http_push_stream_validate_channels (r=0x7fa585ae27b0, requested_channels=0x7fa585aefc68, status_code=0x7ffd4af96ef0, explain_error_message=0x7ffd4af96ee8) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:318
#4  0x00007fa5848a3085 in ngx_http_push_stream_subscriber_handler (r=0x7fa585ae27b0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_subscriber.c:94
#5  0x00007fa584840496 in ngx_http_core_content_phase (r=0x7fa585ae27b0, ph=0x7fa585ae2c18) at src/http/ngx_http_core_module.c:1388
#6  0x00007fa58483ea25 in ngx_http_core_run_phases (r=<optimized out>) at src/http/ngx_http_core_module.c:860
#7  ngx_http_handler (r=0x7fa585ae27b0) at src/http/ngx_http_core_module.c:843
#8  0x00007fa5848492db in ngx_http_process_request (r=0x7fa585ae27b0) at src/http/ngx_http_request.c:1948
#9  0x00007fa58484afb9 in ngx_http_process_request_line (rev=0x7fa37aac7df0) at src/http/ngx_http_request.c:1048
#10 0x00007fa5848335c2 in ngx_epoll_process_events (cycle=0x7fa5858da250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#11 0x00007fa584828b98 in ngx_process_events_and_timers (cycle=0x7fa5858da250) at src/event/ngx_event.c:242
#12 0x00007fa584831eb7 in ngx_worker_process_cycle (cycle=0x7fa5858da250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#13 0x00007fa58482fec7 in ngx_spawn_process (cycle=0x7fa5858da250, proc=0x7fa584831e20 <ngx_worker_process_cycle>, data=0x3, name=0x7fa5848ac0d9 "worker process", respawn=3) at src/os/unix/ngx_process.c:198
#14 0x00007fa584831142 in ngx_reap_children (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:621
#15 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:174
#16 0x00007fa584806747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 9952) detached]

Stacktraces from single worker setup:

19836
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f77b25ebf76 in do_sigsuspend (set=0x7ffd55c46d28) at ../sysdeps/unix/sysv/linux/sigsuspend.c:31
31  ../sysdeps/unix/sysv/linux/sigsuspend.c: No such file or directory.

Thread 1 (Thread 0x7f77b4405740 (LWP 19836)):
#0  0x00007f77b25ebf76 in do_sigsuspend (set=0x7ffd55c46d28) at ../sysdeps/unix/sysv/linux/sigsuspend.c:31
#1  __GI___sigsuspend (set=0x7ffd55c46d28) at ../sysdeps/unix/sysv/linux/sigsuspend.c:41
#2  0x00007f77b4177d47 in ngx_master_process_cycle (cycle=0x7f77b5cb5250) at src/os/unix/ngx_process_cycle.c:163
#3  0x00007f77b414d747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 19836) detached]
19837
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
85  ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: No such file or directory.

Thread 1 (Thread 0x7f77b4405740 (LWP 19837)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007f77b415e938 in ngx_shmtx_lock (mtx=0x7f75b2ef4428) at src/core/ngx_shmtx.c:111
#2  0x00007f77b41e4705 in ngx_http_push_stream_worker_subscriber_cleanup (worker_subscriber=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1554
#3  ngx_http_push_stream_cleanup_request_context (r=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:1528
#4  0x00007f77b41e0f4e in ngx_http_push_stream_run_cleanup_pool_handler (p=<optimized out>, handler=0xffffffffffffffff) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:863
#5  ngx_http_push_stream_send_response_finalize (r=0x7f77b5f37340) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:878
#6  0x00007f77b41e0485 in ngx_http_push_stream_delete_channels_data (data=<optimized out>) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:131
#7  0x00007f77b41ed6bc in ngx_http_push_stream_delete_channels () at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:78
#8  ngx_http_push_stream_delete_worker_channel () at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_utils.c:194
#9  ngx_http_push_stream_channel_handler (ev=0x7f75aa40e1f0) at /home/packages/nginx/nginx/debian/modules/../../../nginx-push-stream-module/src/ngx_http_push_stream_module_ipc.c:284
#10 0x00007f77b417a5c2 in ngx_epoll_process_events (cycle=0x7f77b5cb5250, timer=<optimized out>, flags=1) at src/event/modules/ngx_epoll_module.c:902
#11 0x00007f77b416fb98 in ngx_process_events_and_timers (cycle=0x7f77b5cb5250) at src/event/ngx_event.c:242
#12 0x00007f77b4178eb7 in ngx_worker_process_cycle (cycle=0x7f77b5cb5250, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:749
#13 0x00007f77b4176ec7 in ngx_spawn_process (cycle=0x7f77b5cb5250, proc=0x7f77b4178e20 <ngx_worker_process_cycle>, data=0x0, name=0x7f77b41f30d9 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
#14 0x00007f77b4177b80 in ngx_start_worker_processes (cycle=<optimized out>, n=<optimized out>, type=-3) at src/os/unix/ngx_process_cycle.c:358
#15 ngx_master_process_cycle (cycle=<optimized out>) at src/os/unix/ngx_process_cycle.c:130
#16 0x00007f77b414d747 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375
[Inferior 1 (process 19837) detached]

I built nginx from source manually, looks like pushstream is at the latest version:

$ strings /usr/sbin/nginx | fgrep 0.5.4
0.5.4

[wandenberg]

Hi @shadeware Sorry for the long delay. Can you share info about which OS are you using; your configuration; and your use case? (WebSocket only or not, multiple users per channel? ... )

[shadeware]

Hi! My OS is Ubuntu 14.04.06 LTS. Configuration varies, but for example

postpone_output 1;

push_stream_shared_memory_size                8192m;
push_stream_max_messages_stored_per_channel   1000;
push_stream_message_ttl                       1m;
push_stream_subscriber_connection_ttl         1m;
push_stream_channel_inactivity_time           1m;

push_stream_max_channel_id_length             200;

push_stream_longpolling_connection_ttl        1m;
push_stream_timeout_with_body                 off;

...

        location /publish {
            push_stream_publisher the_publisher;
            push_stream_channels_path $arg_id;
            push_stream_store_messages on;
            client_max_body_size 32k;
            client_body_buffer_size 32k;
        }

        location ~ /subscribe/(.*) {
                push_stream_subscriber;
                push_stream_channels_path                   $1;
                push_stream_message_template                "~time~\n~id~\n~channel~\n~size~\n~text~";

                if ($arg_qs = "on") {
                    push_stream_last_received_message_time "$arg_time";
                    push_stream_last_received_message_tag  "$arg_tag";
                    push_stream_last_event_id              "$arg_eventid";
        }

Not WebSocket-only, it has all kinds of subscribers including mobile applications, and maybe WebSocket ones as well. Most of channels have multiple subscribers.

[wandenberg]

Hi, I was not able to reproduce it locally, if it is easy for you to reproduce, can you generate coredumps of the Nginx workers?