search

icing / mod_h2

HTTP/2 module for Apache httpd
https://icing.github.io/mod_h2/
Apache License 2.0
256 stars 41 forks source link

Child process sometimes hang, peg cpu @ 100% #234

Closed Adam7288 closed 2 years ago

Adam7288 commented 2 years ago

Here is pstack trace for one of the hung processes pegging cpu @ 100% - looks like some kind of deadlock

0 __lll_unlock_wake () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:371

1 0x00007f2d659f0f9e in _L_unlock_738 () from /lib64/libpthread.so.0

2 0x00007f2d659f0f10 in __pthread_mutex_unlock_usercnt (decr=1, mutex=0x7f2c7c5248f8) at pthread_mutex_unlock.c:55

3 __GI___pthread_mutex_unlock (mutex=0x7f2c7c5248f8) at pthread_mutex_unlock.c:330

4 0x00007f2d57cb140d in h2_beam_receive () from /etc/httpd/modules/mod_http2.so

5 0x00007f2d57cc8ee3 in buffer_output_receive () from /etc/httpd/modules/mod_http2.so

6 0x00007f2d57ccb1ec in stream_data_cb () from /etc/httpd/modules/mod_http2.so

7 0x00007f2d66cba171 in nghttp2_session_pack_data () from /lib64/libnghttp2.so.14

8 0x00007f2d66cbaedd in nghttp2_session_mem_send_internal () from /lib64/libnghttp2.so.14

9 0x00007f2d66cbbae9 in nghttp2_session_send () from /lib64/libnghttp2.so.14

10 0x00007f2d57cc7544 in h2_session_send () from /etc/httpd/modules/mod_http2.so

11 0x00007f2d57cc777a in h2_session_process () from /etc/httpd/modules/mod_http2.so

12 0x00007f2d57cb2149 in h2_c1_run () from /etc/httpd/modules/mod_http2.so

13 0x00007f2d57cb2569 in h2_c1_hook_process_connection () from /etc/httpd/modules/mod_http2.so

14 0x00005571f66c33c0 in ap_run_process_connection (c=c@entry=0x7f2d4006d5e0) at connection.c:42

15 0x00007f2d5a8ab40a in process_socket (thd=thd@entry=0x5571f72c4510, p=, sock=, cs=0x7f2d4006d530, my_child_num=my_child_num@entry=11, my_thread_num=my_thread_num@entry=12) at event.c:1086

16 0x00007f2d5a8ae6ae in worker_thread (thd=0x5571f72c4510, dummy=) at event.c:2179

17 0x00007f2d659edea5 in start_thread (arg=0x7f2d3a7f4700) at pthread_create.c:307

18 0x00007f2d65512b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

alexskynet commented 2 years ago

pstack of httpd process at 100% running 2.0.5 timout30

pstack 3560 Thread 27 (Thread 0x7fe8649ee700 (LWP 3576)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec56ead90, wctx=0x558ec56e8f98) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fe8641ed700 (LWP 3577)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec56ef090, wctx=0x558ec56e8ff0) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fe8639ec700 (LWP 3578)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec56f3390, wctx=0x558ec56e9048) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fe8631eb700 (LWP 3581)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec56f7690, wctx=0x558ec56e90a0) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fe8629ea700 (LWP 3582)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec56fb990, wctx=0x558ec56e90f8) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fe8621e9700 (LWP 3585)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec56ffc90, wctx=0x558ec56e9150) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fe8619e8700 (LWP 3588)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5703f90, wctx=0x558ec56e91a8) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fe8611e7700 (LWP 3589)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5708290, wctx=0x558ec56e9200) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fe8609e6700 (LWP 3590)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec570c590, wctx=0x558ec56e9258) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fe8601e5700 (LWP 3591)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5710890, wctx=0x558ec56e92b0) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fe85f9e4700 (LWP 3592)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5714b90, wctx=0x558ec56e9308) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fe85f1e3700 (LWP 3595)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5718e90, wctx=0x558ec56e9360) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fe85e9e2700 (LWP 3596)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec571d190, wctx=0x558ec56e93b8) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fe85e1e1700 (LWP 3599)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5721490, wctx=0x558ec56e9410) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fe85d9e0700 (LWP 3600)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5725790, wctx=0x558ec56e9468) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fe85d1df700 (LWP 3603)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5729a90, wctx=0x558ec56e94c0) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fe85c9de700 (LWP 3604)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec572dd90, wctx=0x558ec56e9518) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fe85c1dd700 (LWP 3608)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5732090, wctx=0x558ec56e9570) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fe85b9dc700 (LWP 3611)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5736390, wctx=0x558ec56e95c8) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fe85b1db700 (LWP 3612)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec573a690, wctx=0x558ec56e9620) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fe85a9da700 (LWP 3615)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec573e990, wctx=0x558ec56e9678) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fe85a1d9700 (LWP 3618)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5742c90, wctx=0x558ec56e96d0) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fe8599d8700 (LWP 3619)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec5746f90, wctx=0x558ec56e9728) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fe8591d7700 (LWP 3620)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec574b290, wctx=0x558ec56e9780) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fe8589d6700 (LWP 3621)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in slot_run (thread=0x558ec574f590, wctx=0x558ec56e97d8) at h2_workers.c:329

2 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fe84cff9700 (LWP 3668)):

0 0x00007fe86ff431d4 in h2_brigade_mem_size (bb=0x7fe840002db8) at h2_util.c:1448

1 0x00007fe86ff3b3a4 in buffer_output_receive (stream=stream@entry=0x7fe840002970) at h2_stream.c:925

2 0x00007fe86ff3d96c in stream_data_cb (ng2s=, stream_id=1, buf=, length=16375, data_flags=0x7fe84cff89e4, source=, puser=0x7fe83802fd40) at h2_stream.c:1416

3 0x00007fe87fd6a1a1 in nghttp2_session_pack_data () from /lib64/libnghttp2.so.14

4 0x00007fe87fd6af0d in nghttp2_session_mem_send_internal () from /lib64/libnghttp2.so.14

5 0x00007fe87fd6bb19 in nghttp2_session_send () from /lib64/libnghttp2.so.14

6 0x00007fe86ff39afc in h2_session_send (session=session@entry=0x7fe83802fd40) at h2_session.c:1270

7 0x00007fe86ff39d6c in h2_session_process (session=0x7fe83802fd40, async=) at h2_session.c:1815

8 0x00007fe86ff24bc9 in h2_c1_run (c=c@entry=0x7fe850005798) at h2_c1.c:130

9 0x00007fe86ff24fdd in h2_c1_hook_process_connection (c=0x7fe850005798) at h2_c1.c:278

10 0x0000558ec27273c0 in ap_run_process_connection ()

11 0x00007fe872f29d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

12 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0

13 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fe87fe328c0 (LWP 3560)):

0 0x00007fe87ea87017 in pthread_join () from /lib64/libpthread.so.0

1 0x00007fe87ecc804b in apr_thread_join () from /lib64/libapr-1.so.0

2 0x00007fe872f286f3 in join_workers () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007fe872f2b061 in child_main () from /etc/httpd/modules/mod_mpm_worker.so

4 0x00007fe872f2b48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so

5 0x00007fe872f2b534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so

6 0x00007fe872f2c0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so

7 0x0000558ec272a1ce in ap_run_mpm ()

8 0x0000558ec271388d in main ()

alexskynet commented 2 years ago

Another 100%

hread 1 (process 3668):

0 0x00007fe86ff431b4 in h2_brigade_concat_length () from /etc/httpd/modules/mod_http2.so

1 0x00007fe8380304a0 in ?? ()

2 0x00007fe87f342ef9 in heap_bucket_destroy () from /lib64/libaprutil-1.so.0

3 0x00007fe86ff26a62 in h2_c1_io_init () from /etc/httpd/modules/mod_http2.so

4 0x0000000000000001 in ?? ()

5 0x0000000000000000 in ?? ()

alexskynet commented 2 years ago

And another one: pstack 3560 Thread 27 (Thread 0x7fe8649ee700 (LWP 3576)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7fe8641ed700 (LWP 3577)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7fe8639ec700 (LWP 3578)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7fe8631eb700 (LWP 3581)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7fe8629ea700 (LWP 3582)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7fe8621e9700 (LWP 3585)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7fe8619e8700 (LWP 3588)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7fe8611e7700 (LWP 3589)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7fe8609e6700 (LWP 3590)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7fe8601e5700 (LWP 3591)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fe85f9e4700 (LWP 3592)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fe85f1e3700 (LWP 3595)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fe85e9e2700 (LWP 3596)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fe85e1e1700 (LWP 3599)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fe85d9e0700 (LWP 3600)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fe85d1df700 (LWP 3603)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fe85c9de700 (LWP 3604)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fe85c1dd700 (LWP 3608)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fe85b9dc700 (LWP 3611)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fe85b1db700 (LWP 3612)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fe85a9da700 (LWP 3615)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fe85a1d9700 (LWP 3618)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fe8599d8700 (LWP 3619)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fe8591d7700 (LWP 3620)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fe8589d6700 (LWP 3621)):

0 0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so

2 0x0000558ec56e8df0 in ?? ()

3 0x0000558ec56e8dc0 in ?? ()

4 0x0000558ec56e8dd0 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fe84cff9700 (LWP 3668)):

0 0x00007fe86ff431bb in h2_brigade_concat_length () from /etc/httpd/modules/mod_http2.so

1 0x00007fe8380304a0 in ?? ()

2 0x00007fe87f342ef9 in heap_bucket_destroy () from /lib64/libaprutil-1.so.0

3 0x00007fe86ff26a62 in h2_c1_io_init () from /etc/httpd/modules/mod_http2.so

4 0x0000000000000001 in ?? ()

5 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fe87fe328c0 (LWP 3560)):

0 0x00007fe87ea87017 in pthread_join () from /lib64/libpthread.so.0

1 0x00007fe87ecc804b in apr_thread_join () from /lib64/libapr-1.so.0

2 0x00007fe872f286f3 in join_workers () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007fe872f2b061 in child_main () from /etc/httpd/modules/mod_mpm_worker.so

4 0x00007fe872f2b48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so

5 0x00007fe872f2b534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so

6 0x00007fe872f2c0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so

7 0x0000558ec272a1ce in ap_run_mpm ()

8 0x0000558ec271388d in main ()

icing commented 2 years ago

Thanks for the traces. The first one really helped. The other seem to have the symbols/source mixed up.

Anyways, here is v2.0.6 with three fixes. Hope these work for you as well.

alexskynet commented 2 years ago

Thank you Stefan

Just compiled and installed on one server

I'll let you know soon

Best regards

Il 22 settembre 2022 20:23:49 CEST, Stefan Eissing @.***> ha scritto:

Thanks for the traces. The first one really helped. The other seem to have the symbols/source mixed up.

Anyways, here is v2.0.6 with three fixes. Hope these work for you as well.

-- Reply to this email directly or view it on GitHub: https://github.com/icing/mod_h2/issues/234#issuecomment-1255392015 You are receiving this because you were mentioned.

alexskynet commented 2 years ago

Tested 2.0.6 for about 20 minutes

Load on server greatly increases

It seems httpd processes still hang, but I will investigate better tomorrow morning since I'm operating from a tablet now.

I saw your Apache commit proposal: so it comes from Apache if I uderstand.

Thank you for you time

Best regards

Alessandro

Il 22 settembre 2022 20:23:49 CEST, Stefan Eissing @.***> ha scritto:

Thanks for the traces. The first one really helped. The other seem to have the symbols/source mixed up.

Anyways, here is v2.0.6 with three fixes. Hope these work for you as well.

-- Reply to this email directly or view it on GitHub: https://github.com/icing/mod_h2/issues/234#issuecomment-1255392015 You are receiving this because you were mentioned.

Message ID: @.***>

alexskynet commented 2 years ago

Here I am Unfortunately as soon as I start httpd with 2.0.6 all httpd processes go to 100% CPU.

Here are a couple of pstack(s) `pstack 3491 Thread 75 (Thread 0x7ff9fb98f700 (LWP 3496)):

0 0x00007ffa14d41b43 in select () from /lib64/libc.so.6

1 0x00007ffa15468725 in apr_sleep () from /lib64/libapr-1.so.0

2 0x00007ffa0a351066 in wd_worker () from /etc/httpd/modules/mod_watchdog.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 74 (Thread 0x7ff9fb18e700 (LWP 3498)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 73 (Thread 0x7ff9fa98d700 (LWP 3499)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 72 (Thread 0x7ff9fa18c700 (LWP 3502)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 71 (Thread 0x7ff9f998b700 (LWP 3503)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 70 (Thread 0x7ff9f918a700 (LWP 3505)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 69 (Thread 0x7ff9f8989700 (LWP 3508)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 68 (Thread 0x7ff9f3fff700 (LWP 3511)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 67 (Thread 0x7ff9f37fe700 (LWP 3513)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 66 (Thread 0x7ff9f2ffd700 (LWP 3517)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 65 (Thread 0x7ff9f27fc700 (LWP 3516)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068858e0, wctx=0x55ce06883ae8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 64 (Thread 0x7ff9f17fa700 (LWP 3520)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7ff9f1ffb700 (LWP 3521)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce06889be0, wctx=0x55ce06883b40) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7ff9f0ff9700 (LWP 3524)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7ff9f07f8700 (LWP 3526)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce0688dee0, wctx=0x55ce06883b98) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7ff9efff7700 (LWP 3527)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 59 (Thread 0x7ff9ef7f6700 (LWP 3531)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068921e0, wctx=0x55ce06883bf0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7ff9eeff5700 (LWP 3530)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7ff9ee7f4700 (LWP 3533)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7ff9edff3700 (LWP 3535)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068964e0, wctx=0x55ce06883c48) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7ff9ed7f2700 (LWP 3536)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7ff9ecff1700 (LWP 3539)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7ff9ec7f0700 (LWP 3538)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce0689a7e0, wctx=0x55ce06883ca0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7ff9ebfef700 (LWP 3542)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce0689eae0, wctx=0x55ce06883cf8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7ff9eb7ee700 (LWP 3544)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068a2de0, wctx=0x55ce06883d50) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7ff9eafed700 (LWP 3546)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068a70e0, wctx=0x55ce06883da8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7ff9ea7ec700 (LWP 3548)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068ab3e0, wctx=0x55ce06883e00) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7ff9e9feb700 (LWP 3549)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068af6e0, wctx=0x55ce06883e58) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7ff9e97ea700 (LWP 3550)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068b39e0, wctx=0x55ce06883eb0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7ff9e8fe9700 (LWP 3551)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068b7ce0, wctx=0x55ce06883f08) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7ff9e87e8700 (LWP 3552)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068bbfe0, wctx=0x55ce06883f60) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7ff9e7fe7700 (LWP 3553)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068c02e0, wctx=0x55ce06883fb8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7ff9e77e6700 (LWP 3554)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068c45e0, wctx=0x55ce06884010) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7ff9e6fe5700 (LWP 3555)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068c88e0, wctx=0x55ce06884068) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7ff9e67e4700 (LWP 3556)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068ccbe0, wctx=0x55ce068840c0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7ff9e5fe3700 (LWP 3557)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068d0ee0, wctx=0x55ce06884118) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7ff9e57e2700 (LWP 3558)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068d51e0, wctx=0x55ce06884170) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7ff9e4fe1700 (LWP 3559)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068d94e0, wctx=0x55ce068841c8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7ff9e47e0700 (LWP 3560)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068dd7e0, wctx=0x55ce06884220) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7ff9e3fdf700 (LWP 3561)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068e1ae0, wctx=0x55ce06884278) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7ff9e37de700 (LWP 3562)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068e73a0, wctx=0x55ce068842d0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7ff9e2fdd700 (LWP 3563)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068eb3c0, wctx=0x55ce06884328) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7ff9e1fdb700 (LWP 3571)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7ff9e17da700 (LWP 3573)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7ff9e0fd9700 (LWP 3575)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7ff9dbfff700 (LWP 3577)):

0 h2_session_process (session=0x7ff9a4016ba0, async=) at h2_session.c:1814

1 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9dc00ba38) at h2_c1.c:130

2 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9dc00ba38) at h2_c1.c:278

3 0x000055ce02ab13c0 in ap_run_process_connection ()

4 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

5 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

6 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7ff9db7fe700 (LWP 3579)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7ff9daffd700 (LWP 3581)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7ff9da7fc700 (LWP 3583)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7ff9d9ffb700 (LWP 3585)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7ff9d97fa700 (LWP 3587)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7ff9d8ff9700 (LWP 3589)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7ff9d87f8700 (LWP 3590)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7ff9d7ff7700 (LWP 3592)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7ff9d77f6700 (LWP 3595)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7ff9d6ff5700 (LWP 3597)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7ff9d67f4700 (LWP 3599)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7ff9d5ff3700 (LWP 3601)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7ff9d57f2700 (LWP 3603)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7ff9d4ff1700 (LWP 3605)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7ff9d47f0700 (LWP 3607)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0

2 0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0

3 0x000055ce02ad2e71 in ap_core_input_filter ()

4 0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so

5 0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so

6 0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1

7 0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1

8 0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1

9 0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1

10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1

11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1

12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1

13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so

14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so

15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9cc084ee0) at h2_c1_io.c:486

16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9cc084ee0) at h2_c1_io.c:513

17 0x00007ffa066da08b in h2_session_process (session=0x7ff9cc084ee0, async=) at h2_session.c:1865

18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9dc005798) at h2_c1.c:130

19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9dc005798) at h2_c1.c:278

20 0x000055ce02ab13c0 in ap_run_process_connection ()

21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7ff9d3fef700 (LWP 3609)):

0 0x00007ffa14d3fddd in poll () from /lib64/libc.so.6

1 0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so

2 0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so

3 0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so

4 0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so

5 0x000055ce02aaf558 in ap_scan_script_header_err_core_ex ()

6 0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so

7 0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so

8 0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so

9 0x000055ce02a9f550 in ap_run_handler ()

10 0x000055ce02a9fa99 in ap_invoke_handler ()

11 0x000055ce02add40c in ap_internal_redirect ()

12 0x00007ffa0c3961ec in handler_redirect () from /etc/httpd/modules/mod_rewrite.so

13 0x000055ce02a9f550 in ap_run_handler ()

14 0x000055ce02a9fa99 in ap_invoke_handler ()

15 0x000055ce02ade0ea in ap_process_async_request ()

16 0x000055ce02ade3ce in ap_process_request ()

17 0x000055ce02ada304 in ap_process_http_connection ()

18 0x000055ce02ab13c0 in ap_run_process_connection ()

19 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

20 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

21 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7ff9d37ee700 (LWP 3610)):

0 0x000055ce02ab42d7 in ap_run_mpm_query ()

1 0x000055ce02ab52be in ap_mpm_query ()

2 0x00007ffa066c4c77 in h2_c1_run (c=c@entry=0x7ff9dc007878) at h2_c1.c:142

3 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9dc007878) at h2_c1.c:278

4 0x000055ce02ab13c0 in ap_run_process_connection ()

5 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

6 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

7 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7ff9d2fed700 (LWP 3613)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7ff9d27ec700 (LWP 3614)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7ff9d1feb700 (LWP 3615)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7ff9d17ea700 (LWP 3616)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7ff9d0fe9700 (LWP 3617)):

0 0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6

1 0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0

2 0x00007ffa096ca708 in listener_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7ff9cbfff700 (LWP 3629)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7ff9cb7fe700 (LWP 3643)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7ff9caffd700 (LWP 3644)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7ff9ca7fc700 (LWP 3645)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ff9c9ffb700 (LWP 3646)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7ff9c97fa700 (LWP 3707)):

0 0x00007ffa15229de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa019d8253 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d0337 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffa165d28c0 (LWP 3491)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x000055ce02ab68e7 in ap_mpm_podx_check ()

2 0x00007ffa096cb004 in child_main () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa096cb48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so

4 0x00007ffa096cb534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so

5 0x00007ffa096cc0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so

6 0x000055ce02ab41ce in ap_run_mpm ()

7 0x000055ce02a9d88d in main ()

[root@argo ~]# pstack 3492 Thread 59 (Thread 0x7ff9fb98f700 (LWP 3497)):

0 0x00007ffa14d41b43 in select () from /lib64/libc.so.6

1 0x00007ffa15468725 in apr_sleep () from /lib64/libapr-1.so.0

2 0x00007ffa0a350f02 in wd_worker () from /etc/httpd/modules/mod_watchdog.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7ff9fb18e700 (LWP 3500)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068858e0, wctx=0x55ce06883ae8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7ff9fa98d700 (LWP 3501)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce06889be0, wctx=0x55ce06883b40) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7ff9fa18c700 (LWP 3504)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce0688dee0, wctx=0x55ce06883b98) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7ff9f998b700 (LWP 3506)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068921e0, wctx=0x55ce06883bf0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7ff9f918a700 (LWP 3507)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068964e0, wctx=0x55ce06883c48) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7ff9f8989700 (LWP 3509)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce0689a7e0, wctx=0x55ce06883ca0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7ff9f8188700 (LWP 3510)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce0689eae0, wctx=0x55ce06883cf8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7ff9f7987700 (LWP 3512)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068a2de0, wctx=0x55ce06883d50) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7ff9f7186700 (LWP 3514)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068a70e0, wctx=0x55ce06883da8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7ff9f6985700 (LWP 3515)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068ab3e0, wctx=0x55ce06883e00) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7ff9f6184700 (LWP 3518)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068af6e0, wctx=0x55ce06883e58) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7ff9f5983700 (LWP 3519)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068b39e0, wctx=0x55ce06883eb0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7ff9f5182700 (LWP 3522)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068b7ce0, wctx=0x55ce06883f08) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7ff9f4981700 (LWP 3523)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068bbfe0, wctx=0x55ce06883f60) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7ff9f4180700 (LWP 3525)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068c02e0, wctx=0x55ce06883fb8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7ff9f397f700 (LWP 3528)):

0 0x00007ffa14d3fddd in poll () from /lib64/libc.so.6

1 0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so

2 0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so

3 0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so

4 0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so

5 0x000055ce02aaf558 in ap_scan_script_header_err_core_ex ()

6 0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so

7 0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so

8 0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so

9 0x000055ce02a9f550 in ap_run_handler ()

10 0x000055ce02a9fa99 in ap_invoke_handler ()

11 0x000055ce02ade0ea in ap_process_async_request ()

12 0x000055ce02ade3ce in ap_process_request ()

13 0x00007ffa066c8744 in c2_process (c=0x7ff91c0ce2a0, conn_ctx=0x7ff91c0ce840) at h2_c2.c:603

14 h2_c2_hook_process (c=0x7ff91c0ce2a0) at h2_c2.c:633

15 0x000055ce02ab13c0 in ap_run_process_connection ()

16 0x00007ffa066c957a in h2_c2_process (c2=c2@entry=0x7ff91c0ce2a0, thread=thread@entry=0x55ce068c45e0, worker_id=) at h2_c2.c:545

17 0x00007ffa066e44ae in slot_run (thread=0x55ce068c45e0, wctx=0x55ce06884010) at h2_workers.c:289

18 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

19 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7ff9f317e700 (LWP 3529)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068c88e0, wctx=0x55ce06884068) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7ff9f297d700 (LWP 3532)):

0 0x00007ffa14d3fddd in poll () from /lib64/libc.so.6

1 0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so

2 0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so

3 0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so

4 0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so

5 0x000055ce02aaf558 in ap_scan_script_header_err_core_ex ()

6 0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so

7 0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so

8 0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so

9 0x000055ce02a9f550 in ap_run_handler ()

10 0x000055ce02a9fa99 in ap_invoke_handler ()

11 0x000055ce02ade0ea in ap_process_async_request ()

12 0x000055ce02ade3ce in ap_process_request ()

13 0x00007ffa066c8744 in c2_process (c=0x7ff92804c0f0, conn_ctx=0x7ff92804c690) at h2_c2.c:603

14 h2_c2_hook_process (c=0x7ff92804c0f0) at h2_c2.c:633

15 0x000055ce02ab13c0 in ap_run_process_connection ()

16 0x00007ffa066c957a in h2_c2_process (c2=c2@entry=0x7ff92804c0f0, thread=thread@entry=0x55ce068ccbe0, worker_id=) at h2_c2.c:545

17 0x00007ffa066e44ae in slot_run (thread=0x55ce068ccbe0, wctx=0x55ce068840c0) at h2_workers.c:289

18 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

19 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7ff9f217c700 (LWP 3534)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068d0ee0, wctx=0x55ce06884118) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7ff9f197b700 (LWP 3537)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068d51e0, wctx=0x55ce06884170) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7ff9f117a700 (LWP 3540)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068d94e0, wctx=0x55ce068841c8) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7ff9f0979700 (LWP 3541)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068dd7e0, wctx=0x55ce06884220) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7ff9f0178700 (LWP 3543)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068e1ae0, wctx=0x55ce06884278) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7ff9ef977700 (LWP 3545)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068e73a0, wctx=0x55ce068842d0) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7ff9ef176700 (LWP 3547)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa066e4682 in slot_run (thread=0x55ce068eb3c0, wctx=0x55ce06884328) at h2_workers.c:329

2 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

3 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7ff9ee174700 (LWP 3565)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7ff9ed973700 (LWP 3566)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7ff9ed172700 (LWP 3568)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7ff9ec971700 (LWP 3569)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0

2 0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0

3 0x000055ce02ad2e71 in ap_core_input_filter ()

4 0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so

5 0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so

6 0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1

7 0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1

8 0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1

9 0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1

10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1

11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1

12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1

13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so

14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so

15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff99c07fde0) at h2_c1_io.c:486

16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff99c07fde0) at h2_c1_io.c:513

17 0x00007ffa066da08b in h2_session_process (session=0x7ff99c07fde0, async=) at h2_session.c:1865

18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8009958) at h2_c1.c:130

19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8009958) at h2_c1.c:278

20 0x000055ce02ab13c0 in ap_run_process_connection ()

21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7ff9e7fff700 (LWP 3570)):

0 0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6

1 0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0

2 0x00007ffa066d1661 in mplx_pollset_poll (on_ctx=0x7ff97401b400, on_stream_output=0x7ffa066d5b50 , on_stream_input=0x7ffa066d5ca0 , timeout=30000000, m=0x7ff97401bc20) at h2_mplx.c:1126

3 h2_mplx_c1_poll (m=0x7ff97401bc20, timeout=30000000, on_stream_input=on_stream_input@entry=0x7ffa066d5ca0 , on_stream_output=on_stream_output@entry=0x7ffa066d5b50 , on_ctx=on_ctx@entry=0x7ff97401b400) at h2_mplx.c:606

4 0x00007ffa066da200 in h2_session_process (session=0x7ff97401b400, async=) at h2_session.c:1903

5 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8015e98) at h2_c1.c:130

6 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8015e98) at h2_c1.c:278

7 0x000055ce02ab13c0 in ap_run_process_connection ()

8 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

9 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

10 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7ff9e77fe700 (LWP 3572)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7ff9e6ffd700 (LWP 3574)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7ff9e67fc700 (LWP 3576)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7ff9e5ffb700 (LWP 3578)):

0 0x00007ffa14d3fddd in poll () from /lib64/libc.so.6

1 0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so

2 0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so

3 0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so

4 0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so

5 0x000055ce02aaf558 in ap_scan_script_header_err_core_ex ()

6 0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so

7 0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so

8 0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so

9 0x000055ce02a9f550 in ap_run_handler ()

10 0x000055ce02a9fa99 in ap_invoke_handler ()

11 0x000055ce02add40c in ap_internal_redirect ()

12 0x00007ffa0c3961ec in handler_redirect () from /etc/httpd/modules/mod_rewrite.so

13 0x000055ce02a9f550 in ap_run_handler ()

14 0x000055ce02a9fa99 in ap_invoke_handler ()

15 0x000055ce02ade0ea in ap_process_async_request ()

16 0x000055ce02ade3ce in ap_process_request ()

17 0x000055ce02ada304 in ap_process_http_connection ()

18 0x000055ce02ab13c0 in ap_run_process_connection ()

19 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

20 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

21 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7ff9e57fa700 (LWP 3580)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7ff9e4ff9700 (LWP 3582)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7ff9e47f8700 (LWP 3584)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7ff9e3ff7700 (LWP 3586)):

0 0x00007ffa14d3fddd in poll () from /lib64/libc.so.6

1 0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so

2 0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so

3 0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so

4 0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so

5 0x000055ce02aaf558 in ap_scan_script_header_err_core_ex ()

6 0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so

7 0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so

8 0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so

9 0x000055ce02a9f550 in ap_run_handler ()

10 0x000055ce02a9fa99 in ap_invoke_handler ()

11 0x000055ce02ade0ea in ap_process_async_request ()

12 0x000055ce02ade3ce in ap_process_request ()

13 0x000055ce02ada304 in ap_process_http_connection ()

14 0x000055ce02ab13c0 in ap_run_process_connection ()

15 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

16 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

17 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7ff9e37f6700 (LWP 3588)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7ff9e2ff5700 (LWP 3591)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7ff9e27f4700 (LWP 3593)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7ff9e1ff3700 (LWP 3594)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7ff9e17f2700 (LWP 3596)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7ff9e0ff1700 (LWP 3598)):

0 0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6

1 0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0

2 0x00007ffa066d1661 in mplx_pollset_poll (on_ctx=0x7ff96c0140b0, on_stream_output=0x7ffa066d5b50 , on_stream_input=0x7ffa066d5ca0 , timeout=30000000, m=0x7ff96c0148d0) at h2_mplx.c:1126

3 h2_mplx_c1_poll (m=0x7ff96c0148d0, timeout=30000000, on_stream_input=on_stream_input@entry=0x7ffa066d5ca0 , on_stream_output=on_stream_output@entry=0x7ffa066d5b50 , on_ctx=on_ctx@entry=0x7ff96c0140b0) at h2_mplx.c:606

4 0x00007ffa066da200 in h2_session_process (session=0x7ff96c0140b0, async=) at h2_session.c:1903

5 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8017f68) at h2_c1.c:130

6 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8017f68) at h2_c1.c:278

7 0x000055ce02ab13c0 in ap_run_process_connection ()

8 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

9 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

10 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7ff9e07f0700 (LWP 3600)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7ff9dffef700 (LWP 3602)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7ff9df7ee700 (LWP 3604)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7ff9defed700 (LWP 3606)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0

2 0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0

3 0x000055ce02ad2e71 in ap_core_input_filter ()

4 0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so

5 0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so

6 0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1

7 0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1

8 0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1

9 0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1

10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1

11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1

12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1

13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so

14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so

15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff98c040fb0) at h2_c1_io.c:486

16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff98c040fb0) at h2_c1_io.c:513

17 0x00007ffa066da08b in h2_session_process (session=0x7ff98c040fb0, async=) at h2_session.c:1865

18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8007878) at h2_c1.c:130

19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8007878) at h2_c1.c:278

20 0x000055ce02ab13c0 in ap_run_process_connection ()

21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7ff9de7ec700 (LWP 3608)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0

2 0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0

3 0x000055ce02ad2e71 in ap_core_input_filter ()

4 0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so

5 0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so

6 0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1

7 0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1

8 0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1

9 0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1

10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1

11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1

12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1

13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so

14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so

15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9d4082110) at h2_c1_io.c:486

16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9d4082110) at h2_c1_io.c:513

17 0x00007ffa066da08b in h2_session_process (session=0x7ff9d4082110, async=) at h2_session.c:1865

18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8005798) at h2_c1.c:130

19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8005798) at h2_c1.c:278

20 0x000055ce02ab13c0 in ap_run_process_connection ()

21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7ff9ddfeb700 (LWP 3611)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x000055ce02ab789d in ap_queue_pop_something ()

2 0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7ff9dd7ea700 (LWP 3612)):

0 0x00007ffa15223bb8 in __pthread_mutex_lock_full () from /lib64/libpthread.so.0

1 0x00007ffa15459b09 in proc_mutex_pthread_acquire_ex () from /lib64/libapr-1.so.0

2 0x00007ffa096ca6c8 in listener_thread () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7ff9dcfe9700 (LWP 3659)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7ff9cffff700 (LWP 3665)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7ff9cf7fe700 (LWP 3666)):

0 0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6

1 0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0

2 0x00007ffa019da3de in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa019cfb77 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

4 0x00007ffa019d02e5 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

5 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

6 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7ff9ceffd700 (LWP 3667)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ff9ce7fc700 (LWP 3668)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7ff9cdffb700 (LWP 3669)):

0 0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

2 0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

3 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

4 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffa165d28c0 (LWP 3492)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x000055ce02ab68e7 in ap_mpm_podx_check ()

2 0x00007ffa096cb004 in child_main () from /etc/httpd/modules/mod_mpm_worker.so

3 0x00007ffa096cb48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so

4 0x00007ffa096cb534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so

5 0x00007ffa096cc0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so

6 0x000055ce02ab41ce in ap_run_mpm ()

7 0x000055ce02a9d88d in main ()

` Reverting to 2.0.3 fixes CPU load but occasionally some httpd processes hang

icing commented 2 years ago

Thanks @alexskynet for putting v2.0.6 into your grinder. Sorry, that it did nothing to improve.😢

Analyzing the backtraces now.

icing commented 2 years ago

There was one change from v2.0.3 to newer ones that involved thread creation for the h2 workers. A quick check to see if that is causing problems would be to configure your server with a fixed number of h2 workers, like in:

H2MinWorkers 25
H2MaxWorkers 25

so all workers are created at startup and no dynamic creation/desctruction of thread does happen.

Could you give this a shot?

alexskynet commented 2 years ago

unfortunately no change Still 100% CPU with your settings and 2.0.6

alexskynet commented 2 years ago

Thread 17 (Thread 0x7ff9e1ff3700 (LWP 8225)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0

2 0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0

3 0x000055ce02ad2e71 in ap_core_input_filter ()

4 0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so

5 0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so

6 0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1

7 0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1

8 0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1

9 0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1

10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1

11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1

12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1

13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so

14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so

15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9d4004c60) at h2_c1_io.c:486

16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9d4004c60) at h2_c1_io.c:513

17 0x00007ffa066da08b in h2_session_process (session=0x7ff9d4004c60, async=) at h2_session.c:1865

18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8007878) at h2_c1.c:130

19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8007878) at h2_c1.c:278

alexskynet commented 2 years ago

Thread 25 (Thread 0x7ff9e5ffb700 (LWP 8217)):

0 0x00007ffa1522c75d in read () from /lib64/libpthread.so.0

1 0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0

2 0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0

3 0x000055ce02ad2e71 in ap_core_input_filter ()

4 0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so

5 0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so

6 0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1

7 0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1

8 0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1

9 0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1

10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1

11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1

12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1

13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so

14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so

15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9d002cbd0) at h2_c1_io.c:486

16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9d002cbd0) at h2_c1_io.c:513

17 0x00007ffa066da08b in h2_session_process (session=0x7ff9d002cbd0, async=) at h2_session.c:1865

18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8005798) at h2_c1.c:130

19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8005798) at h2_c1.c:278

20 0x000055ce02ab13c0 in ap_run_process_connection ()

21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so

22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0

23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

icing commented 2 years ago

v2.0.7 released with a fix for mpm_worker setups that could (does?) result in busy loops.

Background: the v2.0.x line had improvements to return to mpm_event connection monitoring asap for efficiency. Unfortunately, the mpm_worker situation was not properly accounted for.

@nono303 I mentioned worker, but this should apply to Windows setups as well.

alexskynet commented 2 years ago

Time to compile and run and I'll let you know

alexskynet commented 2 years ago

running

alexskynet commented 2 years ago

Very first impression is good: no immediate lock so I cross my fingers

alexskynet commented 2 years ago

A giant step @icing ! It is running with no apparent problems The only thing I noticed is that httpd CPU usage raised from 6-7% to something around 10% but this is really not a problem

icing commented 2 years ago

Thanks for your patience. Happy to hear that. Let's see what the day brings.

As to performance, I made some conservative changes to bring stability. When this version proves to be stable, I can dare to tighten the screws again somewhat.

alexskynet commented 2 years ago

You're doing a great job @icing

I'm happy to have been useful to help a little bit: that's called opensource!

alexskynet commented 2 years ago

running till now ok Starting a second test server Thank you @icing great job!

alexskynet commented 2 years ago

I just want to confirm that 2.0.7 works like a charm

CodeIt has already released the mod_http2-2.0.7 rpm with the fix so all the world should be happy now :-)

It has been running for several hours on two server with absolutely no troubles

Problem solved: well done @icing ! :1st_place_medal:

icing commented 2 years ago

Thanks again @alexskynet for the help!

Closing this as fixed in v2.0.7.

nono303 commented 2 years ago

… I came after the battle (quite busy now) but many Thx @icing for your work and responsiveness I compiled and running over 2.0.7 on Windows and all work like a charm 😉 No more cpu usage against this stack

libapr-1.dll!impl_pollset_poll+0xba
mod_http2.so!mplx_pollset_poll+0x154
mod_http2.so!h2_mplx_c1_poll+0x52
mod_http2.so!h2_session_process+0xca5
mod_http2.so!h2_c1_run+0xaf
mod_http2.so!h2_c1_hook_process_connection+0x4e2
libhttpd.dll!ap_run_process_connection+0x35
libhttpd.dll!worker_main+0x3a8
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d

just maybe a little bit more cpu cycle on mod_watchdog, but not sure is related to h2

mod_watchdog.so!wd_worker+0x24f
libapr-1.dll!dummy_worker+0x43
ucrtbase.DLL!o__realloc_base+0x60
kernel32.dll!BaseThreadInitThunk+0xd
alexskynet commented 2 years ago

Sorry to reopen this bt today we had hangs on two different servers both running 2.0.7 Reverting to 2.0.3 seems to fix it We runned 2.0.7 ok for nearly 4 days before the hang The interesting part of pstack seems this at a first glance: Thread 7 (Thread 0x7fdcfb7fe700 (LWP 20589)):

0 0x00007fdd26670cd4 in h2_fifo_remove () from /etc/httpd/modules/mod_http2.so

1 0x00007fdcdc03e478 in ?? ()

2 0x00007fdd35a7167a in apr_bucket_shared_split () from /lib64/libaprutil-1.so.0

3 0x00007fdcfc0057d8 in ?? ()

4 0x0000000000003727 in ?? ()

5 0x00007fdd26882f40 in http2_module () from /etc/httpd/modules/mod_http2.so

6 0x0000000000003ff7 in ?? ()

7 0x00007fdd2666b2cc in h2_stream_recv_DATA () from /etc/httpd/modules/mod_http2.so

8 0x5613754aae48f400 in ?? ()

9 0x00007fdcdc018c80 in ?? ()

10 0x00007fdce80028d0 in ?? ()

11 0x00007fdce800b0e0 in ?? ()

12 0x00007fdce8003b60 in ?? ()

13 0x00007fdce800b080 in ?? ()

14 0x00007fdce800e0c8 in ?? ()

15 0x00007fdce800e0c0 in ?? ()

16 0x00007fdd364981a1 in nghttp2_session_pack_data () from /lib64/libnghttp2.so.14

17 0x00007fdd36498f0d in nghttp2_session_mem_send_internal () from /lib64/libnghttp2.so.14

18 0x00007fdd36499b19 in nghttp2_session_send () from /lib64/libnghttp2.so.14

19 0x00007fdd266672ec in on_stream_state_enter () from /etc/httpd/modules/mod_http2.so

20 0x00007fdd26667574 in on_stream_state_enter () from /etc/httpd/modules/mod_http2.so

21 0x00007fdcdc018c08 in ?? ()

22 0x00007fdcfb7fdb18 in ?? ()

23 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fdcf97fa700 (LWP 20593)):

0 0x00007fdd26668946 in h2_session_process () from /etc/httpd/modules/mod_http2.so

1 0x00007fdc8c072970 in ?? ()

2 0x0000000000000142 in ?? ()

3 0x00007fdcf97f9a50 in ?? ()

4 0x00007fdcd0015b20 in ?? ()

5 0xffffffffffffffff in ?? ()

6 0x0000000000000000 in ?? ()

icing commented 2 years ago

@alexskynet this strack trace does not look right. I assume there is another version of mod_http2 loaded than the v2.0.7 one. v2.0.7 never invokes h2_fifo_remove(). Assuming the APR lib symbols are correct, the apr_bucket_split is also never called in v2.0.7 (but was in versions v2.0.4 and earlier).

Could you double check?

icing commented 2 years ago

v2.0.8 released. The fixes are unrelated to this, but I added a assertion before the point where the 100% cpu loop seems to happen in reports by @alexskynet. It would be interesting to know if this triggers and if so, what is logged (at level critical) in such a case.

icing commented 2 years ago

The epic battle continues with v2.0.9:

alexskynet commented 2 years ago

Hi @icing and thank you for your hard work

Testing 2.0.9 with worker

Till now what I see in mod_status looks OK: it has been running for 20 minutes now.

I cross my fingers and wait ...

I'll let you know

icing commented 2 years ago

You tricked me before! I remain sceptical...😉

alexskynet commented 2 years ago

12 hours still running fine ...

It looks promising

alexskynet commented 2 years ago

Hi @icing did some testing for three days now and it works fine Five servers are running 2.0.9 and one is running master with no anomalies The battle is over and you won Many compliments for the very good work

icing commented 2 years ago

Hi @alexskynet! This is excellent news. We won. Thank you very much for helping on this.

In the meantime, I have added more edge test cases and made some more improvements on reliability. Will release that in some days as a v2.0.10.

alexskynet commented 2 years ago

Thanks you!

I'll start a test as soon as possible

Il 30 settembre 2022 16:59:17 CEST, Stefan Eissing @.***> ha scritto:

The epic battle continues with v2.0.9:

  • Fixed a bug where errors during reponse body handling did not lead to a proper RST_STREAM. Instead processing went into an infinite loop. Extended test cases to catch this condition.

-- Reply to this email directly or view it on GitHub: https://github.com/icing/mod_h2/issues/234#issuecomment-1263686057 You are receiving this because you were mentioned.

Message ID: @.***> SKNT Group SRLU Via Maggiate 67/a 28021 Borgomanero (NO) tel. +39 0322-836487/834765 fax +39 0322-836608 http://sknt.it

icing commented 2 years ago

With v2.0.10 just being released and Alessandro's extensive testing, I think we have solve the issues.

Many, many thanks to everyone.