search

haproxytech / spoa-mirror

Mirror HTTP requests using the HAProxy SPOP
GNU Lesser General Public License v2.1
39 stars 16 forks source link

repeatable segfault in spoa-mirror, takes only two HTTP requests #16

Open mmelo-yottaa opened 4 years ago

mmelo-yottaa commented 4 years ago

I'm seeing segfault in spoa-mirror when using it to receive HTTP traffic from HAProxy. I'm running HA-Proxy version 2.1.4 2020/04/02 with spoa-mirror v1.2.9 [build 2409] on CentOS 7.6.1810 with 3.10.0-514.16.1.el7.x86_64 kernel. The segfault is repeatable, happening on the second HTTP request I send to HAProxy.

this is backtrace (its repeatable):

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff071b700 (LWP 7472)] 0x000000000040526f in mir_ptr_free (data=0x7ffff0714be0) at spoa-message.c:499 499 list_for_each_entry_safe(hdr, hdr_back, (*data)->hdrs, list) (gdb) bt

0 0x000000000040526f in mir_ptr_free (data=0x7ffff0714be0) at spoa-message.c:499

1 0x0000000000405009 in spoa_msg_mirror (frame=0x7fffe000aff0, buf=0x7ffff0714c80, end=0x7fffe000b3cf "") at spoa-message.c:456

2 0x00000000004063d1 in process_frame_cb (loop=0x7fffe00008c0, ev=0x7fffe000b028, revents=256) at spoa.c:348

3 0x00007ffff7942587 in ev_invoke_pending () from /lib64/libev.so.4

4 0x00007ffff794578d in ev_run () from /lib64/libev.so.4

5 0x0000000000411f39 in worker_thread (data=0x62dd60) at worker.c:300

6 0x00007ffff7729ea5 in start_thread () from /lib64/libpthread.so.0

7 0x00007ffff74528dd in clone () from /lib64/libc.so.6

mmelo-yottaa commented 4 years ago

2.1.4.zip

zipfile includes: haproxy and mirror config files, the HTTP request causing the SEGV, debug tracing showing the SEGV (line 63).

thank you

zaga00 commented 4 years ago

Hello @mmelo-yottaa,

please check if the latest version of the program solves your problem.

Best regards, Miroslav Zagorac

gdriban commented 3 years ago

Running version 1.2.12 of spoa-mirror, which I believe is beyond the suggested latest version in comment from Aug 17 that was to have solved the issue.

./spoa-mirror -V spoa-mirror v1.2.12 [build 2415] by Miroslav Zagorac mzagorac@haproxy.com, Jul 19 2020`

These messages continually appear using dmesg [10196349.094441] sm/wrk: 6[23039]: segfault at 0 ip 000000000040350f sp 00007fcab955eba0 error 4 in spoa-mirror[400000+10000]

Had been experiencing crashes of haproxy and stopping spoa-mirror seems to have solved the issue.

OS is Oracle Linux 7 stable v7.8.0

gdriban commented 3 years ago

This is ongoing issue. Could somebody please have a look at my prior comment and advise. Thank you.

zaga00 commented 3 years ago

Hello @gdriban,

does your bug report have anything to do with @mmelo-yottaa's bug report (except that you reopened that bug report)? If there is not, i.e. does not refer to debug logs that he uploaded, then your bug reporting does not mean too much to me because I have too little data related to the behavior of the program (except that the program crashes).

gdriban commented 3 years ago

it has something to do with it in that spoa-mirror is segfaulting. I can open a new issue if you want

gdriban commented 3 years ago

If you want another ticket, please advise what information you'd like included. Thanks

zaga00 commented 3 years ago

Hello @gdriban,

thank you for reporting the bug.

The bug that occurs can (but does not have to) have the same cause as the one reported by user @mmelo-yottaa. But, we don't know that and that's why it's better to report the bug as a new issue, and we'll close this one.

As for the data that would help to find and fix the bug, it would be good to compile the program in debug mode and run it by setting the debug output to a file. Then, the program that crashes usually leaves a core file that you can load into gdb and find the exact place where the program crashed.