search

haproxytech / spoa-mirror

Mirror HTTP requests using the HAProxy SPOP
GNU Lesser General Public License v2.1
39 stars 16 forks source link

spoa mirror crashes after multiple segfaults #21

Open gdriban opened 3 years ago

gdriban commented 3 years ago

Over time, spoa-mirror crashes and also causes crash of forwarding haproxy.

Using dmesg you can see repeated segfaults from spoa mirror. Example: [10196349.094441] sm/wrk: 6[23039]: segfault at 0 ip 000000000040350f sp 00007fcab955eba0 error 4 in spoa-mirror[400000+10000]

Running version 1.2.12 of spoa-mirror. ./spoa-mirror -V spoa-mirror v1.2.12 [build 2415] by Miroslav Zagorac mzagorac@haproxy.com, Jul 19 2020 Had been experiencing crashes of haproxy and stopping spoa-mirror seems to have solved the issue.

OS is Oracle Linux 7 stable v7.8.0

This issue was original reported as a piggyback on https://github.com/haproxytech/spoa-mirror/issues/16. Opening new issue. Was requested by @zaga00 to run debug version and get a core file. May not be able to do that because spoa-mirror is only running on production servers. If we can get that done will post core file when we have it.

Thanks.

zaga00 commented 3 years ago

Hello @gdriban,

thank you for reporting the bug. I couldn't reproduce the bug so in the next few days I'll take a closer look at what it's all about, now unfortunately I don't have time for that.

gdriban commented 3 years ago

It is an issue that occurs over time and probably only with actual traffic going through the mirror. You'd have to leave it run for several days at minimum. Will let you know of any additional info we can provide when we have it.

gdriban commented 3 years ago

@zaga00 followed instructions for building spoa-mirror in the README. Before I run the binary on one of my production servers, I just want to make certain that performing the build this way will add debug symbols to the binary so that we get a core file on segfault: ./configure --enable-debug make all Thank you.

zaga00 commented 3 years ago

Hello @gdriban,

for the debugging process it is most important not to delete symbols (debug information) from the executable (with the 'strip' command). If you also use the '--enable-debug' option when configuring the source before compiling, then the '-O0' optimization level will be used when compiling the source (instead of '-O2' which is used if '--enable-debug' option is not used). Of course, the 'xx' option also allows the use of the '--debug' option when starting the program, as well as a detailed printout of the functions used with parameters within the program itself which can be written to a log file (with the '--logfile' option). In short, however you configure the source, it is only important that you do not strip symbols from the program.