Closed Rohlik closed 4 months ago
Ok, can you describe the traffic? do you have a lot of connections that stay alive, or are they all connect+disconnects? I'd really like a stacktrace, but I guess I'll have to reproduce the case for that.
Here is a data flow:
(nodes with collectd metrics) -> (local carbon-c-relay) -> (remote carbon-dev
server using SSL endpoint)
Metrics data are scraped every 60sec via collectd and we are collecting metrics from 88 servers.
Ok, some questions:
Hello @grobian
I created backtrace via gdb
during error:
*** Error in `/usr/bin/carbon-c-relay': double free or corruption (fasttop): 0x000000000069a240 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7ffff666f329]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7ffff734f90d]
/lib64/libcrypto.so.10(X509_get_pubkey_parameters+0x120)[0x7ffff7445da0]
/lib64/libcrypto.so.10(X509_verify_cert+0xb0)[0x7ffff7445e60]
/lib64/libssl.so.10(ssl_add_cert_chain+0x147)[0x7ffff778c4a7]
/lib64/libssl.so.10(ssl3_output_cert_chain+0x35)[0x7ffff7771125]
/lib64/libssl.so.10(ssl3_send_server_certificate+0x3d)[0x7ffff7762fad]
/lib64/libssl.so.10(ssl3_accept+0x425)[0x7ffff7763bc5]
/lib64/libssl.so.10(ssl23_accept+0x1c8)[0x7ffff7773598]
/lib64/libssl.so.10(ssl23_read+0x32)[0x7ffff7774642]
/usr/bin/carbon-c-relay[0x407dbd]
/usr/bin/carbon-c-relay[0x409783]
/lib64/libpthread.so.0(+0x7ea5)[0x7ffff69c3ea5]
/lib64/libc.so.6(clone+0x6d)[0x7ffff66ec9fd]
======= Memory map: ========
00400000-00429000 r-xp 00000000 fd:01 25848 /usr/bin/carbon-c-relay
00628000-00629000 r--p 00028000 fd:01 25848 /usr/bin/carbon-c-relay
00629000-0062a000 rw-p 00029000 fd:01 25848 /usr/bin/carbon-c-relay
0062a000-006b9000 rw-p 00000000 00:00 0 [heap]
7fff90000000-7fff9349a000 rw-p 00000000 00:00 0
7fff9349a000-7fff94000000 ---p 00000000 00:00 0
7fff98000000-7fff993d2000 rw-p 00000000 00:00 0
7fff993d2000-7fff9c000000 ---p 00000000 00:00 0
7fff9c000000-7fff9c029000 rw-p 00000000 00:00 0
7fff9c029000-7fffa0000000 ---p 00000000 00:00 0
7fffa0000000-7fffa0021000 rw-p 00000000 00:00 0
7fffa0021000-7fffa4000000 ---p 00000000 00:00 0
7fffa8000000-7fffa8021000 rw-p 00000000 00:00 0
7fffa8021000-7fffac000000 ---p 00000000 00:00 0
7fffadfff000-7fffae000000 ---p 00000000 00:00 0
7fffae000000-7fffb0000000 rw-p 00000000 00:00 0
7fffb0000000-7fffb0021000 rw-p 00000000 00:00 0
7fffb0021000-7fffb4000000 ---p 00000000 00:00 0
7fffb5fff000-7fffb6000000 ---p 00000000 00:00 0
7fffb6000000-7fffb8000000 rw-p 00000000 00:00 0
7fffb8000000-7fffb8021000 rw-p 00000000 00:00 0
7fffb8021000-7fffbc000000 ---p 00000000 00:00 0
7fffbc000000-7fffbc021000 rw-p 00000000 00:00 0
7fffbc021000-7fffc0000000 ---p 00000000 00:00 0
7fffc1fff000-7fffc2000000 ---p 00000000 00:00 0
7fffc2000000-7fffc4000000 rw-p 00000000 00:00 0
7fffc4000000-7fffc8000000 rw-p 00000000 00:00 0
7fffc9ffd000-7fffc9ffe000 ---p 00000000 00:00 0
7fffc9ffe000-7fffcbffe000 rw-p 00000000 00:00 0
7fffcbffe000-7fffcbfff000 ---p 00000000 00:00 0
7fffcbfff000-7fffcdfff000 rw-p 00000000 00:00 0
7fffcdfff000-7fffce000000 ---p 00000000 00:00 0
7fffce000000-7fffd0000000 rw-p 00000000 00:00 0
7fffd0000000-7fffd4000000 rw-p 00000000 00:00 0
7fffd5fff000-7fffd6000000 ---p 00000000 00:00 0
7fffd6000000-7fffd8000000 rw-p 00000000 00:00 0
7fffd8000000-7fffdbffd000 rw-p 00000000 00:00 0
7fffdbffd000-7fffdc000000 ---p 00000000 00:00 0
7fffdc59c000-7fffdc5b1000 r-xp 00000000 fd:01 586 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc5b1000-7fffdc7b0000 ---p 00015000 fd:01 586 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc7b0000-7fffdc7b1000 r--p 00014000 fd:01 586 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc7b1000-7fffdc7b2000 rw-p 00015000 fd:01 586 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc7b2000-7fffdc7b3000 ---p 00000000 00:00 0
7fffdc7b3000-7fffde7b3000 rw-p 00000000 00:00 0
7fffde7b3000-7fffde7b4000 ---p 00000000 00:00 0
7fffde7b4000-7fffe07b4000 rw-p 00000000 00:00 0
7fffe07b4000-7fffe07b5000 ---p 00000000 00:00 0
7fffe07b5000-7ffff4db0000 rw-p 00000000 00:00 0
7ffff4db0000-7ffff4db6000 r-xp 00000000 fd:01 3807 /usr/lib64/libnss_dns-2.17.so
7ffff4db6000-7ffff4fb5000 ---p 00006000 fd:01 3807 /usr/lib64/libnss_dns-2.17.so
7ffff4fb5000-7ffff4fb6000 r--p 00005000 fd:01 3807 /usr/lib64/libnss_dns-2.17.so
7ffff4fb6000-7ffff4fb7000 rw-p 00006000 fd:01 3807 /usr/lib64/libnss_dns-2.17.so
7ffff4fb7000-7ffff4fc3000 r-xp 00000000 fd:01 3809 /usr/lib64/libnss_files-2.17.so
7ffff4fc3000-7ffff51c2000 ---p 0000c000 fd:01 3809 /usr/lib64/libnss_files-2.17.so
7ffff51c2000-7ffff51c3000 r--p 0000b000 fd:01 3809 /usr/lib64/libnss_files-2.17.so
7ffff51c3000-7ffff51c4000 rw-p 0000c000 fd:01 3809 /usr/lib64/libnss_files-2.17.so
7ffff51c4000-7ffff51ca000 rw-p 00000000 00:00 0
7ffff51ca000-7ffff51ee000 r-xp 00000000 fd:01 4182 /usr/lib64/libselinux.so.1
7ffff51ee000-7ffff53ed000 ---p 00024000 fd:01 4182 /usr/lib64/libselinux.so.1
7ffff53ed000-7ffff53ee000 r--p 00023000 fd:01 4182 /usr/lib64/libselinux.so.1
7ffff53ee000-7ffff53ef000 rw-p 00024000 fd:01 4182 /usr/lib64/libselinux.so.1
7ffff53ef000-7ffff53f1000 rw-p 0
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffde7b2700 (LWP 8030)]
0x00007ffff66243d7 in raise () from /lib64/libc.so.6
Output file: backtrace.log.
If you need any additional stuff, feel free to let me know. Thank you.
@grobian I have an update for you.
After migrating our carbon-dev
instance with carbon-c-relay to Centos 8, problem is gone, no restart/error in past 5 days so far.
Info about updated instance: OS: Centos 8 Kernel: 4.18.0-240.22.1.el8_3.x86_64 carbon-c-relay: 3.7.2-1
Related packages: openssl-1.1.1g-15.el8_3.x86_64 openssl-libs-1.1.1g-15.el8_3.x86_64
So for us is problem solved but feel free to dig in if you are interested what cause problem on Centos 7. Anyway, thank you for your time.
Hey sorry for not responding, your backtrace is interesting, and I meant to look at this, but life is a bit hectic around here at the moment. Glad to hear the problem is gone (or just less?) on a newer version of CentOS. I'll keep this one open to investigate if I can find anything though. It may be openssl-version related, but it usually means the calls are not entirely according to specs. Thanks.
@grobian We are runnig server with Centos 8 just a couple of days (5) and there is no problem so far. So I would say that problem is gone completely. 👍🏼
this issue might actually be related to concurrency
Hello guys 👋🏼 , We hit on problem with enabled native SSL in carbon-c-relay. In our case, service is running under systemd and crash multiple times during the day. If SSL part is removed from config then everything works like a charm. But we would like to use native SSL.
Logs:
Here you can see how many times carbon-c-relay starting over and over:
One of the crash, copied from
journactl -xe -u carbon-c-relay
:Info:
OS: Centos 7 Kernel: 3.10.0-1160.24.1.el7.x86_64 carbon-c-relay: 3.7.2-1 Related packages:
Config: