*** Error in `/usr/bin/carbon-c-relay': double free or corruption (fasttop): 0x0000000002325240 ***

Rohlik commented 3 years ago

Hello guys 👋🏼 , We hit on problem with enabled native SSL in carbon-c-relay. In our case, service is running under systemd and crash multiple times during the day. If SSL part is removed from config then everything works like a charm. But we would like to use native SSL.

Logs:

Here you can see how many times carbon-c-relay starting over and over:

[2021-05-03 00:10:22] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=20487
[2021-05-03 00:34:17] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=22868
[2021-05-03 01:00:51] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=1822
[2021-05-03 01:31:52] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=5012
[2021-05-03 02:05:50] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=9356
[2021-05-03 02:08:45] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=9662
[2021-05-03 02:09:23] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=9705
[2021-05-03 02:27:01] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=11603
[2021-05-03 02:29:50] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=11903
[2021-05-03 02:44:43] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=14221
[2021-05-03 02:57:36] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=15618
[2021-05-03 03:44:38] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=21805
[2021-05-03 03:55:36] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=22854
[2021-05-03 04:05:47] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=24031
[2021-05-03 04:16:52] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=25254
[2021-05-03 04:16:55] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=25269
[2021-05-03 04:57:54] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=20156
[2021-05-03 05:02:51] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=20723
[2021-05-03 05:03:53] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=20787
[2021-05-03 05:32:26] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=31436
[2021-05-03 05:45:55] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=1198
[2021-05-03 06:41:01] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=7915
[2021-05-03 06:57:01] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=9512
[2021-05-03 07:35:38] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=13526
[2021-05-03 07:55:44] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=16237
[2021-05-03 08:02:01] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=17020
[2021-05-03 08:15:34] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=18476
[2021-05-03 08:15:50] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=18501
[2021-05-03 08:48:27] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=22519
[2021-05-03 09:55:23] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=20408
[2021-05-03 10:06:47] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=29436
[2021-05-03 10:35:53] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=32337
[2021-05-03 10:43:02] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=1579
[2021-05-03 11:01:42] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=3705
[2021-05-03 11:14:44] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=5040
[2021-05-03 11:16:58] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=5256
[2021-05-03 12:04:02] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=10939
[2021-05-03 12:21:01] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=12700
[2021-05-03 12:26:59] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=13297
[2021-05-03 12:32:28] (MSG) starting carbon-c-relay v3.7.2 (2020-07-10), pid=13867

One of the crash, copied from journactl -xe -u carbon-c-relay:

May 03 12:21:01 carbon-dev carbon-c-relay[10939]: *** Error in `/usr/bin/carbon-c-relay': double free or corruption (fasttop): 0x000000000115f240 ***
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: *** Error in `/usr/bin/carbon-c-relay': double free or corruption (fasttop): 0x000000000115f240 ***
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: ======= Backtrace: =========
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libc.so.6(+0x81329)[0x7f35400d7329]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7f3540db790d]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libcrypto.so.10(X509_get_pubkey_parameters+0x120)[0x7f3540eadda0]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libcrypto.so.10(X509_verify_cert+0xb0)[0x7f3540eade60]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libssl.so.10(ssl_add_cert_chain+0x147)[0x7f35411f44a7]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libssl.so.10(ssl3_output_cert_chain+0x35)[0x7f35411d9125]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libssl.so.10(ssl3_send_server_certificate+0x3d)[0x7f35411cafad]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libssl.so.10(ssl3_accept+0x425)[0x7f35411cbbc5]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libssl.so.10(ssl23_accept+0x1c8)[0x7f35411db598]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libssl.so.10(ssl23_read+0x32)[0x7f35411dc642]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /usr/bin/carbon-c-relay[0x407dbd]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /usr/bin/carbon-c-relay[0x409783]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libpthread.so.0(+0x7ea5)[0x7f354042bea5]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: /lib64/libc.so.6(clone+0x6d)[0x7f35401549fd]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: ======= Memory map: ========
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 00400000-00429000 r-xp 00000000 fd:01 25848                              /usr/bin/carbon-c-relay
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 00628000-00629000 r--p 00028000 fd:01 25848                              /usr/bin/carbon-c-relay
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 00629000-0062a000 rw-p 00029000 fd:01 25848                              /usr/bin/carbon-c-relay
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 010ef000-0117e000 rw-p 00000000 00:00 0                                  [heap]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f34ec000000-7f34ed8af000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f34ed8af000-7f34f0000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f34f4000000-7f34fc000000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f34fc000000-7f34fc021000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f34fc021000-7f3500000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3500000000-7f3500029000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3500029000-7f3504000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3504000000-7f3504021000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3504021000-7f3508000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3508000000-7f3508021000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3508021000-7f350c000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f350c000000-7f350c297000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f350c297000-7f3510000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3510000000-7f3510021000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3510021000-7f3514000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3514000000-7f351425d000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f351425d000-7f3518000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f351b7ff000-7f351b800000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f351b800000-7f351c000000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f351c000000-7f351c021000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f351c021000-7f3520000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35235e9000-7f35235fe000 r-xp 00000000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35235fe000-7f35237fd000 ---p 00015000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35237fd000-7f35237fe000 r--p 00014000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35237fe000-7f35237ff000 rw-p 00015000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35237ff000-7f3523800000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3523800000-7f3524000000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3524000000-7f3527ffe000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3527ffe000-7f3528000000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3528215000-7f3528216000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3528216000-7f3528a16000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3528a16000-7f3528a17000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3528a17000-7f3529217000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3529217000-7f3529218000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3529218000-7f3529a18000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3529a18000-7f3529a19000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3529a19000-7f352a219000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352a219000-7f352a21a000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352a21a000-7f352aa1a000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352aa1a000-7f352aa1b000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352aa1b000-7f352b21b000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352b21b000-7f352b21c000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352b21c000-7f352ba1c000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352ba1c000-7f352ba1d000 ---p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f352ba1d000-7f353e818000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353e818000-7f353e81e000 r-xp 00000000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353e81e000-7f353ea1d000 ---p 00006000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ea1d000-7f353ea1e000 r--p 00005000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ea1e000-7f353ea1f000 rw-p 00006000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ea1f000-7f353ea2b000 r-xp 00000000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ea2b000-7f353ec2a000 ---p 0000c000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ec2a000-7f353ec2b000 r--p 0000b000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ec2b000-7f353ec2c000 rw-p 0000c000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ec2c000-7f353ec32000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ec32000-7f353ec56000 r-xp 00000000 fd:01 4182                       /usr/lib64/libselinux.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ec56000-7f353ee55000 ---p 00024000 fd:01 4182                       /usr/lib64/libselinux.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ee55000-7f353ee56000 r--p 00023000 fd:01 4182                       /usr/lib64/libselinux.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ee56000-7f353ee57000 rw-p 00024000 fd:01 4182                       /usr/lib64/libselinux.so.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ee57000-7f353ee59000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ee59000-7f353ee6f000 r-xp 00000000 fd:01 3819                       /usr/lib64/libresolv-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353ee6f000-7f353f06f000 ---p 00016000 fd:01 3819                       /usr/lib64/libresolv-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f06f000-7f353f070000 r--p 00016000 fd:01 3819                       /usr/lib64/libresolv-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f070000-7f353f071000 rw-p 00017000 fd:01 3819                       /usr/lib64/libresolv-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f071000-7f353f073000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f073000-7f353f076000 r-xp 00000000 fd:01 4236                       /usr/lib64/libkeyutils.so.1.5
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f076000-7f353f275000 ---p 00003000 fd:01 4236                       /usr/lib64/libkeyutils.so.1.5
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f275000-7f353f276000 r--p 00002000 fd:01 4236                       /usr/lib64/libkeyutils.so.1.5
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f276000-7f353f277000 rw-p 00003000 fd:01 4236                       /usr/lib64/libkeyutils.so.1.5
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f277000-7f353f285000 r-xp 00000000 fd:01 9304                       /usr/lib64/libkrb5support.so.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f285000-7f353f485000 ---p 0000e000 fd:01 9304                       /usr/lib64/libkrb5support.so.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f485000-7f353f486000 r--p 0000e000 fd:01 9304                       /usr/lib64/libkrb5support.so.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f486000-7f353f487000 rw-p 0000f000 fd:01 9304                       /usr/lib64/libkrb5support.so.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f487000-7f353f4e7000 r-xp 00000000 fd:01 4168                       /usr/lib64/libpcre.so.1.2.0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f4e7000-7f353f6e7000 ---p 00060000 fd:01 4168                       /usr/lib64/libpcre.so.1.2.0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f6e7000-7f353f6e8000 r--p 00060000 fd:01 4168                       /usr/lib64/libpcre.so.1.2.0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f6e8000-7f353f6e9000 rw-p 00061000 fd:01 4168                       /usr/lib64/libpcre.so.1.2.0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f6e9000-7f353f71a000 r-xp 00000000 fd:01 9296                       /usr/lib64/libk5crypto.so.3.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f71a000-7f353f919000 ---p 00031000 fd:01 9296                       /usr/lib64/libk5crypto.so.3.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f919000-7f353f91b000 r--p 00030000 fd:01 9296                       /usr/lib64/libk5crypto.so.3.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f91b000-7f353f91c000 rw-p 00032000 fd:01 9296                       /usr/lib64/libk5crypto.so.3.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f91c000-7f353f91f000 r-xp 00000000 fd:01 4187                       /usr/lib64/libcom_err.so.2.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353f91f000-7f353fb1e000 ---p 00003000 fd:01 4187                       /usr/lib64/libcom_err.so.2.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fb1e000-7f353fb1f000 r--p 00002000 fd:01 4187                       /usr/lib64/libcom_err.so.2.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fb1f000-7f353fb20000 rw-p 00003000 fd:01 4187                       /usr/lib64/libcom_err.so.2.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fb20000-7f353fbf9000 r-xp 00000000 fd:01 9302                       /usr/lib64/libkrb5.so.3.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fbf9000-7f353fdf8000 ---p 000d9000 fd:01 9302                       /usr/lib64/libkrb5.so.3.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fdf8000-7f353fe06000 r--p 000d8000 fd:01 9302                       /usr/lib64/libkrb5.so.3.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fe06000-7f353fe09000 rw-p 000e6000 fd:01 9302                       /usr/lib64/libkrb5.so.3.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fe09000-7f353fe53000 r-xp 00000000 fd:01 9292                       /usr/lib64/libgssapi_krb5.so.2.2
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f353fe53000-7f3540053000 ---p 0004a000 fd:01 9292                       /usr/lib64/libgssapi_krb5.so.2.2
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540053000-7f3540054000 r--p 0004a000 fd:01 9292                       /usr/lib64/libgssapi_krb5.so.2.2
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540054000-7f3540056000 rw-p 0004b000 fd:01 9292                       /usr/lib64/libgssapi_krb5.so.2.2
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540056000-7f354021a000 r-xp 00000000 fd:01 3791                       /usr/lib64/libc-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354021a000-7f3540419000 ---p 001c4000 fd:01 3791                       /usr/lib64/libc-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540419000-7f354041d000 r--p 001c3000 fd:01 3791                       /usr/lib64/libc-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354041d000-7f354041f000 rw-p 001c7000 fd:01 3791                       /usr/lib64/libc-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354041f000-7f3540424000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540424000-7f354043b000 r-xp 00000000 fd:01 3817                       /usr/lib64/libpthread-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354043b000-7f354063a000 ---p 00017000 fd:01 3817                       /usr/lib64/libpthread-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354063a000-7f354063b000 r--p 00016000 fd:01 3817                       /usr/lib64/libpthread-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354063b000-7f354063c000 rw-p 00017000 fd:01 3817                       /usr/lib64/libpthread-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354063c000-7f3540640000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540640000-7f3540741000 r-xp 00000000 fd:01 3799                       /usr/lib64/libm-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540741000-7f3540940000 ---p 00101000 fd:01 3799                       /usr/lib64/libm-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540940000-7f3540941000 r--p 00100000 fd:01 3799                       /usr/lib64/libm-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540941000-7f3540942000 rw-p 00101000 fd:01 3799                       /usr/lib64/libm-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540942000-7f3540944000 r-xp 00000000 fd:01 3797                       /usr/lib64/libdl-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540944000-7f3540b44000 ---p 00002000 fd:01 3797                       /usr/lib64/libdl-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540b44000-7f3540b45000 r--p 00002000 fd:01 3797                       /usr/lib64/libdl-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540b45000-7f3540b46000 rw-p 00003000 fd:01 3797                       /usr/lib64/libdl-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540b46000-7f3540b48000 r-xp 00000000 fd:01 4176                       /usr/lib64/libpcreposix.so.0.0.1
May 03 12:21:01 carbon-dev systemd[1]: carbon-c-relay.service: main process exited, code=killed, status=6/ABRT
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540b48000-7f3540d47000 ---p 00002000 fd:01 4176                       /usr/lib64/libpcreposix.so.0.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540d47000-7f3540d48000 r--p 00001000 fd:01 4176                       /usr/lib64/libpcreposix.so.0.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540d48000-7f3540d49000 rw-p 00002000 fd:01 4176                       /usr/lib64/libpcreposix.so.0.0.1
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540d49000-7f3540f7f000 r-xp 00000000 fd:01 9312                       /usr/lib64/libcrypto.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3540f7f000-7f354117f000 ---p 00236000 fd:01 9312                       /usr/lib64/libcrypto.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354117f000-7f354119b000 r--p 00236000 fd:01 9312                       /usr/lib64/libcrypto.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354119b000-7f35411a8000 rw-p 00252000 fd:01 9312                       /usr/lib64/libcrypto.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35411a8000-7f35411ac000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f35411ac000-7f3541213000 r-xp 00000000 fd:01 9314                       /usr/lib64/libssl.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541213000-7f3541413000 ---p 00067000 fd:01 9314                       /usr/lib64/libssl.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541413000-7f3541417000 r--p 00067000 fd:01 9314                       /usr/lib64/libssl.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541417000-7f354141e000 rw-p 0006b000 fd:01 9314                       /usr/lib64/libssl.so.1.0.2k
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354141e000-7f354142c000 r-xp 00000000 fd:01 4729                       /usr/lib64/liblz4.so.1.8.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354142c000-7f354162b000 ---p 0000e000 fd:01 4729                       /usr/lib64/liblz4.so.1.8.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354162b000-7f354162c000 r--p 0000d000 fd:01 4729                       /usr/lib64/liblz4.so.1.8.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354162c000-7f354162d000 rw-p 0000e000 fd:01 4729                       /usr/lib64/liblz4.so.1.8.3
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f354162d000-7f3541642000 r-xp 00000000 fd:01 4185                       /usr/lib64/libz.so.1.2.7
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541642000-7f3541841000 ---p 00015000 fd:01 4185                       /usr/lib64/libz.so.1.2.7
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541841000-7f3541842000 r--p 00014000 fd:01 4185                       /usr/lib64/libz.so.1.2.7
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541842000-7f3541843000 rw-p 00015000 fd:01 4185                       /usr/lib64/libz.so.1.2.7
May 03 12:21:01 carbon-dev systemd[1]: Unit carbon-c-relay.service entered failed state.
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541843000-7f3541865000 r-xp 00000000 fd:01 3784                       /usr/lib64/ld-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541a11000-7f3541a5c000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541a61000-7f3541a64000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541a64000-7f3541a65000 r--p 00021000 fd:01 3784                       /usr/lib64/ld-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541a65000-7f3541a66000 rw-p 00022000 fd:01 3784                       /usr/lib64/ld-2.17.so
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7f3541a66000-7f3541a67000 rw-p 00000000 00:00 0
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7ffc8569e000-7ffc856bf000 rw-p 00000000 00:00 0                          [stack]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: 7ffc85796000-7ffc85798000 r-xp 00000000 00:00 0                          [vdso]
May 03 12:21:01 carbon-dev carbon-c-relay[10939]: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
May 03 12:21:01 carbon-dev systemd[1]: carbon-c-relay.service failed.
May 03 12:21:01 carbon-dev systemd[1]: carbon-c-relay.service holdoff time over, scheduling restart.
May 03 12:21:01 carbon-dev systemd[1]: Stopped Enhanced C implementation of Carbon relay, aggregator and rewriter.

Info:

OS: Centos 7 Kernel: 3.10.0-1160.24.1.el7.x86_64 carbon-c-relay: 3.7.2-1 Related packages:

openssl-libs-1.0.2k-21.el7_9.x86_64
openssl-1.0.2k-21.el7_9.x86_64

Config:

listen type linemode transport plain ssl /etc/ssl/certs/carbon-c-relay-all.pem 0.0.0.0:2443 proto tcp;
listen type linemode 0.0.0.0:2003 proto tcp;

cluster to_carbon_cache
  carbon_ch
    cache-dev01:2003=0
    cache-dev02:2013=1
  ;

statistics
  submit every 60 seconds
  reset counters after interval
  ;

match  *
  send to to_carbon_cache
  stop
  ;

grobian commented 3 years ago

Ok, can you describe the traffic? do you have a lot of connections that stay alive, or are they all connect+disconnects? I'd really like a stacktrace, but I guess I'll have to reproduce the case for that.

Rohlik commented 3 years ago

Here is a data flow: (nodes with collectd metrics) -> (local carbon-c-relay) -> (remote carbon-dev server using SSL endpoint)

Metrics data are scraped every 60sec via collectd and we are collecting metrics from 88 servers.

grobian commented 3 years ago

Ok, some questions:

did you compile this carbon-c-relay yourself or did it come from some distro repo?
if you compiled it yourself, would you be willing to re-compile it with debug symbols, and
submit a stacktrace of the crashing binary with debug symbols the last thing might give me a hint at what stage in the relay the crash occurs

Rohlik commented 3 years ago

Hello @grobian I created backtrace via gdb during error:

*** Error in `/usr/bin/carbon-c-relay': double free or corruption (fasttop): 0x000000000069a240 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7ffff666f329]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7ffff734f90d]
/lib64/libcrypto.so.10(X509_get_pubkey_parameters+0x120)[0x7ffff7445da0]
/lib64/libcrypto.so.10(X509_verify_cert+0xb0)[0x7ffff7445e60]
/lib64/libssl.so.10(ssl_add_cert_chain+0x147)[0x7ffff778c4a7]
/lib64/libssl.so.10(ssl3_output_cert_chain+0x35)[0x7ffff7771125]
/lib64/libssl.so.10(ssl3_send_server_certificate+0x3d)[0x7ffff7762fad]
/lib64/libssl.so.10(ssl3_accept+0x425)[0x7ffff7763bc5]
/lib64/libssl.so.10(ssl23_accept+0x1c8)[0x7ffff7773598]
/lib64/libssl.so.10(ssl23_read+0x32)[0x7ffff7774642]
/usr/bin/carbon-c-relay[0x407dbd]
/usr/bin/carbon-c-relay[0x409783]
/lib64/libpthread.so.0(+0x7ea5)[0x7ffff69c3ea5]
/lib64/libc.so.6(clone+0x6d)[0x7ffff66ec9fd]
======= Memory map: ========
00400000-00429000 r-xp 00000000 fd:01 25848                              /usr/bin/carbon-c-relay
00628000-00629000 r--p 00028000 fd:01 25848                              /usr/bin/carbon-c-relay
00629000-0062a000 rw-p 00029000 fd:01 25848                              /usr/bin/carbon-c-relay
0062a000-006b9000 rw-p 00000000 00:00 0                                  [heap]
7fff90000000-7fff9349a000 rw-p 00000000 00:00 0 
7fff9349a000-7fff94000000 ---p 00000000 00:00 0 
7fff98000000-7fff993d2000 rw-p 00000000 00:00 0 
7fff993d2000-7fff9c000000 ---p 00000000 00:00 0 
7fff9c000000-7fff9c029000 rw-p 00000000 00:00 0 
7fff9c029000-7fffa0000000 ---p 00000000 00:00 0 
7fffa0000000-7fffa0021000 rw-p 00000000 00:00 0 
7fffa0021000-7fffa4000000 ---p 00000000 00:00 0 
7fffa8000000-7fffa8021000 rw-p 00000000 00:00 0 
7fffa8021000-7fffac000000 ---p 00000000 00:00 0 
7fffadfff000-7fffae000000 ---p 00000000 00:00 0 
7fffae000000-7fffb0000000 rw-p 00000000 00:00 0 
7fffb0000000-7fffb0021000 rw-p 00000000 00:00 0 
7fffb0021000-7fffb4000000 ---p 00000000 00:00 0 
7fffb5fff000-7fffb6000000 ---p 00000000 00:00 0 
7fffb6000000-7fffb8000000 rw-p 00000000 00:00 0 
7fffb8000000-7fffb8021000 rw-p 00000000 00:00 0 
7fffb8021000-7fffbc000000 ---p 00000000 00:00 0 
7fffbc000000-7fffbc021000 rw-p 00000000 00:00 0 
7fffbc021000-7fffc0000000 ---p 00000000 00:00 0 
7fffc1fff000-7fffc2000000 ---p 00000000 00:00 0 
7fffc2000000-7fffc4000000 rw-p 00000000 00:00 0 
7fffc4000000-7fffc8000000 rw-p 00000000 00:00 0 
7fffc9ffd000-7fffc9ffe000 ---p 00000000 00:00 0 
7fffc9ffe000-7fffcbffe000 rw-p 00000000 00:00 0 
7fffcbffe000-7fffcbfff000 ---p 00000000 00:00 0 
7fffcbfff000-7fffcdfff000 rw-p 00000000 00:00 0 
7fffcdfff000-7fffce000000 ---p 00000000 00:00 0 
7fffce000000-7fffd0000000 rw-p 00000000 00:00 0 
7fffd0000000-7fffd4000000 rw-p 00000000 00:00 0 
7fffd5fff000-7fffd6000000 ---p 00000000 00:00 0 
7fffd6000000-7fffd8000000 rw-p 00000000 00:00 0 
7fffd8000000-7fffdbffd000 rw-p 00000000 00:00 0 
7fffdbffd000-7fffdc000000 ---p 00000000 00:00 0 
7fffdc59c000-7fffdc5b1000 r-xp 00000000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc5b1000-7fffdc7b0000 ---p 00015000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc7b0000-7fffdc7b1000 r--p 00014000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc7b1000-7fffdc7b2000 rw-p 00015000 fd:01 586                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fffdc7b2000-7fffdc7b3000 ---p 00000000 00:00 0 
7fffdc7b3000-7fffde7b3000 rw-p 00000000 00:00 0 
7fffde7b3000-7fffde7b4000 ---p 00000000 00:00 0 
7fffde7b4000-7fffe07b4000 rw-p 00000000 00:00 0 
7fffe07b4000-7fffe07b5000 ---p 00000000 00:00 0 
7fffe07b5000-7ffff4db0000 rw-p 00000000 00:00 0 
7ffff4db0000-7ffff4db6000 r-xp 00000000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
7ffff4db6000-7ffff4fb5000 ---p 00006000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
7ffff4fb5000-7ffff4fb6000 r--p 00005000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
7ffff4fb6000-7ffff4fb7000 rw-p 00006000 fd:01 3807                       /usr/lib64/libnss_dns-2.17.so
7ffff4fb7000-7ffff4fc3000 r-xp 00000000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
7ffff4fc3000-7ffff51c2000 ---p 0000c000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
7ffff51c2000-7ffff51c3000 r--p 0000b000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
7ffff51c3000-7ffff51c4000 rw-p 0000c000 fd:01 3809                       /usr/lib64/libnss_files-2.17.so
7ffff51c4000-7ffff51ca000 rw-p 00000000 00:00 0 
7ffff51ca000-7ffff51ee000 r-xp 00000000 fd:01 4182                       /usr/lib64/libselinux.so.1
7ffff51ee000-7ffff53ed000 ---p 00024000 fd:01 4182                       /usr/lib64/libselinux.so.1
7ffff53ed000-7ffff53ee000 r--p 00023000 fd:01 4182                       /usr/lib64/libselinux.so.1
7ffff53ee000-7ffff53ef000 rw-p 00024000 fd:01 4182                       /usr/lib64/libselinux.so.1
7ffff53ef000-7ffff53f1000 rw-p 0
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffde7b2700 (LWP 8030)]
0x00007ffff66243d7 in raise () from /lib64/libc.so.6

Output file: backtrace.log.

If you need any additional stuff, feel free to let me know. Thank you.

Rohlik commented 3 years ago

@grobian I have an update for you. After migrating our carbon-dev instance with carbon-c-relay to Centos 8, problem is gone, no restart/error in past 5 days so far.

Info about updated instance: OS: Centos 8 Kernel: 4.18.0-240.22.1.el8_3.x86_64 carbon-c-relay: 3.7.2-1

Related packages: openssl-1.1.1g-15.el8_3.x86_64 openssl-libs-1.1.1g-15.el8_3.x86_64

So for us is problem solved but feel free to dig in if you are interested what cause problem on Centos 7. Anyway, thank you for your time.

grobian commented 3 years ago

Hey sorry for not responding, your backtrace is interesting, and I meant to look at this, but life is a bit hectic around here at the moment. Glad to hear the problem is gone (or just less?) on a newer version of CentOS. I'll keep this one open to investigate if I can find anything though. It may be openssl-version related, but it usually means the calls are not entirely according to specs. Thanks.

Rohlik commented 3 years ago

@grobian We are runnig server with Centos 8 just a couple of days (5) and there is no problem so far. So I would say that problem is gone completely. 👍🏼

grobian commented 2 years ago

this issue might actually be related to concurrency

grobian / carbon-c-relay