Closed Mark11122 closed 4 months ago
Any chance to use the latest 2.7 version?
The 2.7 is EOL now. It is a better choice to switch to 2.8 instead.
We'll try to use version 2.8, but it's a longer process, and now this issue can't be fixed anytime soon
OpenSSL 1.0.2 was not good in terms of threading and here it crashed in lock contention inside OpenSSL. I guess that if you're running on 1.0.2 it's because you have a low load, so the best thing to do with this version of OpenSSL is to significantly reduce the number of threads (e.g. 4) using "nbthread 4" for example in the global section. Of course you need to test carefully because you never know what the low limit will be for your workload without testing. In any case your process is not behaving well, it died because some operations couldn't make any progress for two full seconds, that's an eternity at this level! If you can rebuild with at least 1.1.1 (and an up-to-date haproxy version), you'll get a better experience.
Thank you, I'm going to test the 2.8.9 version, and if the test is normal, I'm going to replace the 2.7.6 version
Detailed Description of the Problem
May 14 23:59:26 haproxy: Thread 14 is about to kill the process. May 14 23:59:26 haproxy: Thread 1 : id=0x7f65e4a6c2c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/1 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5258378644773 now=5258665955274 diff=287310501 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 2 : id=0x7f65dfeae700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/2 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5486068836817 now=5486355652298 diff=286815481 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 3 : id=0x7f65de19c700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/3 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5778666787847 now=5778876699136 diff=209911289 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 4 : id=0x7f65dd814700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/4 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5454785404853 now=5454980545759 diff=195140906 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 5 : id=0x7f65dd013700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/5 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=6221042503982 now=6221327641201 diff=285137219 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: >Thread 6 : id=0x7f65dc812700 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=1 rqsz=4 May 14 23:59:26 haproxy: 1/6 stuck=1 prof=0 harmless=0 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=6083621974346 now=6085511981745 diff=1890007399 May 14 23:59:26 haproxy: curr_task=0x7f65ac157ab0 (tasklet) calls=2 May 14 23:59:26 haproxy: fct=0x434d20(ssl_sock_io_cb) ctx=0x7f65c05b9bd0 May 14 23:59:26 haproxy: call trace(16): May 14 23:59:26 haproxy: | 0x58ea9b [85 c0 89 c5 75 17 48 81]: ha_dump_backtrace+0x2b/0x2d9 May 14 23:59:26 haproxy: | 0x58f558 [64 41 8b 75 00 8d 46 01]: debug_handler+0x118/0x252 May 14 23:59:26 haproxy: | 0x7f65e3bf8100 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0xf100 May 14 23:59:26 haproxy: | 0x7f65e3bf772b [48 3d 00 f0 ff ff 48 89]: libpthread:send+0x7b/0xa9 May 14 23:59:26 haproxy: | 0x5ddbc7 [48 83 f8 00 0f 8e 97 00]: main+0x1b14f7 May 14 23:59:26 haproxy: | 0x431efd [85 c0 41 89 c4 75 19 48]: main+0x582d May 14 23:59:26 haproxy: | 0x7f65e362f9ab [85 c0 7e 07 48 63 d0 48]: libcrypto:BIO_write+0x6b/0x100 May 14 23:59:26 haproxy: | 0x7f65e3632a81 [48 89 ef 41 89 c6 e8 34]: libcrypto:+0x11ea81 May 14 23:59:26 haproxy: | 0x7f65e39968db [85 c0 0f 8e bd 01 00 00]: libssl:ssl3_accept+0x13b/0xe13 May 14 23:59:26 haproxy: | 0x4350cf [83 f8 01 4c 8b 4c 24 08]: ssl_sock_io_cb+0x3af/0xb49 May 14 23:59:26 haproxy: Thread 7 : id=0x7f65d4f27700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/7 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5827196411899 now=5827382801104 diff=186389205 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 8 : id=0x7f65d4726700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/8 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5945594787238 now=5945803756400 diff=208969162 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 9 : id=0x7f65d3f25700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/9 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5465812987950 now=5466001892812 diff=188904862 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 10: id=0x7f65d359d700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/10 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5609236266996 now=5609502567179 diff=266300183 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 11: id=0x7f65d2d9c700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/11 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5998406833643 now=5998698758896 diff=291925253 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 12: id=0x7f65d259b700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/12 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5460791068272 now=5461076825814 diff=285757542 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 13: id=0x7f65d1d9a700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/13 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5257951167579 now=5258103161843 diff=151994264 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: *>Thread 14: id=0x7f65d1599700 act=1 glob=1 wq=1 rq=0 tl=1 tlsz=0 rqsz=3 May 14 23:59:26 haproxy: 1/14 stuck=1 prof=0 harmless=0 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5170878374862 now=5174130648758 diff=3252273896 May 14 23:59:26 haproxy: curr_task=0x7f65b836a110 (tasklet) calls=393 May 14 23:59:26 haproxy: fct=0x573780(sc_conn_io_cb) ctx=0x7f65a81709e0 May 14 23:59:26 haproxy: strm=0x7f65a82bb780,10084b src=52.97.27.93 fe=fe_ex2016 be=be_ex2016_eas dst=ex_80_12 May 14 23:59:26 haproxy: txn=0x7f65a820c9b0,40000 txn.req=MSG_DONE,d txn.rsp=MSG_DATA,d May 14 23:59:26 haproxy: rqf=48c40080 rqa=8000 rpf=80070202 rpa=4000000 May 14 23:59:26 haproxy: scf=0x7f65a814a0b0,EST,0 scb=0x7f65a81709e0,EST,91 May 14 23:59:26 haproxy: af=(nil),0 sab=(nil),0 May 14 23:59:26 haproxy: cof=0x7f65cc686900,80000300:H1(0x7f65b415ebf0)/SSL(0x7f65d8199270)/tcpv4(1319) May 14 23:59:26 haproxy: cob=0x7f659c462b90,300:H1(0x7f65cc3a6530)/SSL(0x7f65cc58d300)/tcpv4(1458) May 14 23:59:26 haproxy: call trace(27): May 14 23:59:26 haproxy: | 0x7f65e3bf8100 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0xf100 May 14 23:59:26 haproxy: | 0x7f65e3bf772b [48 3d 00 f0 ff ff 48 89]: libpthread:send+0x7b/0xa9 May 14 23:59:26 haproxy: | 0x5ddbc7 [48 83 f8 00 0f 8e 97 00]: main+0x1b14f7 May 14 23:59:26 haproxy: | 0x431efd [85 c0 41 89 c4 75 19 48]: main+0x582d May 14 23:59:26 haproxy: | 0x7f65e362f9ab [85 c0 7e 07 48 63 d0 48]: libcrypto:BIO_write+0x6b/0x100 May 14 23:59:26 haproxy: | 0x7f65e39a1722 [8b 8b 1c 01 00 00 39 c8]: libssl:ssl3_write_pending+0x72/0x14b May 14 23:59:26 haproxy: | 0x7f65e39a1e16 [85 c0 7e 36 44 39 f8 74]: libssl:ssl3_write_bytes+0xd6/0x71a May 14 23:59:26 haproxy: | 0x432ed9 [41 8b 56 04 f7 c2 00 00]: main+0x6809 May 14 23:59:26 haproxy: | 0x49c85d [48 85 c0 74 59 44 8b 0d]: main+0x7018d May 14 23:59:26 haproxy: | 0x4a589b [85 c0 74 2c 49 8b 06 f7]: main+0x791cb May 14 23:59:26 haproxy: | 0x570b03 [85 c0 0f 8e e7 fe ff ff]: main+0x144433 May 14 23:59:26 haproxy: | 0x5728a0 [48 8b 73 20 0f b6 4b 01]: main+0x1461d0 May 14 23:59:26 haproxy: | 0x571cf1 [49 8b 44 24 30 49 8b 4c]: main+0x145621 May 14 23:59:26 haproxy: | 0x5724cf [48 8b 7b 28 e8 48 b0 f4]: main+0x145dff May 14 23:59:26 haproxy: | 0x573806 [eb a6 0f 1f 84 00 00 00]: sc_conn_io_cb+0x86/0x88 May 14 23:59:26 haproxy: Thread 15: id=0x7f65d0d98700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/15 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5753946219460 now=5754215037897 diff=268818437 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:26 haproxy: Thread 16: id=0x7f65cbfff700 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0 May 14 23:59:26 haproxy: 1/16 stuck=0 prof=0 harmless=1 wantrdv=0 May 14 23:59:26 haproxy: cpu_ns: poll=5362084364853 now=5362379469875 diff=295105022 May 14 23:59:26 haproxy: curr_task=0 May 14 23:59:27 haproxy: [ALERT] (27217) : Current worker (11379) exited with code 134 (Aborted) May 14 23:59:27 haproxy: [ALERT] (27217) : exit-on-failure: killing every processes with SIGTERM May 14 23:59:27 haproxy: [WARNING] (27217) : All workers exited. Exiting... (134) May 14 23:59:27 Keepalived_vrrp[12876]: Quorum lost for tracked process track_haproxy May 14 23:59:27 Keepalived_vrrp[12876]: (VI_101) Changing effective priority from 120 to 100 May 14 23:59:27 systemd: haproxy.service: main process exited, code=exited, status=134/n/a May 14 23:59:27 systemd: Unit haproxy.service entered failed state. May 14 23:59:27 systemd: haproxy.service failed. May 14 23:59:27 abrt-server: Executable '/usr/local/haproxy/sbin/haproxy' doesn't belong to any package and ProcessUnpackaged is set to 'no' May 14 23:59:27 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2024-05-14-23:59:26-11379' exited with 1 May 14 23:59:27 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2024-05-14-23:59:26-11379'
Expected Behavior
OpenSSL 1.0.2k-fips 26 Jan 2017
HAProxy version 2.7.6-4dadaaa 2023/03/28 - https://haproxy.org/ Status: stable branch - will stop receiving fixes around Q1 2024. Known bugs: http://www.haproxy.org/bugs/bugs-2.7.6.html Running on: Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64
Steps to Reproduce the Behavior
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
No response
What is your configuration?
Output of
haproxy -vv
Last Outputs and Backtraces
No response
Additional Information
No response