search

haproxy / haproxy

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
https://git.haproxy.org/
Other
5.05k stars 804 forks source link

SEGFAULT in libcrypto from quic_tls_aes_decrypt #2606

Open Tristan971 opened 5 months ago

Tristan971 commented 5 months ago

Detailed Description of the Problem

Well that's a new one. Must be somewhere between 7217a9e and 983513d

Expected Behavior

No crash

Steps to Reproduce the Behavior

?

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-curves X25519:prime256v1
ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tlsv12 no-tls-tickets

(all have tfo and allow-0rtt)

Output of haproxy -vv

HAProxy version 3.1-dev1-983513d+mangadex-af185e5 2024-06-16T10:51+00:00 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 6.8.4-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-3 (2024-05-02T11:55Z) x86_64
Build options :
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -ggdb3 -gdwarf-4 -fwrapv -DMAX_SESS_STKCTR=5
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_STATIC_PCRE2=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE +STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1w+quic-mangadex-af185e5 16 Jun 2024
Running on OpenSSL version : OpenSSL 1.1.1w+quic-mangadex-af185e5 16 Jun 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 18.1.6 (++20240518023138+1118c2e05e67-1~exp1~20240518143226.133)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
    [BWLIM] bwlim-in
    [BWLIM] bwlim-out
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace

Last Outputs and Backtraces

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000063eb7ec3c644 in CRYPTO_ctr128_encrypt_ctr32 ()
[Current thread is 1 (Thread 0x749b0a044ac0 (LWP 12642))]
(gdb) t a a bt full

Thread 8 (Thread 0x749af1800640 (LWP 12647)):
#0  0x0000749b0a3bce2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000063eb7e929957 in _do_poll (p=<optimized out>, exp=496551, wake=0) at src/ev_epoll.c:232
        timeout = 14
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 14
        status = <optimized out>
        count = <optimized out>
#2  0x000063eb7ead8c8f in run_poll_loop () at src/haproxy.c:3147
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -283770880
#3  0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x749af0400640 (LWP 12649)):
#0  0x0000749b0a3bce2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000063eb7e929957 in _do_poll (p=<optimized out>, exp=496541, wake=0) at src/ev_epoll.c:232
        timeout = 4
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 4
        status = <optimized out>
        count = <optimized out>
#2  0x000063eb7ead8c8f in run_poll_loop () at src/haproxy.c:3147
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -296353792
#3  0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x749af0e00640 (LWP 12648)):
#0  0x000063eb7eb8d422 in __eb64_insert (root=0x749aea5340a0, new=new@entry=0x749aea68cce0) at include/import/eb64tree.h:319
        side = 1
        troot = 0x749aece60f21
        root_right = 0x1
        newkey = 2899
        old = 0x749aece60f20
        old_node_bit = 9
        new_left = <optimized out>
        new_rght = <optimized out>
        new_leaf = <optimized out>
        old_leaf = <optimized out>
        new_left = <optimized out>
        new_rght = <optimized out>
        new_leaf = <optimized out>
        old_node = <optimized out>
        ret = <optimized out>
#1  eb64_insert (root=<optimized out>, new=new@entry=0x749aea68cce0) at src/eb64tree.c:27
No locals.
#2  0x000063eb7e970e55 in qc_send_ppkts (buf=buf@entry=0x749aecd203b8, ctx=<optimized out>) at src/quic_tx.c:379
        cc = 0x749aecd20650
        tmpbuf = {
          size = 1252,
          area = 0x749ae3a19d04 "\\\223e\030\342c\023!\231+A~\322<P\213\f>\326\316\023#\217\252\027z\224\003;6L\205\372b\241\357\213\306\307\303V\220k\n\230\350\277JI\325k\341\067\071\352\247\360\020\\\341\304\304\215\310a\240\350\207\273T6y\342\251\350\330\263\245\367\343\201\362\352\363\365\312\023\360&\331Mu_\246-.R\271\210\214\234B\n\\\371\220\311\311\f\315e\202\307\032a\274\214[\253\327\273D\266\277Tx{\033\066\r\376\374\355\205\365g\372I#\371\030\060B\327\325\265\334\026\262+\t\361\302^Z\363T\346\321\a}\300$42\022Mm\201\306V\364\220\303Z\272\325\354x\335a\325\315.?\305\030\336\250\035W\033\334\066A\333\331\343}\334\235\332\326\313\363\326_=\"\356\357\364\232\225\313\253\367s\277\261\333\265\033s\f\372\003\336\203\360I\243d\003/E\210\030mg\217\247\264\325b\244\257^\270 \212k\364\026.\327v\341\360\314\212\033\032t.\343Bg\207\315M\311l\260h\330\347\320b[\271<\327D\314\\`\022V\304\023\024\351*\247\276/\204\364\255f\031\340\205y\034\306\067\311\300\361\365\066\215=E\263\307\063ci:\371\335\024+\003\026\061\276\231\341[\377\301\302V\332\324I\376\272h:\326\301\360\373\207kS\025\t\226so\255\301\213\360\216\062vz\306\067\371\362\216\227\334\343\374\271?\252\031\331C\204\353\023\373H\221\211\336\371PQ\230\230\313.\177\273\260\201\341\343\200\070Mr\213x\353Tp\314V'\"\236\217\031SX\274\213\215\344\206\343\301\bv\235 BV,\347@\220\336}\344\360\227\375\234\062\357\241iO\305S\321s\365\266q\032\353\243\305Y\345\257\213\237X\220!)\227G\233dv\257\246t\275\373_X\205\"rG\240o\363\232p\031Z\260_\372\027\373\061&\302\250\267\237\220;\242\067\225\257\344\334#\240Y"...,
          data = 1252,
          head = 0
        }
        pkt = <optimized out>
        pos = <optimized out>
        first_pkt = <optimized out>
        time_sent = 496538
        next_pkt = 0x0
        dglen = <optimized out>
        ret = 0
        skip_sendto = <optimized out>
        qc = 0x749aecd20000
#3  0x000063eb7e9716bf in qc_send (qc=qc@entry=0x749aecd20000, old_data=old_data@entry=0, send_list=send_list@entry=0x749af0df3568) at src/quic_tx.c:721
        ret = 32508
        status = 0
        buf = 0x749aecd203b8
        qel = <optimized out>
        tmp_qel = <optimized out>
#4  0x000063eb7e971368 in qc_send_mux (qc=0x749aecd20000, frms=frms@entry=0x749af0df3730) at src/quic_tx.c:448
        send_list = {
          n = 0x749aeedf5c10,
          p = 0x749aeedf5c10
        }
        ret = <optimized out>
#5  0x000063eb7e962921 in qcc_send_frames (qcc=0x749aeedf79c0, frms=0x749af0df3730) at src/mux_quic.c:1927
No locals.
#6  0x000063eb7e960eea in qcc_io_send (qcc=qcc@entry=0x749aeedf79c0) at src/mux_quic.c:2208
        _ = {
          func = 0x63eb7ee0d693 "qcc_io_send",
          file = 0x63eb7ee0c036 "src/mux_quic.c",
          line = 2255,
          what = 3 '\003',
          arg8 = 0 '\000',
          arg32 = 0
        }
        frms = {
          n = 0x749aea681940,
          p = 0x749aea681b80
        }
        qcs_failed = {
          n = 0x749af0df3720,
          p = 0x749af0df3720
        }
        first_qcs = <optimized out>
        window_conn = <optimized out>
        total = 131072
        qcs = <optimized out>
        qcs_tmp = <optimized out>
        ret = <optimized out>
        resent = <optimized out>
#7  0x000063eb7e96008e in qcc_io_cb (t=<optimized out>, ctx=ctx@entry=0x749aeedf79c0, status=<optimized out>) at src/mux_quic.c:2543
        qcc = <optimized out>
#8  0x000063eb7eb3ec92 in run_tasks_from_lists (budgets=budgets@entry=0x749af0df39c0) at src/task.c:596
        _ = {
          func = 0x63eb7ee6f439 "run_tasks_from_lists",
          file = 0x63eb7ee6f44e "src/task.c",
          line = 657,
          what = 6 '\006',
          arg8 = 0 '\000',
          arg32 = 0
        }
        tl_queues = 0x63eb7f2451d0 <ha_thread_ctx+3280>
        budget_mask = 15 '\017'
        profile_entry = 0x0
        done = 11
        queue = 2
        t = 0x749aea6377c0
        process = 0x63eb7e960060 <qcc_io_cb>
        ctx = 0x749aeedf79c0
        state = 1
#9  0x000063eb7eb3f63a in process_runnable_tasks () at src/task.c:876
        max = {0, 0, 81, 0}
        tt = 0x63eb7f245100 <ha_thread_ctx+3072>
        default_weights = {64, 48, 16, 1}
        heavy_queued = 1
        max_processed = 93
        max_total = <optimized out>
        queue = 4
        budget = 0
        grq = <optimized out>
        lrq = <optimized out>
        gpicked = <optimized out>
        lpicked = <optimized out>
        t = <optimized out>
        tmp_list = <optimized out>
#10 0x000063eb7ead8cc4 in run_poll_loop () at src/haproxy.c:3075
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = <optimized out>
#11 0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#12 0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#13 0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x749af3600640 (LWP 12644)):
#0  0x0000749b0a3bce2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000063eb7e929957 in _do_poll (p=<optimized out>, exp=496551, wake=0) at src/ev_epoll.c:232
        timeout = 14
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 14
        status = <optimized out>
        count = <optimized out>
#2  0x000063eb7ead8c8f in run_poll_loop () at src/haproxy.c:3147
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -275382272
#3  0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x749af2c00640 (LWP 12645)):
#0  0x0000749b0a3bce2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000063eb7e929957 in _do_poll (p=<optimized out>, exp=496551, wake=0) at src/ev_epoll.c:232
        timeout = 13
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 13
        status = <optimized out>
        count = <optimized out>
#2  0x000063eb7ead8c8f in run_poll_loop () at src/haproxy.c:3147
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -279576576
#3  0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x749af2200640 (LWP 12646)):
#0  0x0000749b0a3bce2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000063eb7e929957 in _do_poll (p=<optimized out>, exp=496551, wake=0) at src/ev_epoll.c:232
        timeout = 17
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 17
        status = <optimized out>
        count = <optimized out>
#2  0x000063eb7ead8c8f in run_poll_loop () at src/haproxy.c:3147
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -292159488
#3  0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x749b09600640 (LWP 12643)):
#0  0x0000749b0a3bce2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000063eb7e929957 in _do_poll (p=<optimized out>, exp=496540, wake=0) at src/ev_epoll.c:232
        timeout = 2
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 2
        status = <optimized out>
        count = <optimized out>
#2  0x000063eb7ead8c8f in run_poll_loop () at src/haproxy.c:3147
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = 144714240
#3  0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x0000749b0a32bac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x0000749b0a3bd850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x749b0a044ac0 (LWP 12642)):
#0  0x000063eb7ec3c644 in CRYPTO_ctr128_encrypt_ctr32 ()
No symbol table info available.
#1  0x000063eb7ec1f286 in aes_ctr_cipher ()
No symbol table info available.
#2  0x000063eb7ec2c8cb in evp_EncryptDecryptUpdate ()
No symbol table info available.
#3  0x000063eb7ec2c102 in EVP_DecryptUpdate ()
No symbol table info available.
#4  0x000063eb7e991776 in quic_tls_aes_decrypt (out=out@entry=0x7fff1131f878 "", in=in@entry=0x749ae42919d6 "\373f\306\215uP\242\302f\307\375I\337\327\367\252L\354\313\375 \206E#\020\021\221\031d6\345\372(Y\317\305\377\214\232\326\377\260\061\177\371\325\t\361|\217\036\276\341\370\064\224", inlen=inlen@entry=5, ctx=0x749af3dd5000) at src/quic_tls.c:665
        ret = 0
#5  0x000063eb7e95984f in qc_do_rm_hp (qc=qc@entry=0x749aeb6a5800, pkt=pkt@entry=0x749aeb7f3b80, tls_ctx=<optimized out>, largest_pn=1, pn=0x749ae42919d2 "\266\035\066\021\373f\306\215uP\242\302f\307\375I\337\327\367\252L\354\313\375 \206E#\020\021\221\031d6\345\372(Y\317\305\377\214\232\326\377\260\061\177\371\325\t\361|\217\036\276\341\370\064\224", byte0=byte0@entry=0x749ae42919c0 "\317") at src/quic_rx.c:94
        mask = "\000\000\000\000"
        truncated_pn = 0
        ret = 0
        sample = 0x749af3dd5038 ""
        pnlen = <optimized out>
        i = <optimized out>
        packet_number = <optimized out>
#6  0x000063eb7e957999 in qc_try_rm_hp (qc=0x749aeb6a5800, pkt=0x749aeb7f3b80, beg=0x749ae42919c0 "\317", el=<optimized out>) at src/quic_rx.c:1418
        tls_ctx = 0x1
        ret = 0
        pn = 0x749af3dd5028 "\373f\306\215uP\242\302f\307\375I\337\327\367\252"
        tel = <optimized out>
        qel = 0x749af5a15280
        __x = <optimized out>
        __x = <optimized out>
        qc_qel = <optimized out>
        qc_pktns = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        tls_ctx = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        msg = <optimized out>
#7  qc_rx_pkt_handle (qc=0x749aeb6a5800, pkt=0x749aeb7f3b80, dgram=0x749aeb7ca880, beg=0x749ae42919c0 "\317", tasklist_head=<optimized out>) at src/quic_rx.c:2064
        qv = 0x63eb7ef9f248 <quic_versions+104>
        qel = 0x0
        b_cspace = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        _msg = <optimized out>
        _msg_len = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#8  quic_dgram_parse (dgram=dgram@entry=0x749aeb7ca880, from_qc=from_qc@entry=0x749aeb6a5800, li=0x749b0885aa00) at src/quic_rx.c:2197
        _ = {
          func = 0x63eb7ee0a41e "quic_dgram_parse",
          file = 0x63eb7ee0a45e "src/quic_rx.c",
          line = 2164,
          what = 3 '\003',
          arg8 = 0 '\000',
          arg32 = 0
        }
        qc = 0x749aeb6a5800
        tasklist_head = 0x0
        pos = 0x749ae42919c0 "\317"
        pkt = 0x749aeb7f3b80
        end = <optimized out>
#9  0x000063eb7e98ad12 in qc_rcv_buf (qc=qc@entry=0x749aeb6a5800) at src/quic_sock.c:880
        saddr = {
          ss_family = 2,
          __ss_padding = "A\026\227\207\n\201", '\000' <repeats 111 times>,
          __ss_align = 0
        }
        daddr = {
          ss_family = 2,
          __ss_padding = "\001\273-\201\345\001", '\000' <repeats 111 times>,
          __ss_align = 0
        }
        new_dgram = 0x749aeb7ca880
        buf = {
          size = 32768,
          area = 0x749ae42919c0 "\317",
          data = 1250,
          head = 0
        }
        ret = 1250
        l = 0x749b0885aa00
        params = <optimized out>
        max_sz = 2048
        dgram_buf = 0x749ae42919c0 "\317"
#10 0x000063eb7e97c68d in quic_conn_io_cb (t=t@entry=0x749aeb44c140, context=context@entry=0x749aeb6a5800, state=<optimized out>) at src/quic_conn.c:771
        send_list = {
          n = 0x7fff113202d0,
          p = 0x7fff113202d0
        }
        st = 4
        qc = <optimized out>
        qel = <optimized out>
        tl = <optimized out>
#11 0x000063eb7eb3ec92 in run_tasks_from_lists (budgets=budgets@entry=0x7fff11320460) at src/task.c:596
        _ = {
          func = 0x63eb7ee6f439 "run_tasks_from_lists",
          file = 0x63eb7ee6f44e "src/task.c",
          line = 657,
          what = 6 '\006',
          arg8 = 0 '\000',
          arg32 = 0
        }
        tl_queues = 0x63eb7f2445d0 <ha_thread_ctx+208>
        budget_mask = 15 '\017'
        profile_entry = 0x0
        done = 0
        queue = 2
        t = 0x749aeb44c140
        process = 0x63eb7e97c5d0 <quic_conn_io_cb>
        ctx = 0x749aeb6a5800
        state = 288487212
#12 0x000063eb7eb3f63a in process_runnable_tasks () at src/task.c:876
        max = {0, 0, 92, 0}
        tt = 0x63eb7f244500 <ha_thread_ctx>
        default_weights = {64, 48, 16, 1}
        heavy_queued = 1
        max_processed = 93
        max_total = <optimized out>
        queue = 4
        budget = 0
        grq = <optimized out>
        lrq = <optimized out>
        gpicked = <optimized out>
        lpicked = <optimized out>
        t = <optimized out>
        tmp_list = <optimized out>
#13 0x000063eb7ead8cc4 in run_poll_loop () at src/haproxy.c:3075
        _ = {
          func = 0x63eb7ee51d6b "run_poll_loop",
          file = 0x63eb7ee51d79 "src/haproxy.c",
          line = 3106,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = <optimized out>
#14 0x000063eb7eadd1e0 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3289
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            __wseq = {
              __value64 = 41,
              __value32 = {
                __low = 41,
                __high = 0
              }
            },
            __g1_start = {
              __value64 = 27,
              __value32 = {
                __low = 27,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = ")\000\000\000\000\000\000\000\033", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 41
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#15 0x000063eb7eadbe5a in main (argc=<optimized out>, argv=0x7fff113209a8) at src/haproxy.c:3991
        limit = {
          rlim_cur = 18446744073709551615,
          rlim_max = 18446744073709551615
        }
        pidfd = <optimized out>
        retry = <optimized out>
        err = <optimized out>
        intovf = <optimized out>
(gdb)

Additional Information

No response

haproxyFred commented 5 months ago

Weird variables values: tls_ctx = 0x1 at line 1418 (src/quic_rx.c) after qc_select_tls_ctx() returned.

Tristan971 commented 5 months ago

I have a bunch more of these, and they happen in all LBs with this build, so it has to be something relatively likely to happen and not related to some local network fault or similar.

here's another one (same build) ```plain Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000061685c98b644 in CRYPTO_ctr128_encrypt_ctr32 () [Current thread is 1 (Thread 0x70410d200640 (LWP 20105))] (gdb) t a a bt full Thread 2 (Thread 0x70410e8b2ac0 (LWP 20104)): #0 0x000070410e125e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x000061685c678957 in _do_poll (p=, exp=3326423, wake=0) at src/ev_epoll.c:232 timeout = 3 updt_idx = fd = old_fd = wait_time = 3 status = count = #2 0x000061685c827c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x61685cba0d6b "run_poll_loop", file = 0x61685cba0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -104733696 #3 0x000061685c82c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 3, __value32 = { __low = 3, __high = 0 } }, __g1_start = { __value64 = 1, __value32 = { __low = 1, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 0, __g_signals = {0, 0} }, __size = "\003\000\000\000\000\000\000\000\001", '\000' , "\004", '\000' , __align = 3 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x000061685c82ae5a in main (argc=, argv=0x7ffe0774d408) at src/haproxy.c:3991 limit = { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 } pidfd = retry = err = intovf = Thread 1 (Thread 0x70410d200640 (LWP 20105)): #0 0x000061685c98b644 in CRYPTO_ctr128_encrypt_ctr32 () No symbol table info available. #1 0x000061685c96e286 in aes_ctr_cipher () No symbol table info available. #2 0x000061685c97b8cb in evp_EncryptDecryptUpdate () No symbol table info available. #3 0x000061685c97b102 in EVP_DecryptUpdate () No symbol table info available. #4 0x000061685c6e0776 in quic_tls_aes_decrypt (out=out@entry=0x70410d1f2e68 "", in=in@entry=0x7040f36cf58d "\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357", inlen=inlen@entry=5, ctx=0x7040f67841c0) at src/quic_tls.c:665 ret = 0 #5 0x000061685c6a884f in qc_do_rm_hp (qc=qc@entry=0x70410c7d7000, pkt=pkt@entry=0x7040f786af80, tls_ctx=, largest_pn=23, pn=0x7040f36cf589 "-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357", byte0=byte0@entry=0x7040f36cf580 "C\242\265F\256c \234K-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357") at src/quic_rx.c:94 mask = "\000\000\000\000" truncated_pn = 0 ret = 0 sample = 0x7040f67841f8 "" pnlen = i = packet_number = #6 0x000061685c6a6999 in qc_try_rm_hp (qc=0x70410c7d7000, pkt=0x7040f786af80, beg=0x7040f36cf580 "C\242\265F\256c \234K-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357", el=) at src/quic_rx.c:1418 tls_ctx = 0x1 ret = 0 pn = 0x7040f67841e8 "\005LM\244\334\254]\353\200\366\006\320\070\036&\036" tel = qel = 0x70410c7c7100 __x = __x = qc_qel = qc_pktns = __x = __x = tls_ctx = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = msg = #7 qc_rx_pkt_handle (qc=0x70410c7d7000, pkt=0x7040f786af80, dgram=0x7040f7453180, beg=0x7040f36cf580 "C\242\265F\256c \234K-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357", tasklist_head=) at src/quic_rx.c:2064 qv = 0x0 qel = 0x0 b_cspace = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = _msg = _msg_len = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = #8 quic_dgram_parse (dgram=dgram@entry=0x7040f7453180, from_qc=from_qc@entry=0x70410c7d7000, li=0x7040f9e1b500) at src/quic_rx.c:2197 _ = { func = 0x61685cb5941e "quic_dgram_parse", file = 0x61685cb5945e "src/quic_rx.c", line = 2164, what = 3 '\003', arg8 = 0 '\000', arg32 = 0 } qc = 0x70410c7d7000 tasklist_head = 0x0 pos = 0x7040f36cf580 "C\242\265F\256c \234K-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357" pkt = 0x7040f786af80 end = #9 0x000061685c6d9d12 in qc_rcv_buf (qc=qc@entry=0x70410c7d7000) at src/quic_sock.c:880 saddr = { ss_family = 2, __ss_padding = "\v\177\275\\\372)", '\000' , __ss_align = 0 } daddr = { ss_family = 2, __ss_padding = "\001\273-\201\345\001", '\000' , __ss_align = 0 } new_dgram = 0x7040f7453180 buf = { size = 32768, area = 0x7040f36cf580 "C\242\265F\256c \234K-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357", data = 33, head = 0 } ret = 33 l = 0x7040f9e1b500 params = max_sz = 2048 dgram_buf = 0x7040f36cf580 "C\242\265F\256c \234K-qY\021\005LM\244\334\254]\353\200\366\006\320\070\036&\036\337\067\023\360Bc\214\067\210H\020!T\311\033\001,\001\bs\371\313\241\355eZ\267\365\006\200\303\243\232\017\343S\222\304\200\035Bz\017\357" #10 0x000061685c6ca400 in quic_conn_app_io_cb (t=t@entry=0x7040f70b5e10, context=context@entry=0x70410c7d7000, state=) at src/quic_conn.c:559 send_list = { n = 0x70410d1f3870, p = 0x70410d1f3870 } qc = #11 0x000061685c88dc92 in run_tasks_from_lists (budgets=budgets@entry=0x70410d1f39c0) at src/task.c:596 _ = { func = 0x61685cbbe439 "run_tasks_from_lists", file = 0x61685cbbe44e "src/task.c", line = 657, what = 6 '\006', arg8 = 0 '\000', arg32 = 0 } tl_queues = 0x61685cf937d0 budget_mask = 15 '\017' profile_entry = 0x0 done = 0 queue = 2 t = 0x7040f70b5e10 process = 0x61685c6ca3b0 ctx = 0x70410c7d7000 state = 220146972 #12 0x000061685c88e63a in process_runnable_tasks () at src/task.c:876 max = {0, 0, 185, 0} tt = 0x61685cf93700 default_weights = {64, 48, 16, 1} heavy_queued = 1 max_processed = 186 max_total = queue = 4 budget = 0 grq = lrq = gpicked = lpicked = t = tmp_list = #13 0x000061685c827cc4 in run_poll_loop () at src/haproxy.c:3075 _ = { func = 0x61685cba0d6b "run_poll_loop", file = 0x61685cba0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = #14 0x000061685c82c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 3, __value32 = { __low = 3, __high = 0 } }, __g1_start = { __value64 = 1, __value32 = { __low = 1, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 0, __g_signals = {0, 0} }, __size = "\003\000\000\000\000\000\000\000\001", '\000' , "\004", '\000' , __align = 3 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #15 0x000070410e094ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #16 0x000070410e126850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. ```
a-denoyelle commented 5 months ago

Most variables are optimized out, so it's difficult to draw any conclusion for this backtrace for the moment. If the crash happens often enough, could it be possible to activate quic traces please ?

haproxyFred commented 5 months ago

According to the two dumps, this is same bug but at different steps. First dump is during the handshake, and the second one after the handshake. Cannot tell very much more as everything is optimized. Some traces should help.

Tristan971 commented 5 months ago

It is every 2-3 hours or so, depending on the loadbalancer, so I can certainly get traces somewhere with relatively low traffic (or if all the recent trace perf. improvements fix the CPU use on my end).

Either way, I'll get traces for you sometime tomorrow 👍

Tristan971 commented 5 months ago

And there you go, I have 2x core + traces, which happened with only 2 or 3 seconds of interval (after 30 minutes or so of uptime).

Interestingly, the stacktraces are quite different (the second looks like a bug inside the traces code), but maybe they're related.

traces-1.gz

core 1 - in quic_tls_aes_decrypt ```plain Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/haproxy -sf 45329 -x sockpair@4 -Ws -f /etc/haproxy/haproxy.cfg -p /r'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00006077dd1a3644 in CRYPTO_ctr128_encrypt_ctr32 () [Current thread is 1 (Thread 0x7b1cef400640 (LWP 45548))] (gdb) t a a bt full Thread 8 (Thread 0x7b1d05e00640 (LWP 45546)): #0 0x00006077dd025f56 in htx_get_blk_type (blk=0x7b1cdeb75130) at include/haproxy/htx.h:160 No locals. #1 __http_find_header (htx=htx@entry=0x7b1cdeb6d1c0, pattern=pattern@entry=0x7b1d05df34b0, ctx=ctx@entry=0x7b1d05df7eb0, flags=flags@entry=1) at src/http_htx.c:161 blk = 0x7b1cdeb75130 v = type = n = next_blk = #2 0x00006077dd028137 in http_find_header (htx=0x7b1cdeb6d1c0, name=..., ctx=0x7b1d05df7eb0, full=0) at src/http_htx.c:248 No locals. #3 http_get_htx_hdr (htx=0x7b1cdeb6d1c0, hdr=..., occ=-1, ctx=0x7b1d05df7eb0, vptr=vptr@entry=0x7b1d05df3638, vlen=0x7b1d05df3640) at src/http_htx.c:922 local_ctx = { blk = 0x7b1ceb9ddec0, value = { ptr = 0x6077dd0758ed "H\211\306\205\333\017\204\322\002", len = 135364253966336 }, lws_before = 32768, lws_after = 0 } val_hist = {{ ptr = 0x7b1cdeb6d390 "MISStransfer-encodingchunked\304access-control-allow-credentialstrueaccess-control-expose-headers*varyOriginmd-edge-crsTk9ORQL\240\270\374\373\255\304\070\211\253aN\245\324\265Ni\324\070\064\022\033V\243\246\233\316L\363\005\273sz\254\071\246\064rc\305\206\334[\226;\222\027\220\071\225\204\005$-J$g\307,e\337\367c#\333\303\323\372\361\221\365~\036O7\002\016\063\030\314y|\307\217\223#\304}\236]1\347\355\036L\374\271\036\035\375\337\240\327^\320\236\377", len = 4 }, { ptr = 0x7b1ceb9dded0 "", len = 135364251711819 }, { ptr = 0x7b1ceb995c00 "\002W\356\004\016", len = 135364121367576 }, { ptr = 0xffffffff , len = 135364254059896 }, { ptr = 0x7b1cea5e9400 "\t\b", len = 0 }, { ptr = 0x0, len = 135364582782496 }, { ptr = 0x7b1cf245e000 "\002D\001\022", len = 135364582782496 }, { ptr = 0x7b1ceb995c00 "\002W\356\004\016", len = 135364582782496 }, { ptr = 0x7b1cf245e000 "\002D\001\022", len = 135364582782496 }, { ptr = 0x7b1cea5e9400 "\t\b", len = 0 }} found = 1 hist_idx = 1 #4 0x00006077dd078791 in smp_fetch_hdr (args=0x7b1cf247fd00, smp=0x7b1d05df3620, kw=, private=) at src/http_fetch.c:964 chn = check = htx = 0x30000001 ctx = 0x6077dd026080 <__http_find_header+576> occ = name = { ptr = 0x11 , len = 17 } #5 0x00006077dcfdd372 in sample_process (px=, sess=, strm=, opt=, expr=0x7b1cf2432aa0, p=p@entry=0x7b1d05df3620) at src/sample.c:1370 No locals. #6 0x00006077dd083deb in action_store (rule=0x7b1cf2476ac0, px=0x7b1cdeb6d3f9, sess=0x11, s=0x11, flags=) at src/vars.c:814 smp = { flags = 144, data = { type = 7, u = { sint = 0, ipv4 = { s_addr = 0 }, ipv6 = { __in6_u = { __u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0} } }, str = { size = 0, area = 0x0, data = 0, head = 0 }, meth = { meth = HTTP_METH_OPTIONS, str = { size = 0, area = 0x0, data = 0, head = 0 } } } }, ctx = { p = 0x7b1d05df7eb0, i = 98533040, ll = 135364582801072, d = 6.6878990025642532e-310, a = {0x7b1d05df7eb0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0} }, px = 0x7b1cf245e000, sess = 0x7b1ceb9ddec0, strm = 0x7b1cea5e9400, opt = 3 } fmtstr = 0x0 dir = 805306369 #7 0x00006077dcfbfc75 in http_res_get_intercept_rule (px=0x7b1cf245e000, def_rules=0x0, rules=0x7b1cf245e068, s=s@entry=0x7b1cea5e9400, final=final@entry=1 '\001') at src/http_ana.c:2862 sess = 0x7b1ceb9ddec0 txn = 0x7b1ceb995c00 rule_ret = act_opts = 3 rule = 0x7b1cf2476ac0 #8 0x00006077dcfc0837 in http_eval_after_res_rules (s=s@entry=0x7b1cea5e9400) at src/http_ana.c:2977 sess = 0x7b1ceb9ddec0 ret = HTTP_RULE_RES_CONT def_rules = 0x11 rules = 0x11 #9 0x00006077dcfbf40c in http_process_res_common (s=0x7b1cea5e9400, rep=rep@entry=0x7b1cea5e9470, an_bit=an_bit@entry=16777216, px=) at src/http_ana.c:1893 sess = 0x7b1ceb9ddec0 txn = 0x7b1ceb995c00 msg = 0x7b1ceb995c00 ret = htx = 0x7b1cdeb6d1c0 cur_proxy = #10 0x00006077dcf851a2 in process_stream (t=t@entry=0x7b1ce70f6080, context=context@entry=0x7b1cea5e9400, state=) at src/stream.c:2091 max_loops = 199 ana_back = 16777216 ana_list = 16777216 scb_flags_ana = 4145 scf_flags_ana = 1058 s = 0x7b1cea5e9400 sess = 0x7b1ceb9ddec0 req = 0x7b1cea5e9428 res = 0x7b1cea5e9470 scf = 0x7b1ceb9f58a0 scb = 0x7b1cea518020 rate = rqf_last = 0 rpf_last = scf_flags = 1058 scb_flags = 4145 rq_prod_last = 8 rq_cons_last = 8 rp_prod_last = rp_cons_last = req_ana_back = 32768 res_ana_back = srv = #11 0x00006077dd0a5acb in run_tasks_from_lists (budgets=budgets@entry=0x7b1d05df39c0) at src/task.c:632 _ = { func = 0x6077dd3d6439 "run_tasks_from_lists", file = 0x6077dd3d644e "src/task.c", line = 657, what = 6 '\006', arg8 = 0 '\000', arg32 = 0 } tl_queues = 0x6077dd7ab7d0 budget_mask = 15 '\017' profile_entry = 0x0 done = 0 queue = 1 t = 0x7b1ce70f6080 process = 0x6077dcf83d60 ctx = 0x7b1cea5e9400 state = #12 0x00006077dd0a663a in process_runnable_tasks () at src/task.c:876 max = {0, 68, 23, 0} tt = 0x6077dd7ab700 default_weights = {64, 48, 16, 1} heavy_queued = 1 max_processed = 91 max_total = queue = 4 budget = 69 grq = lrq = gpicked = lpicked = t = tmp_list = #13 0x00006077dd03fcc4 in run_poll_loop () at src/haproxy.c:3075 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = #14 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #15 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #16 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 7 (Thread 0x7b1ceea00640 (LWP 45549)): #0 0x00007b1d06ca5e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00006077dce90957 in _do_poll (p=, exp=852128, wake=0) at src/ev_epoll.c:232 timeout = 25 updt_idx = fd = old_fd = wait_time = 25 status = count = #2 0x00006077dd03fc8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = 85993984 #3 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 6 (Thread 0x7b1cee000640 (LWP 45550)): #0 0x00007b1d06ca5e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00006077dce90957 in _do_poll (p=, exp=852107, wake=0) at src/ev_epoll.c:232 timeout = 5 updt_idx = fd = old_fd = wait_time = 5 status = count = #2 0x00006077dd03fc8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -338296832 #3 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7b1ced600640 (LWP 45551)): #0 0x00006077dd1942d3 in EVP_CIPHER_CTX_set_num () No symbol table info available. #1 0x00006077dd1862b4 in aes_ctr_cipher () No symbol table info available. #2 0x00006077dd1938cb in evp_EncryptDecryptUpdate () No symbol table info available. #3 0x00006077dcef86a6 in quic_tls_aes_encrypt (out=out@entry=0x7b1ced5f2a08 "W\246\066\033\016", in=in@entry=0x7b1ce097abfd "\002Qd\305)){\027\325\266\061\310=W\221O\037\243\350\211b-\237\360\215\224\210\"\225[\f\f\\\361\033\323\372Q\363\312\365\211:\326R\243D\225\034\251l\215i\024dR\301\307\346\276\212=\320\223\to\205\303\204N\030P\336\256Dcv\024\272\245\205\025\317\327%,\257\260X\024\237\324\255f4\211\\\375\b&/\340\061\060\027.I\324J(N@r\376(M\035\016\062J\fQh\201\203_.\033\374\006d\223\342\331\070fm\352\065\066V\211\030\064\001.b\361\071\330\357\020n\327\315m\202\217\t\307\063\266\234m\267\210!\241\370\343\332\036^Xr~\363i\210\325[\261\230\303s/o \f\340\353\v\245\303x>\361\016\020f\340pO\"2\312\070cH\276\333\243c<)\312\341h*\350(\363\321\262\244\271`\277\250\322i\232ij\371f\334T~\275\232/\213\340\266r\n\265\376\360M\345-\353\036j\252\337\244R\322\202\240.\256%3}\335\062}\331\377\263e\317\361\272\267\025$U\337\337\265\312\255u\316\272\230[u\355\375s\331$u\261\002\fu0\306\353\177\257)\314\211m2\201\027\326\n\214\t\366I\037\037\266\253j\035'\315\001fY\247\256\212Y;P\243\277$UN\334\067\002\"\267.\214\332d\237\307\t44\032\343\264z\342C_\330\f*\320\214\342\320\177\206\310,\271\357Qo1\202\017\003\211\336\376\071\345\253\310n\v\317\265\324\356I\351}\367FgW\234\216\334,t\241\002\006\227!\375\036\330\274u\235dN\024\352\335\004\016X\330\063\252:W\257@\367\020\032\rj)f8\254\311\021\024#\335\340\364G\006\317\266Q-\356\273U\372\263\334\257\306C\334\343\335S\354\246\220\\\260/u-E\v\255\261:\251\177\211\362\336IW\251\017}\265\024Q%\217k\003\002\260L\360\024\375\344r\233\206dj1"..., inlen=inlen@entry=5, ctx=0x7b1ce941ddc0) at src/quic_tls.c:625 ret = 0 #4 0x00006077dcedf290 in quic_apply_header_protection (qc=qc@entry=0x7b1ce9521800, pos=0x7b1ce097abf8 "`\261\340>\234\002Qd\305)){\027\325\266\061\310=W\221O\037\243\350\211b-\237\360\215\224\210\"\225[\f\f\\\361\033\323\372Q\363\312\365\211:\326R\243D\225\034\251l\215i\024dR\301\307\346\276\212=\320\223\to\205\303\204N\030P\336\256Dcv\024\272\245\205\025\317\327%,\257\260X\024\237\324\255f4\211\\\375\b&/\340\061\060\027.I\324J(N@r\376(M\035\016\062J\fQh\201\203_.\033\374\006d\223\342\331\070fm\352\065\066V\211\030\064\001.b\361\071\330\357\020n\327\315m\202\217\t\307\063\266\234m\267\210!\241\370\343\332\036^Xr~\363i\210\325[\261\230\303s/o \f\340\353\v\245\303x>\361\016\020f\340pO\"2\312\070cH\276\333\243c<)\312\341h*\350(\363\321\262\244\271`\277\250\322i\232ij\371f\334T~\275\232/\213\340\266r\n\265\376\360M\345-\353\036j\252\337\244R\322\202\240.\256%3}\335\062}\331\377\263e\317\361\272\267\025$U\337\337\265\312\255u\316\272\230[u\355\375s\331$u\261\002\fu0\306\353\177\257)\314\211m2\201\027\326\n\214\t\366I\037\037\266\253j\035'\315\001fY\247\256\212Y;P\243\277$UN\334\067\002\"\267.\214\332d\237\307\t44\032\343\264z\342C_\330\f*\320\214\342\320\177\206\310,\271\357Qo1\202\017\003\211\336\376\071\345\253\310n\v\317\265\324\356I\351}\367FgW\234\216\334,t\241\002\006\227!\375\036\330\274u\235dN\024\352\335\004\016X\330\063\252:W\257@\367\020\032\rj)f8\254\311\021\024#\335\340\364G\006\317\266Q-\356\273U\372\263\334\257\306C\334\343\335S\354\246\220\\\260/u-E\v\255\261:\251\177\211\362\336IW\251\017}\265\024Q%\217k\003\002\260L\360\024\375\344r"..., pn=0x7b1ce097abf9 "\261\340>\234\002Qd\305)){\027\325\266\061\310=W\221O\037\243\350\211b-\237\360\215\224\210\"\225[\f\f\\\361\033\323\372Q\363\312\365\211:\326R\243D\225\034\251l\215i\024dR\301\307\346\276\212=\320\223\to\205\303\204N\030P\336\256Dcv\024\272\245\205\025\317\327%,\257\260X\024\237\324\255f4\211\\\375\b&/\340\061\060\027.I\324J(N@r\376(M\035\016\062J\fQh\201\203_.\033\374\006d\223\342\331\070fm\352\065\066V\211\030\064\001.b\361\071\330\357\020n\327\315m\202\217\t\307\063\266\234m\267\210!\241\370\343\332\036^Xr~\363i\210\325[\261\230\303s/o \f\340\353\v\245\303x>\361\016\020f\340pO\"2\312\070cH\276\333\243c<)\312\341h*\350(\363\321\262\244\271`\277\250\322i\232ij\371f\334T~\275\232/\213\340\266r\n\265\376\360M\345-\353\036j\252\337\244R\322\202\240.\256%3}\335\062}\331\377\263e\317\361\272\267\025$U\337\337\265\312\255u\316\272\230[u\355\375s\331$u\261\002\fu0\306\353\177\257)\314\211m2\201\027\326\n\214\t\366I\037\037\266\253j\035'\315\001fY\247\256\212Y;P\243\277$UN\334\067\002\"\267.\214\332d\237\307\t44\032\343\264z\342C_\330\f*\320\214\342\320\177\206\310,\271\357Qo1\202\017\003\211\336\376\071\345\253\310n\v\317\265\324\356I\351}\367FgW\234\216\334,t\241\002\006\227!\375\036\330\274u\235dN\024\352\335\004\016X\330\063\252:W\257@\367\020\032\rj)f8\254\311\021\024#\335\340\364G\006\317\266Q-\356\273U\372\263\334\257\306C\334\343\335S\354\246\220\\\260/u-E\v\255\261:\251\177\211\362\336IW\251\017}\265\024Q%\217k\003\002\260L\360\024\375\344r\233"..., pnlen=pnlen@entry=1, tls_ctx=tls_ctx@entry=0x7b1ce6a2e570, fail=fail@entry=0x7b1ced5f2b6c) at src/quic_tx.c:1391 mask = "W\246\066\033\016" aes_ctx = i = #5 0x00006077dcedc2a0 in qc_build_pkt (qel=0x7b1ce6a2e540, tls_ctx=0x7b1ce6a2e570, frms=0x7b1ced5f3730, qc=0x7b1ce9521800, ver=0x6077dd506248 , dglen=0, must_ack=, padding=, probe=, cc=0, pos=, end=, pkt_type=, err=) at src/quic_tx.c:2109 encrypt_failure = 0 pkt = 0x7b1ce6935180 pn_len = buf_pn = 0x7b1ce097abf9 "\261\340>\234\002Qd\305)){\027\325\266\061\310=W\221O\037\243\350\211b-\237\360\215\224\210\"\225[\f\f\\\361\033\323\372Q\363\312\365\211:\326R\243D\225\034\251l\215i\024dR\301\307\346\276\212=\320\223\to\205\303\204N\030P\336\256Dcv\024\272\245\205\025\317\327%,\257\260X\024\237\324\255f4\211\\\375\b&/\340\061\060\027.I\324J(N@r\376(M\035\016\062J\fQh\201\203_.\033\374\006d\223\342\331\070fm\352\065\066V\211\030\064\001.b\361\071\330\357\020n\327\315m\202\217\t\307\063\266\234m\267\210!\241\370\343\332\036^Xr~\363i\210\325[\261\230\303s/o \f\340\353\v\245\303x>\361\016\020f\340pO\"2\312\070cH\276\333\243c<)\312\341h*\350(\363\321\262\244\271`\277\250\322i\232ij\371f\334T~\275\232/\213\340\266r\n\265\376\360M\345-\353\036j\252\337\244R\322\202\240.\256%3}\335\062}\331\377\263e\317\361\272\267\025$U\337\337\265\312\255u\316\272\230[u\355\375s\331$u\261\002\fu0\306\353\177\257)\314\211m2\201\027\326\n\214\t\366I\037\037\266\253j\035'\315\001fY\247\256\212Y;P\243\277$UN\334\067\002\"\267.\214\332d\237\307\t44\032\343\264z\342C_\330\f*\320\214\342\320\177\206\310,\271\357Qo1\202\017\003\211\336\376\071\345\253\310n\v\317\265\324\356I\351}\367FgW\234\216\334,t\241\002\006\227!\375\036\330\274u\235dN\024\352\335\004\016X\330\063\252:W\257@\367\020\032\rj)f8\254\311\021\024#\335\340\364G\006\317\266Q-\356\273U\372\263\334\257\306C\334\343\335S\354\246\220\\\260/u-E\v\255\261:\251\177\211\362\336IW\251\017}\265\024Q%\217k\003\002\260L\360\024\375\344r\233"... pn = 4273 payload = 0x7b1ce097abfa "\340>\234\002Qd\305)){\027\325\266\061\310=W\221O\037\243\350\211b-\237\360\215\224\210\"\225[\f\f\\\361\033\323\372Q\363\312\365\211:\326R\243D\225\034\251l\215i\024dR\301\307\346\276\212=\320\223\to\205\303\204N\030P\336\256Dcv\024\272\245\205\025\317\327%,\257\260X\024\237\324\255f4\211\\\375\b&/\340\061\060\027.I\324J(N@r\376(M\035\016\062J\fQh\201\203_.\033\374\006d\223\342\331\070fm\352\065\066V\211\030\064\001.b\361\071\330\357\020n\327\315m\202\217\t\307\063\266\234m\267\210!\241\370\343\332\036^Xr~\363i\210\325[\261\230\303s/o \f\340\353\v\245\303x>\361\016\020f\340pO\"2\312\070cH\276\333\243c<)\312\341h*\350(\363\321\262\244\271`\277\250\322i\232ij\371f\334T~\275\232/\213\340\266r\n\265\376\360M\345-\353\036j\252\337\244R\322\202\240.\256%3}\335\062}\331\377\263e\317\361\272\267\025$U\337\337\265\312\255u\316\272\230[u\355\375s\331$u\261\002\fu0\306\353\177\257)\314\211m2\201\027\326\n\214\t\366I\037\037\266\253j\035'\315\001fY\247\256\212Y;P\243\277$UN\334\067\002\"\267.\214\332d\237\307\t44\032\343\264z\342C_\330\f*\320\214\342\320\177\206\310,\271\357Qo1\202\017\003\211\336\376\071\345\253\310n\v\317\265\324\356I\351}\367FgW\234\216\334,t\241\002\006\227!\375\036\330\274u\235dN\024\352\335\004\016X\330\063\252:W\257@\367\020\032\rj)f8\254\311\021\024#\335\340\364G\006\317\266Q-\356\273U\372\263\334\257\306C\334\343\335S\354\246\220\\\260/u-E\v\255\261:\251\177\211\362\336IW\251\017}\265\024Q%\217k\003\002\260L\360\024\375\344r\233\206"... payload_len = aad_len = first_byte = last_byte = __x = __x = __x = __x = __x = __x = msg = msg = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = #6 qc_prep_pkts (qc=qc@entry=0x7b1ce9521800, buf=buf@entry=0x7b1ce9521bb8, qels=qels@entry=0x7b1ced5f3568) at src/quic_tx.c:580 err = QC_BUILD_PKT_ERR_NONE cur_pkt = pkt_type = frms = 0x7b1ced5f3730 ver = 0x6077dd506248 tls_ctx = 0x7b1ce6a2e570 next_qel = probe = must_ack = ret = -1 cc = 0 padding = prv_pkt = 0x0 first_pkt = 0x0 pos = 0x7b1ce097abf8 "`\261\340>\234\002Qd\305)){\027\325\266\061\310=W\221O\037\243\350\211b-\237\360\215\224\210\"\225[\f\f\\\361\033\323\372Q\363\312\365\211:\326R\243D\225\034\251l\215i\024dR\301\307\346\276\212=\320\223\to\205\303\204N\030P\336\256Dcv\024\272\245\205\025\317\327%,\257\260X\024\237\324\255f4\211\\\375\b&/\340\061\060\027.I\324J(N@r\376(M\035\016\062J\fQh\201\203_.\033\374\006d\223\342\331\070fm\352\065\066V\211\030\064\001.b\361\071\330\357\020n\327\315m\202\217\t\307\063\266\234m\267\210!\241\370\343\332\036^Xr~\363i\210\325[\261\230\303s/o \f\340\353\v\245\303x>\361\016\020f\340pO\"2\312\070cH\276\333\243c<)\312\341h*\350(\363\321\262\244\271`\277\250\322i\232ij\371f\334T~\275\232/\213\340\266r\n\265\376\360M\345-\353\036j\252\337\244R\322\202\240.\256%3}\335\062}\331\377\263e\317\361\272\267\025$U\337\337\265\312\255u\316\272\230[u\355\375s\331$u\261\002\fu0\306\353\177\257)\314\211m2\201\027\326\n\214\t\366I\037\037\266\253j\035'\315\001fY\247\256\212Y;P\243\277$UN\334\067\002\"\267.\214\332d\237\307\t44\032\343\264z\342C_\330\f*\320\214\342\320\177\206\310,\271\357Qo1\202\017\003\211\336\376\071\345\253\310n\v\317\265\324\356I\351}\367FgW\234\216\334,t\241\002\006\227!\375\036\330\274u\235dN\024\352\335\004\016X\330\063\252:W\257@\367\020\032\rj)f8\254\311\021\024#\335\340\364G\006\317\266Q-\356\273U\372\263\334\257\306C\334\343\335S\354\246\220\\\260/u-E\v\255\261:\251\177\211\362\336IW\251\017}\265\024Q%\217k\003\002\260L\360\024\375\344r"... end = 0x7b1ce097b0dc "U\202\262\216&\223\231\373\031\350\243=\225\216y\035\321\233\337\031\275\035\333\301>K\r\345\365\021Xsc\020v|S\212-W\262\061w3y\267\236\244\206\242y\206h\272\223\205\061\333\234\340_.\201\230q\322\210\036\336\017\003\207\353\303?[\227|\307\367\201\212\253$t\203\373\300*E\021\256\205F\360/t\204\266\223\034\071\037\322\360p\322c\242r\177gk\f\360pD\220=\017!\216|\004\333\221\322\306\002\241\266\346\bs0%\363\230\300\313\305" dglen = total = 1252 qel = tmp_qel = #7 0x00006077dced86a6 in qc_send (qc=qc@entry=0x7b1ce9521800, old_data=old_data@entry=0, send_list=send_list@entry=0x7b1ced5f3568) at src/quic_tx.c:719 ret = status = 0 buf = 0x7b1ce9521bb8 qel = tmp_qel = #8 0x00006077dced8368 in qc_send_mux (qc=0x7b1ce9521800, frms=frms@entry=0x7b1ced5f3730) at src/quic_tx.c:448 send_list = { n = 0x7b1ce6a2e550, p = 0x7b1ce6a2e550 } ret = #9 0x00006077dcec9921 in qcc_send_frames (qcc=0x7b1ceb5e3a80, frms=0x7b1ced5f3730) at src/mux_quic.c:1927 No locals. #10 0x00006077dcec7eea in qcc_io_send (qcc=qcc@entry=0x7b1ceb5e3a80) at src/mux_quic.c:2208 _ = { func = 0x6077dd374693 "qcc_io_send", file = 0x6077dd373036 "src/mux_quic.c", line = 2255, what = 3 '\003', arg8 = 0 '\000', arg32 = 0 } frms = { n = 0x7b1ce32366c0, p = 0x7b1ce3237f80 } qcs_failed = { n = 0x7b1ced5f3720, p = 0x7b1ced5f3720 } first_qcs = window_conn = total = 94634 qcs = qcs_tmp = ret = resent = #11 0x00006077dcec708e in qcc_io_cb (t=, ctx=ctx@entry=0x7b1ceb5e3a80, status=) at src/mux_quic.c:2543 qcc = #12 0x00006077dd0a5c92 in run_tasks_from_lists (budgets=budgets@entry=0x7b1ced5f39c0) at src/task.c:596 _ = { func = 0x6077dd3d6439 "run_tasks_from_lists", file = 0x6077dd3d644e "src/task.c", line = 657, what = 6 '\006', arg8 = 0 '\000', arg32 = 0 } tl_queues = 0x6077dd7ac1d0 budget_mask = 15 '\017' profile_entry = 0x0 done = 2 queue = 2 t = 0x7b1ce95af070 process = 0x6077dcec7060 ctx = 0x7b1ceb5e3a80 state = 3982436620 #13 0x00006077dd0a663a in process_runnable_tasks () at src/task.c:876 max = {0, 0, 90, 0} tt = 0x6077dd7ac100 default_weights = {64, 48, 16, 1} heavy_queued = 1 max_processed = 93 max_total = queue = 4 budget = 0 grq = lrq = gpicked = lpicked = t = tmp_list = #14 0x00006077dd03fcc4 in run_poll_loop () at src/haproxy.c:3075 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = #15 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #16 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #17 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7b1cecc00640 (LWP 45552)): #0 0x00006077dd0c020f in vp_putblk_ofs (ofs=0, blk=0x6077dd3d9a84 "[", len=1, v1=, v2=) at include/haproxy/vecpair.h:319 block = 123 ret = 0 #1 vp_putblk (blk=0x6077dd3d9a84 "[", len=1, v1=, v2=) at include/haproxy/vecpair.h:367 No locals. #2 ring_write (ring=0x7b1d064a1a00, maxlen=159, pfx=, npfx=2, msg=msg@entry=0x7b1cecbf3280, nmsg=nmsg@entry=11) at src/ring.c:398 len = 1 _ = { func = 0x6077dd3e5730 "ring_write", file = 0x6077dd3dc3d0 "src/ring.c", line = 443, what = 7 '\a', arg8 = 0 '\000', arg32 = 0 } cell = { to_send_self = 161, needed_tot = 161, maxlen = 159, pfx = 0x7b1cecbf5b40, npfx = 2, msg = 0x7b1cecbf3280, nmsg = 11, next = 0x0 } v1 = v2 = ring_queue_ptr = tail_ptr = msglen = 36 sent = 0 i = 0 lenlen = needed = ring_area = 0x7b1cf26000c0 "lags=0x6000008" ring_size = next_cell = curr_cell = 0x7b1cecbf3140 tail_ofs = head_ofs = lock_ptr = 0x7b1cf5acf9e9 "0 flags=0x6038018" readers = 0 '\000' dellen = dellenlen = new_tail_ofs = leave = wait_for_flush = #3 0x00006077dd090f4d in __sink_write (sink=sink@entry=0x7b1d0649a1c0, hdr=..., maxlen=, maxlen@entry=0, msg=msg@entry=0x7b1cecbf3280, nmsg=nmsg@entry=11) at src/sink.c:198 npfx = 2 pfx = 0x0 #4 0x00006077dd0adb03 in sink_write (sink=0x7b1d0649a1c0, hdr=..., maxlen=0, msg=0x7b1cecbf3280, nmsg=11) at include/haproxy/sink.h:68 sent = #5 __trace (level=level@entry=TRACE_LEVEL_DEVELOPER, mask=mask@entry=131072, src=src@entry=0x6077dd54ae58 , where=..., func=, a1=0x7b1cec018000, a2=0x0, a3=0x0, a4=0x0, cb=, msg=...) at src/trace.c:294 lockon_ptr = 0x0 ist_func = { ptr = 0x6077dd371d21 "qc_newly_acked_pkts", len = 19 } tnum = "07|" line = {{ ptr = 0x6077dd3d9a84 "[", len = 1 }, { ptr = 0x7b1cecbf3244 "07|", len = 3 }, { ptr = 0x6077dd3daaca "quic", len = 4 }, { ptr = 0x6077dd3de4a3 "|", len = 1 }, { ptr = 0x6077dd3d80b5 "5", len = 1 }, { ptr = 0x6077dd3de4a3 "|", len = 1 }, { ptr = 0x6077dd371d39 "quic_rx.c:356", len = 13 }, { ptr = 0x6077dd3fad10 "] ", len = 2 }, { ptr = 0x6077dd371d21 "qc_newly_acked_pkts", len = 19 }, { ptr = 0x6077dd41c84a "(): ", len = 4 }, { ptr = 0x7b1cec04eec0 "leaving : qc@0x7b1cec018000 idle_timer_task@0x7b1ce6ce7500 flags=0x6038018", len = 74 }, { ptr = 0x1 , len = 106067928748167 }} words = 11 ret = #6 0x00006077dceb9639 in qc_newly_acked_pkts (qc=0x7b1cec018000, pkts=0x7b1cec1cd168, newly_acked_pkts=0x7b1cecbf3418, largest_node=0x0, largest=2226, smallest=) at src/quic_rx.c:356 node = pkt = __x = __x = __x = __x = #7 qc_parse_ack_frm (qc=0x7b1cec018000, frm=0x7b1cecbf3548, qel=0x7b1cec196d00, pos=0x7b1cecbf33e0, end=0x7b1ce665b51a "d/m\016\355\027\201d\331\264\260%\234\021\036\257@)v\326\364\364\\M\353\035D\307\a}\322\026\312\227\227\063\263\326G\230m>\t7O(\253\270\261\333\063\236\230\271\224(X\276@)v\326\364\364\\M\353\026\002N\370@\363\003@\266", rtt_sample=) at src/quic_rx.c:580 gap = ack_range = smallest = 202 largest = 2226 newly_acked_pkts = lost_pkts = { n = 0x7b1cecbf3638, p = 0x7b1cecbf3638 } ret = 0 new_largest_acked_pn = pkts = 0x7b1cec1cd168 pkt_flags = 0 largest_node = time_sent = pkt = tmp = ack_frm = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = gap = ack_range = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __n = __ret = __p = __x = __x = #8 qc_parse_pkt_frms (qc=, pkt=, qel=) at src/quic_rx.c:872 rtt_sample = 4294967295 frm = pos = fast_retrans = 0 ret = end = 0x7b1ce665b51a "d/m\016\355\027\201d\331\264\260%\234\021\036\257@)v\326\364\364\\M\353\035D\307\a}\322\026\312\227\227\063\263\326G\230m>\t7O(\253\270\261\333\063\236\230\271\224(X\276@)v\326\364\364\\M\353\026\002N\370@\363\003@\266" __x = __x = nb_streams = fin = strm_frm = __x = __x = __x = __x = __x = __x = __x = __x = rtt_sample = ack_delay = _a = _b = __x = __x = rs_frm = md_frm = __x = __x = msd_frm = __x = __x = ss_frm = __x = __x = conn_id = tree = __lk_r = __set_r = __msk_r = __pl_r = __ptr = __x = __x = __x = __x = iqel = hqel = __x = __x = __x = __x = __x = __x = #9 qc_treat_rx_pkts (qc=qc@entry=0x7b1cec018000) at src/quic_rx.c:1227 pkt = 0x7b1cea91a880 ret = 0 largest_pn = -1 largest_pn_time_received = 0 qel = 0x7b1cec196d00 node = 0x7b1cea91a920 qelbak = #10 0x00006077dcee245b in quic_conn_app_io_cb (t=t@entry=0x7b1cec170000, context=context@entry=0x7b1cec018000, state=) at src/quic_conn.c:578 send_list = { n = 0x7b1cecbf3870, p = 0x7b1cecbf3870 } qc = #11 0x00006077dd0a5c92 in run_tasks_from_lists (budgets=budgets@entry=0x7b1cecbf39c0) at src/task.c:596 _ = { func = 0x6077dd3d6439 "run_tasks_from_lists", file = 0x6077dd3d644e "src/task.c", line = 657, what = 6 '\006', arg8 = 0 '\000', arg32 = 0 } tl_queues = 0x6077dd7ac3d0 budget_mask = 15 '\017' profile_entry = 0x0 done = 8 queue = 2 t = 0x7b1cec170000 process = 0x6077dcee23b0 ctx = 0x7b1cec018000 state = 0 #12 0x00006077dd0a663a in process_runnable_tasks () at src/task.c:876 max = {74, 0, 11, 0} tt = 0x6077dd7ac300 default_weights = {64, 48, 16, 1} heavy_queued = 1 max_processed = 93 max_total = queue = 4 budget = 0 grq = lrq = gpicked = lpicked = t = tmp_list = #13 0x00006077dd03fcc4 in run_poll_loop () at src/haproxy.c:3075 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = #14 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #15 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #16 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7b1cefe00640 (LWP 45547)): #0 0x00007b1d06ca5e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00006077dce90957 in _do_poll (p=, exp=852122, wake=0) at src/ev_epoll.c:232 timeout = 20 updt_idx = fd = old_fd = wait_time = 20 status = count = #2 0x00006077dd03fc8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -355074048 #3 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7b1d0692dac0 (LWP 45545)): #0 0x00007b1d06ca5e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00006077dce90957 in _do_poll (p=, exp=852120, wake=0) at src/ev_epoll.c:232 timeout = 17 updt_idx = fd = old_fd = wait_time = 17 status = count = #2 0x00006077dd03fc8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -232758272 #3 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00006077dd042e5a in main (argc=, argv=0x7ffc2f95fd28) at src/haproxy.c:3991 limit = { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 } pidfd = retry = err = intovf = Thread 1 (Thread 0x7b1cef400640 (LWP 45548)): #0 0x00006077dd1a3644 in CRYPTO_ctr128_encrypt_ctr32 () No symbol table info available. #1 0x00006077dd186286 in aes_ctr_cipher () No symbol table info available. #2 0x00006077dd1938cb in evp_EncryptDecryptUpdate () No symbol table info available. #3 0x00006077dd193102 in EVP_DecryptUpdate () No symbol table info available. #4 0x00006077dcef8776 in quic_tls_aes_decrypt (out=out@entry=0x7b1cef3f2e68 "", in=in@entry=0x7b1ce533d6cd "\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b", inlen=inlen@entry=5, ctx=0x7b1cea0413c0) at src/quic_tls.c:665 ret = 0 #5 0x00006077dcec084f in qc_do_rm_hp (qc=qc@entry=0x7b1ceb014800, pkt=pkt@entry=0x7b1cea075c80, tls_ctx=, largest_pn=1544, pn=0x7b1ce533d6c9 "\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b", byte0=byte0@entry=0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b") at src/quic_rx.c:94 mask = "\000\000\000\000" truncated_pn = 0 ret = 0 sample = 0x7b1cea0413f8 "" pnlen = i = packet_number = #6 0x00006077dcebe999 in qc_try_rm_hp (qc=0x7b1ceb014800, pkt=0x7b1cea075c80, beg=0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b", el=) at src/quic_rx.c:1418 tls_ctx = 0x1 ret = 0 pn = 0x7b1cea0413e8 "\255\363\252B\354E\032\347m\030\305\253\206\006\001>" tel = qel = 0x7b1ceb1f1b80 __x = __x = qc_qel = qc_pktns = __x = __x = tls_ctx = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = msg = #7 qc_rx_pkt_handle (qc=0x7b1ceb014800, pkt=0x7b1cea075c80, dgram=0x7b1cea075e00, beg=0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b", tasklist_head=) at src/quic_rx.c:2064 qv = 0x0 qel = 0x0 b_cspace = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = _msg = _msg_len = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = #8 quic_dgram_parse (dgram=dgram@entry=0x7b1cea075e00, from_qc=from_qc@entry=0x7b1ceb014800, li=0x7b1cf2424b00) at src/quic_rx.c:2197 _ = { func = 0x6077dd37141e "quic_dgram_parse", file = 0x6077dd37145e "src/quic_rx.c", line = 2164, what = 3 '\003', arg8 = 0 '\000', arg32 = 0 } qc = 0x7b1ceb014800 tasklist_head = 0x0 pos = 0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b" pkt = 0x7b1cea075c80 end = #9 0x00006077dcef1d12 in qc_rcv_buf (qc=qc@entry=0x7b1ceb014800) at src/quic_sock.c:880 saddr = { ss_family = 2, __ss_padding = "\243\357O_WI", '\000' , __ss_align = 0 } daddr = { ss_family = 2, __ss_padding = "\001\273-\201\345\001", '\000' , __ss_align = 0 } new_dgram = 0x7b1cea075e00 buf = { size = 32768, area = 0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b", data = 40, head = 0 } ret = 40 l = 0x7b1cf2424b00 params = max_sz = 2048 dgram_buf = 0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b" #10 0x00006077dcee2400 in quic_conn_app_io_cb (t=t@entry=0x7b1cea26bd00, context=context@entry=0x7b1ceb014800, state=) at src/quic_conn.c:559 send_list = { n = 0x7b1cef3f3870, p = 0x7b1cef3f3870 } qc = #11 0x00006077dd0a5c92 in run_tasks_from_lists (budgets=budgets@entry=0x7b1cef3f39c0) at src/task.c:596 _ = { func = 0x6077dd3d6439 "run_tasks_from_lists", file = 0x6077dd3d644e "src/task.c", line = 657, what = 6 '\006', arg8 = 0 '\000', arg32 = 0 } tl_queues = 0x6077dd7abbd0 budget_mask = 15 '\017' profile_entry = 0x0 done = 6 queue = 2 t = 0x7b1cea26bd00 process = 0x6077dcee23b0 ctx = 0x7b1ceb014800 state = 4013894940 #12 0x00006077dd0a663a in process_runnable_tasks () at src/task.c:876 max = {44, 34, 9, 1} tt = 0x6077dd7abb00 default_weights = {64, 48, 16, 1} heavy_queued = 1 max_processed = 93 max_total = queue = 4 budget = 35 grq = lrq = gpicked = lpicked = t = tmp_list = #13 0x00006077dd03fcc4 in run_poll_loop () at src/haproxy.c:3075 _ = { func = 0x6077dd3b8d6b "run_poll_loop", file = 0x6077dd3b8d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = #14 0x00006077dd0441e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 41, __value32 = { __low = 41, __high = 0 } }, __g1_start = { __value64 = 31, __value32 = { __low = 31, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 20, __wrefs = 0, __g_signals = {0, 0} }, __size = ")\000\000\000\000\000\000\000\037", '\000' , "\024", '\000' , __align = 41 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #15 0x00007b1d06c14ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #16 0x00007b1d06ca6850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. ```

traces-2.gz

core 2 - in chunk_appendf ```plain Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -dMno-mer'. Program terminated with signal SIGABRT, Aborted. #0 0x00007667b0f149fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6 [Current thread is 1 (Thread 0x766798000640 (LWP 34076))] (gdb) t a a bt full Thread 8 (Thread 0x766798a00640 (LWP 34075)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764793, wake=0) at src/ev_epoll.c:232 timeout = 4 updt_idx = fd = old_fd = wait_time = 4 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1783234560 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 7 (Thread 0x76679a800640 (LWP 34072)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764809, wake=0) at src/ev_epoll.c:232 timeout = 20 updt_idx = fd = old_fd = wait_time = 20 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1766457344 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 6 (Thread 0x766797600640 (LWP 34077)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764835, wake=0) at src/ev_epoll.c:232 timeout = 46 updt_idx = fd = old_fd = wait_time = 46 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1787428864 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 5 (Thread 0x766799400640 (LWP 34074)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764835, wake=0) at src/ev_epoll.c:232 timeout = 46 updt_idx = fd = old_fd = wait_time = 46 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1779040256 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7667b0c2bac0 (LWP 34070)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764808, wake=0) at src/ev_epoll.c:232 timeout = 19 updt_idx = fd = old_fd = wait_time = 19 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1665113088 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00005d932ba7ae5a in main (argc=, argv=0x7ffdba802e48) at src/haproxy.c:3991 limit = { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 } pidfd = retry = err = intovf = Thread 3 (Thread 0x766799e00640 (LWP 34073)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764835, wake=0) at src/ev_epoll.c:232 timeout = 46 updt_idx = fd = old_fd = wait_time = 46 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1770651648 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7667b0200640 (LWP 34071)): #0 0x00007667b0fa3e2e in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00005d932b8c8957 in _do_poll (p=, exp=764835, wake=0) at src/ev_epoll.c:232 timeout = 46 updt_idx = fd = old_fd = wait_time = 46 status = count = #2 0x00005d932ba77c8f in run_poll_loop () at src/haproxy.c:3147 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = -1352652288 #3 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #4 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #5 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. Thread 1 (Thread 0x766798000640 (LWP 34076)): #0 0x00007667b0f149fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007667b0ec0476 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007667b0ea67f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #3 0x00005d932ba8ed43 in ha_panic () at src/debug.c:624 old = 0x0 thr = #4 No symbol table info available. #5 0x00007667b0f0be0c in _IO_default_xsputn () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #6 0x00007667b0ef500c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #7 0x00007667b0f0649a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #8 0x00005d932bb2cc48 in chunk_appendf (chk=chk@entry=0x766797ff8408, fmt=0xfbad8000 ) at src/chunk.c:153 argp = {{ gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x766797ff2b40, reg_save_area = 0x766797ff2a70 }} ret = room = #9 0x00005d932b9209d2 in chunk_frm_appendf (buf=buf@entry=0x766797ff8408, frm=frm@entry=0x76678f069180) at src/quic_frame.c:112 No locals. #10 0x00005d932b933b1b in quic_trace (level=, mask=4398046511104, src=, where=..., func=..., a1=0x766793f84800, a2=0x766793f04a50, a3=0x0, a4=0x0) at src/quic_trace.c:219 frm = 0x76678f069180 l = 0x766793f04a50 tls_ctx = qc = 0x766793f84800 #11 0x00005d932bae5aa5 in __trace (level=level@entry=TRACE_LEVEL_PROTO, mask=mask@entry=4398046511104, src=src@entry=0x5d932bf82e58 , where=..., func=func@entry=0x0, a1=0x766793f84800, a2=0x766793f04a50, a3=0x0, a4=0x0, cb=, msg=...) at src/trace.c:280 lockon_ptr = 0x0 ist_func = { ptr = 0x0, len = 0 } tnum = "06|" line = {{ ptr = 0x5d932be11a84 "[", len = 1 }, { ptr = 0x766797ff2c44 "06|", len = 3 }, { ptr = 0x5d932be12aca "quic", len = 4 }, { ptr = 0x5d932be164a3 "|", len = 1 }, { ptr = 0x5d932be100b2 "2345", len = 1 }, { ptr = 0x5d932be164a3 "|", len = 1 }, { ptr = 0x5d932bdb066b "uic_tx.c:1852", len = 13 }, { ptr = 0x5d932be32d10 "] ", len = 2 }, { ptr = 0x5d932bdb0767 "quic_build_packet_short_header", len = 30 }, { ptr = 0x5d932be5484a "(): ", len = 4 }, { ptr = 0x76679624ec00 "Avail. ack eliciting frames : qc@0x766793f84800 idle_timer_task@0x766793ed2e20 flags=0x6038018 frm@0x766793611340 STREAM_E uni=0 fin=0 id=0 off=9192758 len=657 frm@0x7667936114c0 STREAM_E uni=0 fin=0 id=0 off=9193415 len=1225 frm@0x766793611640 STREAM_E uni=0 fin=0 id=0 off=9194640 len=1225 frm@0x7667936117c0 STREAM_E uni=0 fin=0 id=0 off=9195865 len=1225 frm@0x766793611940 STREAM_E uni=0 fin=0 id=0 off=9197090 len=1225 frm@0x766793611ac0 STREAM_E uni=0 fin=0 id=0 off=9198315 len=1225 frm@0x766793611c40 STR"..., len = 74 }, { ptr = 0x5d932b9110da "H\205\300H\211D$@\017\204\006;", len = 102886674275428 }} words = 8 ret = #12 0x00005d932b91362d in qc_do_build_pkt (pos=0x76678f08663b "R\327\023\277t*?\303WF\252`W\310\371\003\311v\333%\372\236\205\004\fKJ\336\343\033\331\024\224+i\211\223M\267B\203q\233?\243\351\363|P\223\310\302;6\374KQ\254\324\347\334\227\025\374l0j\255\215\006pel\305\232\065\221ITW\324S\177a4*F\366sSY\257\233\244sfU(\235\023\353\262\005\210[\316\274M\271VG\353\024tl\t\242\327h#d\252\203K\257\020\204\023\206\205\244i\264\251\a\361\316\226 \317\241\310\274A\nZ`\361>\273\345\245\016y\230\005\322\240s\321\227\230", end=0x76678f086b0e "\332\bm\366=o\353\v\350Q\325GHl4q\344\004\300Uq\223gv", dglen=0, pkt=0x7667937340c0, pn=24323, must_ack=, padding=0, cc=0, probe=, qel=0x7667963cedc0, qc=0x766793f84800, ver=0x5d932bf3e248 , frms=0x766793f04a50, pn_len=, buf_pn=) at src/quic_tx.c:1852 room = 1235 len_frms = 0 frm = ack_frm = cc_frm = frm_list = { n = 0x766797ff2e10, p = 0x766797ff2e10 } ret = 0 beg = 0x76678f08663a "AR\327\023\277t*?\303WF\252`W\310\371\003\311v\333%\372\236\205\004\fKJ\336\343\033\331\024\224+i\211\223M\267B\203q\233?\243\351\363|P\223\310\302;6\374KQ\254\324\347\334\227\025\374l0j\255\215\006pel\305\232\065\221ITW\324S\177a4*F\366sSY\257\233\244sfU(\235\023\353\262\005\210[\316\274M\271VG\353\024tl\t\242\327h#d\252\203K\257\020\204\023\206\205\244i\264\251\a\361\316\226 \317\241\310\274A\nZ`\361>\273\345\245\016y\230\005\322\240s\321\227\230" ack_frm_len = 0 len = 2 add_ping_frm = padding_len = len_sz = cf = payload = head_len = rx_largest_acked_pn = __x = __x = path_room = __x = __x = __x = __x = arngs = msg = room = __x = __x = __x = __x = tmp_cf = room = __x = __x = __ret = __n = __p = __x = __x = __x = __x = #13 qc_build_pkt (qel=0x7667963cedc0, tls_ctx=0x7667963cedf0, frms=0x766793f04a50, qc=0x766793f84800, ver=0x5d932bf3e248 , dglen=0, must_ack=, padding=0, probe=, cc=0, pos=, end=, pkt_type=, err=) at src/quic_tx.c:2087 encrypt_failure = 0 pkt = 0x7667937340c0 pn_len = buf_pn = 0x0 pn = 24323 payload = payload_len = aad_len = first_byte = last_byte = __x = __x = __x = __x = __x = __x = msg = msg = __x = __x = __x = __x = __x = __x = __x = __x = __x = __x = #14 qc_prep_pkts (qc=qc@entry=0x766793f84800, buf=buf@entry=0x766793f84bb8, qels=qels@entry=0x766797ff3870) at src/quic_tx.c:580 err = QC_BUILD_PKT_ERR_NONE cur_pkt = pkt_type = frms = 0x766793f04a50 ver = 0x5d932bf3e248 tls_ctx = 0x7667963cedf0 next_qel = probe = must_ack = ret = -1 cc = 0 padding = prv_pkt = 0x0 first_pkt = 0x0 pos = 0x76678f08663a "AR\327\023\277t*?\303WF\252`W\310\371\003\311v\333%\372\236\205\004\fKJ\336\343\033\331\024\224+i\211\223M\267B\203q\233?\243\351\363|P\223\310\302;6\374KQ\254\324\347\334\227\025\374l0j\255\215\006pel\305\232\065\221ITW\324S\177a4*F\366sSY\257\233\244sfU(\235\023\353\262\005\210[\316\274M\271VG\353\024tl\t\242\327h#d\252\203K\257\020\204\023\206\205\244i\264\251\a\361\316\226 \317\241\310\274A\nZ`\361>\273\345\245\016y\230\005\322\240s\321\227\230" end = 0x76678f086b1e "\344\004\300Uq\223gv" dglen = total = 10016 qel = tmp_qel = #15 0x00005d932b9106a6 in qc_send (qc=qc@entry=0x766793f84800, old_data=old_data@entry=0, send_list=send_list@entry=0x766797ff3870) at src/quic_tx.c:719 ret = status = 0 buf = 0x766793f84bb8 qel = tmp_qel = #16 0x00005d932b91a5d8 in quic_conn_app_io_cb (t=t@entry=0x76679402da00, context=context@entry=0x766793f84800, state=) at src/quic_conn.c:598 send_list = { n = 0x7667963cedd0, p = 0x7667963cedd0 } qc = #17 0x00005d932baddc92 in run_tasks_from_lists (budgets=budgets@entry=0x766797ff39c0) at src/task.c:596 _ = { func = 0x5d932be0e439 "run_tasks_from_lists", file = 0x5d932be0e44e "src/task.c", line = 657, what = 6 '\006', arg8 = 0 '\000', arg32 = 0 } tl_queues = 0x5d932c1e41d0 budget_mask = 15 '\017' profile_entry = 0x0 done = 4 queue = 2 t = 0x76679402da00 process = 0x5d932b91a3b0 ctx = 0x766793f84800 state = 77 #18 0x00005d932bade63a in process_runnable_tasks () at src/task.c:876 max = {0, 0, 88, 0} tt = 0x5d932c1e4100 default_weights = {64, 48, 16, 1} heavy_queued = 1 max_processed = 93 max_total = queue = 4 budget = 0 grq = lrq = gpicked = lpicked = t = tmp_list = #19 0x00005d932ba77cc4 in run_poll_loop () at src/haproxy.c:3075 _ = { func = 0x5d932bdf0d6b "run_poll_loop", file = 0x5d932bdf0d79 "src/haproxy.c", line = 3106, what = 1 '\001', arg8 = 0 '\000', arg32 = 0 } wake = next = #20 0x00005d932ba7c1e0 in run_thread_poll_loop (data=) at src/haproxy.c:3289 init_left = 0 init_mutex = { __data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = { __prev = 0x0, __next = 0x0 } }, __size = '\000' , __align = 0 } init_cond = { __data = { __wseq = { __value64 = 25, __value32 = { __low = 25, __high = 0 } }, __g1_start = { __value64 = 21, __value32 = { __low = 21, __high = 0 } }, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 8, __wrefs = 0, __g_signals = {0, 0} }, __size = "\031\000\000\000\000\000\000\000\025", '\000' , "\b", '\000' , __align = 25 } warn_fail = 0 warn_fail = 0 ptaf = ptif = ptdf = ptff = #21 0x00007667b0f12ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #22 0x00007667b0fa4850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. ```

Finally, it's worth mentionning that the performance work for traces has done wonders!

I could now feasibly run them permanently, as the CPU impact went from 600-800% to about 40% (compared to trace-less load).

Amazing job :+1:

haproxyFred commented 5 months ago

For the second traces, we have a crash when dumping frames values (with TRACE()).

haproxyFred commented 5 months ago

Unfortunately, I did not find any clue which could lead to the resolution. :disappointed: Perhaps a gdb dump of the connections could help: (gdb) p *qc where object is available (from qc_do_rm_hp() frame #5 for instance).

haproxyFred commented 5 months ago

I meant, where qc object is available...

Tristan971 commented 5 months ago

Here you go (from core-1)

(gdb) p *qc
$1 = {
  {
    fd = 305,
    flags = 100892696,
    err = {
      code = 0,
      app = 0
    },
    nb_pkt_for_cc = 1,
    nb_pkt_since_cc = 0,
    wait_event = {
      tasklet = 0x7b1cea26bd00,
      events = 0
    },
    subs = 0x7b1ceb1f3fb0,
    local_addr = {
      ss_family = 2,
      __ss_padding = "\001\273-\201\345\001", '\000' <repeats 111 times>,
      __ss_align = 0
    },
    peer_addr = {
      ss_family = 2,
      __ss_padding = "\243\357O_WI", '\000' <repeats 111 times>,
      __ss_align = 0
    },
    bytes = {
      prep = 3450484,
      tx = 3450484,
      rx = 61872
    },
    odcid = {
      data = "H\242\177\200]\022\024\277", '\000' <repeats 11 times>,
      len = 8 '\b'
    },
    dcid = {
      data = '\000' <repeats 19 times>,
      len = 0 '\000'
    },
    scid = {
      data = "\377\027\347\204\266\374\352?\335w`\000\000\b\000\000\000\000\000",
      len = 8 '\b'
    },
    cids = 0x7b1ceb01e150,
    li = 0x7b1cf2424b00,
    idle_timer_task = 0x7b1cea122520,
    idle_expire = 882015,
    cntrs = {
      dropped_pkt = 0,
      dropped_pkt_bufoverrun = 0,
      dropped_parsing = 0,
      socket_full = 0,
      sendto_err = 0,
      sendto_err_unknown = 0,
      sent_pkt = 3448,
      lost_pkt = 0,
      conn_migration_done = 0,
      data_blocked = 0,
      stream_data_blocked = 0,
      streams_blocked_bidi = 0,
      streams_blocked_uni = 0
    },
    conn = 0x7b1ce88d3a00
  },
  xprt_ctx = 0x7b1ceb01cb60,
  original_version = 0x6077dd506248 <quic_versions+104>,
  negotiated_version = 0x0,
  nictx = 0x0,
  tps_tls_ext = 57,
  state = 7,
  mux_state = QC_MUX_READY,
  next_cid_seq_num = 2,
  hash64 = 15836526897143549771,
  iel = 0x0,
  eel = 0x7b1ceb1f0140,
  hel = 0x0,
  ael = 0x7b1ceb1f1b80,
  qel_list = {
    n = 0x7b1ceb1f0140,
    p = 0x7b1ceb1f1b80
  },
  ipktns = 0x0,
  hpktns = 0x0,
  apktns = 0x7b1cea1cc200,
  pktns_list = {
    n = 0x7b1cea1cc200,
    p = 0x7b1cea1cc200
  },
  tx = {
    params = {
      max_idle_timeout = 30000,
      max_udp_payload_size = 1472,
      initial_max_data = 15728640,
      initial_max_stream_data_bidi_local = 6291456,
      initial_max_stream_data_bidi_remote = 6291456,
      initial_max_stream_data_uni = 6291456,
      initial_max_streams_bidi = 100,
      initial_max_streams_uni = 103,
      ack_delay_exponent = 3,
      max_ack_delay = 25,
      active_connection_id_limit = 2,
      disable_active_migration = 0 '\000',
      with_stateless_reset_token = 0 '\000',
      with_preferred_address = 0 '\000',
      original_destination_connection_id_present = 0 '\000',
      initial_source_connection_id_present = 1 '\001',
      stateless_reset_token = '\000' <repeats 15 times>,
      original_destination_connection_id = {
        len = 0 '\000',
        data = '\000' <repeats 19 times>
      },
      retry_source_connection_id = {
        len = 0 '\000',
        data = '\000' <repeats 19 times>
      },
      initial_source_connection_id = {
        len = 0 '\000',
        data = '\000' <repeats 19 times>
      },
      preferred_address = {
        ipv4_port = 0,
        ipv6_port = 0,
        ipv4_addr = {
          s_addr = 0
        },
        ipv6_addr = {
          __in6_u = {
            __u6_addr8 = '\000' <repeats 15 times>,
            __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
            __u6_addr32 = {0, 0, 0, 0}
          }
        },
        cid = {
          len = 0 '\000',
          data = '\000' <repeats 19 times>
        },
        stateless_reset_token = '\000' <repeats 15 times>
      },
      version_information = {
        chosen = 0,
        negotiated_version = 0x0
      }
    },
    buf = {
      size = 0,
      area = 0x0,
      data = 0,
      head = 0
    },
    cc_buf = {
      size = 0,
      area = 0x0,
      data = 0,
      head = 0
    },
    cc_buf_area = 0x0,
    cc_dgram_len = 0
  },
  rx = {
    params = {
      max_idle_timeout = 30000,
      max_udp_payload_size = 2048,
      initial_max_data = 3374692,
      initial_max_stream_data_bidi_local = 32764,
      initial_max_stream_data_bidi_remote = 32764,
      initial_max_stream_data_uni = 32764,
      initial_max_streams_bidi = 100,
      initial_max_streams_uni = 3,
      ack_delay_exponent = 3,
      max_ack_delay = 25,
      active_connection_id_limit = 8,
      disable_active_migration = 1 '\001',
      with_stateless_reset_token = 1 '\001',
      with_preferred_address = 0 '\000',
      original_destination_connection_id_present = 0 '\000',
      initial_source_connection_id_present = 0 '\000',
      stateless_reset_token = "]\227\253\224\253\232g)\330\065\243{\347\260\377\347",
      original_destination_connection_id = {
        len = 8 '\b',
        data = "H\242\177\200]\022\024\277", '\000' <repeats 11 times>
      },
      retry_source_connection_id = {
        len = 0 '\000',
        data = '\000' <repeats 19 times>
      },
      initial_source_connection_id = {
        len = 8 '\b',
        data = "\377\027\347\204\266\374\352?", '\000' <repeats 11 times>
      },
      preferred_address = {
        ipv4_port = 0,
        ipv6_port = 0,
        ipv4_addr = {
          s_addr = 0
        },
        ipv6_addr = {
          __in6_u = {
            __u6_addr8 = '\000' <repeats 15 times>,
            __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
            __u6_addr32 = {0, 0, 0, 0}
          }
        },
        cid = {
          len = 0 '\000',
          data = '\000' <repeats 19 times>
        },
        stateless_reset_token = '\000' <repeats 15 times>
      },
      version_information = {
        chosen = 0,
        negotiated_version = 0x0
      }
    },
    buf = {
      size = 65536,
      area = 0x7b1ce463b8c0 "@\377\027\347\204\266\374\352?\a\002Mn#\003\a\003",
      data = 0,
      head = 0
    },
    pkt_list = {
      n = 0x7b1ceb014d28,
      p = 0x7b1ceb014d28
    },
    strms = {{
        nb_streams = 10
      }, {
        nb_streams = 0
      }, {
        nb_streams = 0
      }, {
        nb_streams = 1
      }}
  },
  ku = {
    prv_rx = {
      ctx = 0x0,
      secret = 0x7b1ceb171ac0 "",
      secretlen = 32,
      iv = 0x7b1cea25c230 "",
      ivlen = 12,
      key = 0x7b1ceb01f770 "",
      keylen = 32,
      count = 0,
      pn = 0,
      flags = 0 '\000'
    },
    nxt_rx = {
      ctx = 0x7b1cea028e80,
      secret = 0x7b1ceb171a00 "\365\017\252\315\327\247\222s\350\304\303Z\246\355\376\325Q-\222\372!D\365",
      secretlen = 32,
      iv = 0x7b1ceb01e060 "e\207>\237\v\006-\036\304eu|",
      ivlen = 12,
      key = 0x7b1ceb020a60 "\247\227\230",
      keylen = 32,
      count = 0,
      pn = 0,
      flags = 0 '\000'
    },
    nxt_tx = {
      ctx = 0x7b1cea027080,
      secret = 0x7b1ceb171940 "\334\005\003\213X\"W\333\223C6(\216\353E\223\314\355\243",
      secretlen = 32,
      iv = 0x7b1ceb020250 "\350\237\223\251\314\245P'\037\265\354\350",
      ivlen = 12,
      key = 0x7b1ceb01e4b0 "\257\341\220(\252\061\266\004\354\214\071O\310\nZ\026\002",
      keylen = 32,
      count = 0,
      pn = 0,
      flags = 0 '\000'
    }
  },
  max_ack_delay = 25,
  max_idle_timeout = 30000,
  paths = {{
      cc = {
        qc = 0x7b1ceb014800,
        algo = 0x6077dd54b0b0 <quic_cc_algo_cubic>,
        priv = {2, 3657, 2416, 852015, 4440, 1000, 4440, 4447, 1916, 0, 0, 0, 0, 0, 0, 0, 0, 0}
      },
      loss = {
        latest_rtt = 96,
        srtt = 69,
        rtt_var = 19,
        rtt_min = 28,
        pto_count = 0,
        nb_lost_pkt = 458,
        nb_reordered_pkt = 0
      },
      mtu = 1252,
      cwnd = 4064,
      mcwnd = 15648,
      max_cwnd = 4096000,
      min_cwnd = 2504,
      prep_in_flight = 4064,
      in_flight = 4064,
      ifae_pkts = 4
    }},
  path = 0x7b1ceb014e50,
  accept_list = {
    next = 0x7b1ceb014f18,
    prev = 0x7b1ceb014f18
  },
  streams_by_id = {
    b = {0x7b1ceb01a0e1, 0x1}
  },
  stream_buf_count = 3,
  qcc = 0x7b1ceb1f3e80,
  timer_task = 0x7b1cea152fe0,
  timer = 852185,
  ack_expire = 0,
  hs_expire = 830714,
  app_ops = 0x6077dd506120 <h3_ops>,
  prx_counters = 0x7b1cf23cb4f0,
  el_th_ctx = {
    n = 0x7b1cea1f7770,
    p = 0x7b1ce8073770
  },
  back_refs = {
    n = 0x7b1ceb014f80,
    p = 0x7b1ceb014f80
  },
  qc_epoch = 0
}
haproxyFred commented 5 months ago

No useful clue again. Perhaps the tls_ctx variable is not optimized into some frames. Please try into qc_do_rm_hp() to dump these variables:

p *tls_ctx
p pkt->len
p pn
p byte0

and *tls_ctx from other frames.

haproxyFred commented 5 months ago

If for an unknwon reason pkt is corrupted with pkt->len < pn - byte0, this test into qc_do_rm_hp()

if (pkt->len - (pn - byte0) < QUIC_PACKET_PN_MAXLEN + sizeof mask)

will not prevent qc_do_rm_hp() to call quic_tls_aes_decrypt().

I think we should add a BUG_ON(pkt->len < pn - byte0) at the beginning of qc_do_rm_hp().

Tristan971 commented 5 months ago

Here you go:

(gdb) frame 5
#5  0x00006077dcec084f in qc_do_rm_hp (qc=qc@entry=0x7b1ceb014800, pkt=pkt@entry=0x7b1cea075c80, tls_ctx=<optimized out>, largest_pn=1544, pn=0x7b1ce533d6c9 "\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b", byte0=byte0@entry=0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b") at src/quic_rx.c:94
94  src/quic_rx.c: No such file or directory.
(gdb) p *tls_ctx
value has been optimized out
(gdb) p pkt->len
$1 = 40
(gdb) p pn      
$2 = (unsigned char *) 0x7b1ce533d6c9 "\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b"
(gdb) p byte0
$3 = (unsigned char *) 0x7b1ce533d6c0 "\\\377\027\347\204\266\374\352?\341\065\263)\255\363\252B\354E\032\347m\030\305\253\206\006\001>y\315\257vv\210b"

Unfortunately the tls_ctx isn't available higher I think, since it comes from here https://github.com/haproxy/haproxy/blob/c714b6bb55e34c7cd2cb3ff7dbed374e6b6eae65/src/quic_rx.c#L1411-L1412

Maybe the best move forward is for me to mave a -O0 build to get the unoptimized stacktrace, if this doesn't help.

Mind making a patch for the bug_on so I add it into that build?

haproxyFred commented 5 months ago

At least the packet does not provide wrong values to qc_do_rm_hp(). So, the bug_on() is useless here.

Two remaining possibilities: tls_ctx is corrupted or there is an issue in the TLS stack. That said, the code we use here is very simple (quic_tls_aes_decrypt()). There are few chances this code is buggy (used since the begining of QUIC implementation).

Would it be possible to compile haproxy against libasan? I remember this could be painful.

Tristan971 commented 5 months ago

Pain, but yeah that’s probably the cause here for the odd 0x0 address…

I will try, maybe it won’t be as painful this time…

Tristan971 commented 5 months ago

Alright, it's not done yet, but there is 1 piece of good news: GCC seems to have finally fixed their previously-broken -static-libasan flag at least in 14.1.1 (it used to not result in static linkage, somehow, in the past).

I'm going to roll out a build on a single node and we should theoretically have feedback within the hour.

chipitsine commented 5 months ago

in case you'll fail with gcc/asan, I would suggest to try https://github.com/google/gwpsan (looks promising, however, I havent tried it yet)

Tristan971 commented 5 months ago

GWPSan is inspired by GWP-ASan, but their design and implementation are completely different

Couldn't help but chuckle (sadly) a bit... I'm already running the build and patiently waiting for a crash to occur, but noted as something to potentially explore in the future if I feel like raising my blood pressure.

Unrelated but once again I remain surprised at how minuscule the overhead of ASAN is (at least for us with HAProxy)... Especially as we run on pretty weak machines, with high virtualization overhead. If it wasn't such a monumental pain to build/run/symbolicate, I'd really just run it 24/7...

Tristan971 commented 5 months ago

Well, bad news... I'm not getting a crash with asan and -O1... Will let it run a few more hours, but the 2 test instances have yet to crash after 4 hours, when they used to crash every 20-40 minutes before...

haproxyFred commented 5 months ago

Yes this is sometimes difficult to make libasan effective. I did not manage to get haproxy+libasan effective with GCC. I use clang+libasan. When I have doubt, I check that libsasan is able to detect such a silly bug:

$ git diff
diff --git a/src/haproxy.c b/src/haproxy.c
index c987fdbfa..721501498 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -3352,6 +3352,7 @@ int main(int argc, char **argv)
        int pidfd = -1;
        int intovf = (unsigned char)argc + 1; /* let the compiler know it's strictly positive */

+       *(char *)0 = 0;
        /* Catch broken toolchains */
        if (sizeof(long) != sizeof(void *) || (intovf + 0x7FFFFFFF >= intovf)) {
                const char *msg;

It is also often too difficult to obtain a coredump with libasan. I have to set ASAN_OPTIONS variable to do so:

$ ASAN_OPTIONS=disable_coredump=0:unmap_shadow_on_exit=1:abort_on_error=1 ./haproxy -f quic.cfg

But all this is true on my PC. Does not mean this could be the case in your environment.

haproxyFred commented 5 months ago

I think you should set -O2 option again.

About the gdb dump for the qc object, the tls_ctx crypto context with weird values (0x1 then optimized) in qc_do_rm_hp() is inside qc->aelor qc->eeland is reachable with one of these commands:

p *qc->ael
p *qc->eel

Should be great to get their contents.

As a reminder, if qc->eel is not NULL, this is because this connection is a 0-RTT one.

Tristan971 commented 5 months ago

I did not manage to get haproxy+libasan effective with GCC

oh I’m using clang anyway; that was merely a good surprise to see that gcc’s flag is not hopelessly broken anymore it seems 😄

I think you should set -O2 option again.

Yes I was thinking the same, maybe it’s similar to that old SIGFPE case we had where O2 caused rax=0 somewhere which apparently was very rare

is reachable with one of these commands

ok I will look and report back

chipitsine commented 5 months ago

Well, bad news... I'm not getting a crash with asan and -O1... Will let it run a few more hours, but the 2 test instances have yet to crash after 4 hours, when they used to crash every 20-40 minutes before...

that depends on where you expect a crash to appear.

ASAN produces trace in stderr, but does not provide core dump (unless allowed by ASAN_OPTIONS).

Tristan971 commented 5 months ago

that depends on where you expect a crash to appear

I mean a process crash in any form, here. Here’s the process uptime history of HAProxy on my 2 test instances IMG_4933

haproxyFred commented 5 months ago

aww I have missed this gdb dump please: p pkt->type from qc_rx_pkt_handle()frame (7) and lower...

chipitsine commented 5 months ago

I did not manage to get haproxy+libasan effective with GCC

oh I’m using clang anyway; that was merely a good surprise to see that gcc’s flag is not hopelessly broken anymore it seems 😄

I think you should set -O2 option again.

Yes I was thinking the same, maybe it’s similar to that old SIGFPE case we had where O2 caused rax=0 somewhere which apparently was very rare

is reachable with one of these commands

ok I will look and report back

can you please share "haproxy -vv" output ? recently CFLAGS semantic has changed a bit, if you specify DEBUG_CFLAGS, it will be ignored

Tristan971 commented 5 months ago

can you please share "haproxy -vv" output ?

HAProxy version 3.1-dev1-937324d+mangadex- 2024-06-19T20:13+00:00 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 6.8.4-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-3 (2024-05-02T11:55Z) x86_64
Build options :
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O1 -g -ggdb3 -gdwarf-4 -static-libsan -fsanitize=address -fsanitize-address-use-after-scope -fno-omit-frame-pointer -fwrapv -DMAX_SESS_STKCTR=5
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_STATIC_PCRE2=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE +STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1w+quic-mangadex- 19 Jun 2024
Running on OpenSSL version : OpenSSL 1.1.1w+quic-mangadex- 19 Jun 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 18.1.6 (++20240518023138+1118c2e05e67-1~exp1~20240518143226.133) with address sanitizer

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
    [BWLIM] bwlim-in
    [BWLIM] bwlim-out
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace

I'll get you the extra coredump data in a minute, Fred

Tristan971 commented 5 months ago

And I run via systemd with:

$ cat /etc/default/haproxy
# Ansible managed
# Defaults file for HAProxy
#
# This is sourced by both, the initscript and the systemd unit file, so do not
# treat it as a shell script fragment.
CONFIG="/etc/haproxy/haproxy.cfg"
EXTRAOPTS="-dMno-merge,cold-first,caller"

# Extra envvars to ensure that ASAN crashes on error and generally properly coredumps etc
ASAN_OPTIONS="disable_coredump=0:unmap_shadow_on_exit=1:abort_on_error=1"

I do see asan output in stderr on process (re-)start so I believe everything is ok on the build/run end of things.

Just likely that the bug only happens in O2 at this point

chipitsine commented 5 months ago

And I run via systemd with:

$ cat /etc/default/haproxy
# Ansible managed
# Defaults file for HAProxy
#
# This is sourced by both, the initscript and the systemd unit file, so do not
# treat it as a shell script fragment.
CONFIG="/etc/haproxy/haproxy.cfg"
EXTRAOPTS="-dMno-merge,cold-first,caller"

# Extra envvars to ensure that ASAN crashes on error and generally properly coredumps etc
ASAN_OPTIONS="disable_coredump=0:unmap_shadow_on_exit=1:abort_on_error=1"

I do see asan output in stderr on process (re-)start so I believe everything is ok on the build/run end of things.

is there another bug triggered by restart ?

Just likely that the bug only happens in O2 at this point

Tristan971 commented 5 months ago

is there another bug triggered by restart ?

Interesting point. I can confirm that a reload of HAProxy causes the process to crash.

I get some ASAN output which I also can't manage to symbolize anymore, but it's useless minor leaks during config parsing.

systemd[1]: Reloading HAProxy Load Balancer...
haproxy[69316]: ==69316==WARNING: invalid path to external symbolizer!
haproxy[69316]: ==69316==WARNING: Failed to use and restart external symbolizer!
haproxy[69316]: =================================================================
haproxy[69316]: ==69316==ERROR: LeakSanitizer: detected memory leaks
haproxy[69316]: Direct leak of 24 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb34f30ba0  (/usr/sbin/haproxy+0x825ba0) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Direct leak of 16 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb34bb3c78  (/usr/sbin/haproxy+0x4a8c78) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb3508ea60  (/usr/sbin/haproxy+0x983a60) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34f2d039  (/usr/sbin/haproxy+0x822039) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Direct leak of 4 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a1731f  (/usr/sbin/haproxy+0x30c31f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb34da29b7  (/usr/sbin/haproxy+0x6979b7) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb350d4e3d  (/usr/sbin/haproxy+0x9c9e3d) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34db77ec  (/usr/sbin/haproxy+0x6ac7ec) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34db816e  (/usr/sbin/haproxy+0x6ad16e) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34f9b491  (/usr/sbin/haproxy+0x890491) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x62cb3508ea60  (/usr/sbin/haproxy+0x983a60) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #7 0x62cb34f2d039  (/usr/sbin/haproxy+0x822039) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #8 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #9 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #10 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Indirect leak of 240 byte(s) in 3 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb3500b41f  (/usr/sbin/haproxy+0x90041f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb34db7894  (/usr/sbin/haproxy+0x6ac894) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34db816e  (/usr/sbin/haproxy+0x6ad16e) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34f9b491  (/usr/sbin/haproxy+0x890491) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb3508ea60  (/usr/sbin/haproxy+0x983a60) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x62cb34f2d039  (/usr/sbin/haproxy+0x822039) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #7 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #8 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #9 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Indirect leak of 96 byte(s) in 3 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb34f1888e  (/usr/sbin/haproxy+0x80d88e) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb34f19c60  (/usr/sbin/haproxy+0x80ec60) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb3500b71d  (/usr/sbin/haproxy+0x90071d) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34db7894  (/usr/sbin/haproxy+0x6ac894) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34db816e  (/usr/sbin/haproxy+0x6ad16e) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x62cb34f9b491  (/usr/sbin/haproxy+0x890491) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #7 0x62cb3508ea60  (/usr/sbin/haproxy+0x983a60) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #8 0x62cb34f2d039  (/usr/sbin/haproxy+0x822039) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #9 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #10 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #11 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb3506d066  (/usr/sbin/haproxy+0x962066) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb3506e37f  (/usr/sbin/haproxy+0x96337f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34f30b2a  (/usr/sbin/haproxy+0x825b2a) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Indirect leak of 32 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb3506d56b  (/usr/sbin/haproxy+0x96256b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb3506e37f  (/usr/sbin/haproxy+0x96337f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34f30b2a  (/usr/sbin/haproxy+0x825b2a) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Indirect leak of 32 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb3506d4ac  (/usr/sbin/haproxy+0x9624ac) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb3506e37f  (/usr/sbin/haproxy+0x96337f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34f30b2a  (/usr/sbin/haproxy+0x825b2a) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: Indirect leak of 8 byte(s) in 1 object(s) allocated from:
haproxy[69316]:     #0 0x62cb34a174e9  (/usr/sbin/haproxy+0x30c4e9) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #1 0x62cb34bb3cde  (/usr/sbin/haproxy+0x4a8cde) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #2 0x62cb3508ea60  (/usr/sbin/haproxy+0x983a60) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #3 0x62cb34f2d039  (/usr/sbin/haproxy+0x822039) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #4 0x62cb34df849f  (/usr/sbin/haproxy+0x6ed49f) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #5 0x62cb34ee3e4b  (/usr/sbin/haproxy+0x7d8e4b) (BuildId: 0584cfe2f1b8d93e3a646a839a1163f83964f901)
haproxy[69316]:     #6 0x765512573d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
haproxy[69316]: SUMMARY: AddressSanitizer: 516 byte(s) leaked in 13 allocation(s).
systemd[1]: haproxy.service: Control process exited, code=dumped, status=6/ABRT
chipitsine commented 5 months ago

that indeed proves asan is in operation ))

chipitsine commented 5 months ago

btw, do you use "-ggdb" ? as far as I understand it should keep symbols embedded into ELF (if not stripped by "install"). can you share output of 

file `which haproxy`

leak detection might be supressed by ASAN_OPTIONS=detect_leaks=1

a-denoyelle commented 5 months ago

@Tristan971 maybe it would be simpler if you could share with us the coredump in private, along with the binary, loaded libs and debug symbols. Do not hesitate if you need a refresh I can reforward you a procedure for this.

Tristan971 commented 5 months ago

btw, do you use "-ggdb" ? as far as I understand it should keep symbols embedded into ELF (if not stripped by "install").

Yeah, the problem that I lack knowledge of proper deb package building, and the process strips debug symbols by default, and screams loudly when it's not happy about anything (like unstripped symbols)...

root@8b01b399ce6e:/# file $(which haproxy)
/usr/sbin/haproxy: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=fd957288a97517fc884793d059eae967b1672543, for GNU/Linux 3.2.0, stripped

maybe it would be simpler if you could share with us the coredump in private

Yeah as I was trying to organize the very noisy gdb output I thought the same. I'll send it over to you both in a minute like the other times.

Tristan971 commented 5 months ago

Fyi I sent the coredump + necessary libs to both @haproxyFred and @a-denoyelle

As followup to the debug symbols thing btw @chipitsine, while my debian packaging setup is a bit janky, it's so annoyingly sensitive and complains so easily that I mainly mirror the work done by Vincent for the official debian package... I wish debhelper and friends weren't so awful to use, but alas they're clearly designed to fit debian's own needs (reproduceability, src packages, etc etc), and the debian maintainers must have a lot of courage, so I kind of gave up on making it simpler...

chipitsine commented 5 months ago

Fyi I sent the coredump + necessary libs to both @haproxyFred and @a-denoyelle

As followup to the debug symbols thing btw @chipitsine, while my debian packaging setup is a bit janky, it's so annoyingly sensitive and complains so easily that I mainly mirror the work done by Vincent for the official debian package... I wish debhelper and friends weren't so awful to use, but alas they're clearly designed to fit debian's own needs (reproduceability, src packages, etc etc), and the debian maintainers must have a lot of courage, so I kind of gave up on making it simpler...

as far as I recall debian package symbols into dedicated deb package: https://wiki.debian.org/DebugPackage (which should be visible by symbolizer)

alternatively, I would rebuilt a binary manually and replace binary installed by package ))

Tristan971 commented 5 months ago

as far as I recall debian package symbols into dedicated deb package

Correct, and I install it, but asan symbolication seems to rely on also having the same toolchain (ie gcc/clang/…) at build and runtime to symbolicate at runtime…

Either way, hopefully Amaury and Fred find something in the coredump, saving us a lot of trouble in the process 😅

chipitsine commented 5 months ago

I played with symbolisation 5 yrs ago. As far as I know it all is kept in ELF metadata, most probably asan has no idea how to extract it

Anyway, I do not have a 5 sec solution now((

On Thu, Jun 20, 2024, 12:01 Tristan @.***> wrote:

as far as I recall debian package symbols into dedicated deb package

Correct, and I install it, but asan symbolication seems to rely on also having the same toolchain (ie gcc/clang/…) at build and runtime to symbolicate… so…

Either way, hopefully Amaury and Fred find something in the coredump, saving us a lot of trouble in the process 😅

— Reply to this email directly, view it on GitHub https://github.com/haproxy/haproxy/issues/2606#issuecomment-2180293113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ5KUAQ3JQ6GACEITFHAGDZIKSBFAVCNFSM6AAAAABJMQZZN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBQGI4TGMJRGM . You are receiving this because you were mentioned.Message ID: @.***>

wtarreau commented 5 months ago

No, it's a bit more complicated. The symbols are in dedicated sections of the ELF file that are not kept in memory when the executable is loaded. So gdb can find them since it has access to the whole executable but the process itself does not. Given that asan is a shared library that extends/wraps existing code, it runs inside the process and doesn't have access to these symbols either. For this, what it usually does is to execute the external program "addr2line" to resolve the symbols it finds (this program reads haproxy's ELF sections from disk for this). But this doesn't work under a chroot, nor by default given that we prevent haproxy from forking new processes. "insecure-fork-wanted" is necessary for that one to work, and chroot needs to be disabled. That's a lot of options and they weaken the process' security on exposed machines, which is why we prefer to avoid doing it permanently. It's often acceptable for a short test however, given that they're defense-in-depth mechanisms.

a-denoyelle commented 5 months ago

@Tristan971 what is the current status on your side ? Still running with ASAN but with no crash ?

On our side, we did not find anything in the coredump which could explain the bug. I wrote a simple program which try to replicate the identical OpenSSL instruction with the same input extracted for the coredump. Can you try to compile it against your QuicTLS and report me its output please ? Maybe the issue is revealed by some of your compilation flag for the SSL library.

main.c

a-denoyelle commented 5 months ago

Another solution would be to recompile QuicTLS using the extra flag --debug for the ./config script. This would allow to keep debug symbol for another core dump analysis.

Tristan971 commented 5 months ago

what is the current status on your side ? Still running with ASAN but with no crash ?

I ran it with asan for 2 days, and got no crash unfortunately

Can you try to compile it against your QuicTLS and report me its output please ?

Here's the build/output of your sample file:

root@6711b724f156:/build# cc -c -I/opt/quictls/include -o main.o main.c
main.c:30:42: warning: passing 'char[16]' to parameter of type 'const unsigned char *' converts between pointers to integer types where one is of the unique plain 'char' type and the other is not [-Wpointer-sign]
   30 |         if (!EVP_DecryptInit_ex(ctx, aes, NULL, key, NULL))
      |                                                 ^~~
/opt/quictls/include/openssl/evp.h:597:56: note: passing argument to parameter 'key' here
  597 |                                   const unsigned char *key,
      |                                                        ^
main.c:33:49: warning: passing 'char[16]' to parameter of type 'const unsigned char *' converts between pointers to integer types where one is of the unique plain 'char' type and the other is not [-Wpointer-sign]
   33 |         if (!EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, in))
      |                                                        ^~
/opt/quictls/include/openssl/evp.h:598:56: note: passing argument to parameter 'iv' here
  598 |                                   const unsigned char *iv);
      |                                                        ^
2 warnings generated.
root@6711b724f156:/build# cc main.o -L/opt/quictls/lib -lssl -lcrypto -o main
root@6711b724f156:/build# ./main 
0xfa 0x02 0x9b 0xe6 0x53 
done pnlen=1 trunc_pn=18

Maybe the issue is revealed by some of your compilation flag for the SSL library.

I configure it with:

./config --prefix="/opt/quictls" --openssldir="/opt/quictls" --libdir="lib" -DPURIFY no-shared

Another solution would be to recompile QuicTLS using the extra flag --debug for the ./config script. This would allow to keep debug symbol for another core dump analysis.

Ah. If only I'd known before. I will add the flag and get another coredump with them.

a-denoyelle commented 5 months ago

Thanks for the test. Output is as expected, so no issue with your compilation.

Tristan971 commented 5 months ago

Most annoyingly, I don't see the crash anymore with the debug build of QuicTLS alongside ad946a7...

Tomorrow I will make a normal build again and see if something happens, but it sounds awfully much like one of these cases where something is not correct, but depends on the exact build process somehow (like a long time ago with the SIGFPE that happened only in O2 and not O0 because it relied on rax being 0 or something...)

wtarreau commented 5 months ago

Hi Tristan,

Amaury provided me with a disassembled version of the crash area, and it's shocking:

   0x00006064b1ccf760 <+48>:    lea    0x4(%rsp),%rdx
   0x00006064b1ccf765 <+53>:    mov    %r14,%rdi
   0x00006064b1ccf768 <+56>:    mov    %r15,%rsi
   0x00006064b1ccf76b <+59>:    mov    %r15,%rcx
   0x00006064b1ccf76e <+62>:    mov    %r12d,%r8d
   0x00006064b1ccf771 <+65>:    call   0x6064b1f69eb0 <EVP_DecryptUpdate>
=> 0x00006064b1ccf776 <+70>:    test   %eax,%eax
   0x00006064b1ccf778 <+72>:    je     0x6064b1ccf791 <quic_tls_aes_decrypt+97>
   0x00006064b1ccf77a <+74>:    lea    0x4(%rsp),%rdx
   0x00006064b1ccf77f <+79>:    mov    %r14,%rdi
   0x00006064b1ccf782 <+82>:    mov    %r15,%rsi
   0x00006064b1ccf785 <+85>:    call   0x6064b1f6a2d0 <EVP_DecryptFinal_ex>
   0x00006064b1ccf78a <+90>:    xor    %ebx,%ebx

I.e. it's not even during a variable being dereferenced, it's during a call to an existing function. The only possibility I'm envisioning for this is either this code part having been unmapped from memory (which does really not make any sense given that this cannot happen by accident and would have had to be explicitly attempted), or a hardware issue (e.g. memory corruption). In this case this ought to be extremely rare (ideally never happen) and would tend to indicate that other (different) crashes did indeed originate from different causes.

Tristan971 commented 5 months ago

I fully believe you that it's weird, however I can guarantee that whatever it is, it is not a one off. It happened across multiple (5 or 6, iirc) HAProxy instances, and repeatedly.

But I understand that it's near-impossible to investigate at the moment. I'll do that new build today and report back.

wtarreau commented 5 months ago

OK, but what I'm suspecting is that there were several crashes of the same type and maybe only one exactly like this one. Of course, if you've found exactly this one on another machine, I'll completely rule out the tiny possibility of hardware issues. Thanks again for your tests by the way ;-)

Tristan971 commented 5 months ago

Oh. I see what you mean now. I'll look around and see if I can find another core of it laying around.