search

haproxy / haproxy

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
https://git.haproxy.org/
Other
4.89k stars 790 forks source link

Assertion failure in qc_do_build_pkt #2140

Closed Tristan971 closed 1 year ago

Tristan971 commented 1 year ago

Detailed Description of the Problem

HAProxy crashed on a BUG_ON (I'm so sorry... 🥲)

Expected Behavior

no crash

Steps to Reproduce the Behavior

No idea besides serving QUIC traffic

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

as usual

Output of haproxy -vv

HAProxy version 2.8-dev8-7b516d3+mangadex-da8dbb4 2023-04-26T16:35+00:00 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.85-1-pve #1 SMP PVE 5.15.85-1 (2023-02-01T00:00Z) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -ggdb3 -gdwarf-4 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -DMAX_SESS_STKCTR=5
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_STATIC_PCRE2=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE +STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1t+quic-mangadex-da8dbb4 26 Apr 2023
Running on OpenSSL version : OpenSSL 1.1.1t+quic-mangadex-da8dbb4 26 Apr 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 16.0.3 (++20230420052950+464bda7750a3-1~exp1~20230420173056.78)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

FATAL: bug condition "eb_is_empty(&qel->pktns->rx.arngs.root)" matched at src/quic_conn.c:7858
   call trace(1):
   | 0x7f4fda3f7608 [6d 6c 3e 0a 62 66 30 70]: main+0x293de7a50b58
Core was generated by `/usr/sbin/haproxy -sf 467044 -x sockpair@5 -Ws -f /etc/haproxy/haproxy.cfg -p /'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00005611f285bf9c in qc_do_build_pkt (pos=0x7f4fd8eaed2b "\363\021V", end=0x7f4fd8eaf1fe "-ce41-4c63-9370-88c263b4a2bb\r\n\r\nc-979194f8af05&chapter[]=e81a1853-a50b-46a1-9f39-e293c0d714d6&chapter[]=82e0ad48-d6c2-4eaa-a06c-651560950afa&chapter[]=bde3b7c4-d99c-4198-bff3-e4fefc308173&chapter[]=c2cca9d7-f169-434c-971b-be1825a03185&chapter[]=0622715b-35aa-44cd-b0f2-b1a9e47ff2d4 HTTP/1.1\r\nsec-fetch-site: same-site\r\naccept: */*\r\norigin: https://mangadex.org\r\nsec-fetch-dest: empty\r\naccept-language: en-GB,en-US;q=0.9,en;q=0.8\r\nsec-fetch-mode: cors\r\nuser-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_4_1 like M"..., dglen=<optimized out>, pkt=0x7f4fd851a140, pn=148, padding=<optimized out>, probe=<optimized out>, qel=0x7f4fd97ea998, qc=0x7f4fd97ea360, ver=<optimized out>, frms=0x7f4fea750080, pn_len=<optimized out>, buf_pn=<optimized out>, must_ack=<optimized out>, cc=<optimized out>) at src/quic_conn.c:7858
7858    src/quic_conn.c: No such file or directory.
[Current thread is 1 (Thread 0x7f4fea75b640 (LWP 467494))]
(gdb) t a a bt full

Thread 8 (Thread 0x7f4febfff640 (LWP 467491)):
#0  0x00007f4ff3b9efde in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005611f281a7b7 in _do_poll (p=<optimized out>, exp=136431149, wake=0) at src/ev_epoll.c:232
        timeout = 40
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 40
        status = <optimized out>
        count = <optimized out>
#2  0x00005611f29a658c in run_poll_loop () at src/haproxy.c:3029
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -603726864
#3  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f4ff080e640 (LWP 467490)):
#0  0x00007f4ff3b9efde in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005611f281a7b7 in _do_poll (p=<optimized out>, exp=136431149, wake=0) at src/ev_epoll.c:232
        timeout = 42
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 42
        status = <optimized out>
        count = <optimized out>
#2  0x00005611f29a658c in run_poll_loop () at src/haproxy.c:3029
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -469509136
#3  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f4fe9f5a640 (LWP 467495)):
#0  0x00007f4ff3b9efde in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005611f281a7b7 in _do_poll (p=<optimized out>, exp=136431124, wake=0) at src/ev_epoll.c:232
        timeout = 25
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 25
        status = <optimized out>
        count = <optimized out>
#2  0x00005611f29a658c in run_poll_loop () at src/haproxy.c:3029
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -872162320
#3  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f4feaf5c640 (LWP 467493)):
#0  0x00007f4ff3b9efde in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005611f281a7b7 in _do_poll (p=<optimized out>, exp=136431149, wake=0) at src/ev_epoll.c:232
        timeout = 43
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 43
        status = <optimized out>
        count = <optimized out>
#2  0x00005611f29a658c in run_poll_loop () at src/haproxy.c:3029
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -737944592
#3  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f4feb75d640 (LWP 467492)):
#0  0x00007f4ff3b9efde in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005611f281a7b7 in _do_poll (p=<optimized out>, exp=136431110, wake=0) at src/ev_epoll.c:232
        timeout = 2
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 2
        status = <optimized out>
        count = <optimized out>
#2  0x00005611f29a658c in run_poll_loop () at src/haproxy.c:3029
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -536618000
#3  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f4ff37de640 (LWP 467489)):
#0  0x00007f4ff3b9efde in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00005611f281a7b7 in _do_poll (p=<optimized out>, exp=136431129, wake=0) at src/ev_epoll.c:232
        timeout = 20
        updt_idx = <optimized out>
        fd = <optimized out>
        old_fd = <optimized out>
        wait_time = 20
        status = <optimized out>
        count = <optimized out>
#2  0x00005611f29a658c in run_poll_loop () at src/haproxy.c:3029
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = -335287216
#3  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#4  0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f4ff37ea1c0 (LWP 467488)):
#0  0x00005611f2887ad4 in quic_cc_event (cc=0x5611f48c13c8, ev=0x7fffb08e3200) at src/quic_cc.c:43
No locals.
#1  0x00005611f2846d92 in qc_treat_newly_acked_pkts (qc=0x5611f48c05b0, newly_acked_pkts=0x7fffb08e31c0) at src/quic_conn.c:2094
        ev = {
          type = QUIC_CC_EVT_ACK,
          {
            ack = {
              acked = 1252,
              time_sent = 136431067
            },
            loss = {
              time_sent = 1252
            }
          }
        }
        pkt = 0x5611f3f2de20
        tmp = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __ret = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#2  qc_parse_ack_frm (qc=0x5611f48c05b0, frm=0x7fffb08e32e8, qel=0x5611f48c0be8, pos=0x7fffb08e3158, end=0x5611f5003201 "\021\342VT\327\210\034}\273\026\203\243\313^\230\272\372\002\037I2\265\277\211vC\216e\227\030\002`\271\004\002\003\003@B", rtt_sample=<optimized out>) at src/quic_conn.c:2307
        newly_acked_pkts = {
          n = 0x5611f3f2de20,
          p = 0x5611f3c5f320
        }
        lost_pkts = {
          n = 0x7fffb08e3398,
          p = 0x7fffb08e3398
        }
        ret = 0
        largest = <optimized out>
        smallest = <optimized out>
        pkts = 0x5611f48c0e80
        pkt_flags = <optimized out>
        largest_node = <optimized out>
        time_sent = <optimized out>
        ack_frm = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        gap = <optimized out>
        ack_range = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#3  qc_parse_pkt_frms (qc=qc@entry=0x5611f48c05b0, pkt=pkt@entry=0x5611f3eca5b0, qel=qel@entry=0x5611f48c0be8) at src/quic_conn.c:3139
        rtt_sample = 4294967295
        frm = {
          list = {
            n = 0x5611f4390d30,
            p = 0x0
          },
          pkt = 0x5611f4b86d20,
          type = 2 '\002',
          {
            padding = {
              len = 8705
            },
            ack = {
              largest_ack = 8705,
              ack_delay = 3,
              ack_range_num = 18446744073709551615,
              first_ack_range = 157
            },
            tx_ack = {
              ack_delay = 8705,
              arngs = 0x3
            },
            crypto = {
              offset = 8705,
              len = 3,
              qel = 0xffffffffffffffff,
              data = 0x9d <error: Cannot access memory at address 0x9d>
            },
            reset_stream = {
              id = 8705,
              app_error_code = 3,
              final_size = 18446744073709551615
            },
            stop_sending = {
              id = 8705,
              app_error_code = 3
            },
            new_token = {
              len = 8705,
              data = 0x3 <error: Cannot access memory at address 0x3>
            },
            stream = {
              id = 8705,
              stream = 0x3,
              buf = 0xffffffffffffffff,
              offset = {
                node = {
                  branches = {
                    b = {0x9d, 0x0}
                  },
                  node_p = 0x0,
                  leaf_p = 0x5611f453d210,
                  bit = 1,
                  pfx = 0
                },
                {<No data fields>},
                key = 94635085288385
              },
              len = 94635089513008,
              data = 0x5611f453d210 "\004\003",
              dup = 0 '\000'
            },
            max_data = {
              max_data = 8705
            },
            max_stream_data = {
              id = 8705,
              max_stream_data = 3
            },
            max_streams_bidi = {
              max_streams = 8705
            },
            max_streams_uni = {
              max_streams = 8705
            },
            data_blocked = {
              limit = 8705
            },
            stream_data_blocked = {
              id = 8705,
              limit = 3
            },
            streams_blocked_bidi = {
              limit = 8705
            },
            streams_blocked_uni = {
              limit = 8705
            },
            new_connection_id = {
              seq_num = 8705,
              retire_prior_to = 3,
              cid = {
                len = 255 '\377',
                data = 0x9d <error: Cannot access memory at address 0x9d>
              },
              stateless_reset_token = 0x0
            },
            retire_connection_id = {
              seq_num = 8705
            },
            path_challenge = {
              data = "\001\"\000\000\000\000\000"
            },
            path_challenge_response = {
              data = "\001\"\000\000\000\000\000"
            },
            connection_close = {
              error_code = 8705,
              frame_type = 3,
              reason_phrase_len = 18446744073709551615,
              reason_phrase = "\235", '\000' <repeats 23 times>, "\020\322S\364\021V\000\000\001\000\000\000\000\000\000\000\301S\244\362\021V\000\000\060\312\344\362\021V\000\000\020\322S\364\021V\000"
            },
            connection_close_app = {
              error_code = 8705,
              reason_phrase_len = 3,
              reason_phrase = "\377\377\377\377\377\377\377\377\235", '\000' <repeats 23 times>, "\020\322S\364\021V\000\000\001\000\000\000\000\000\000\000\301S\244\362\021V\000\000\060\312\344\362\021V\000"
            }
          },
          origin = 0x7fffb08e3520,
          reflist = {
            n = 0x25a5,
            p = 0x5611f3f62c00
          },
          ref = {
            n = 0x5611f3f62bd8,
            p = 0xb68
          },
          flags = 4092996616,
          loss_count = 22033
        }
        fast_retrans = 0
        ret = 0
        pos = 0x5611f5003201 "\021\342VT\327\210\034}\273\026\203\243\313^\230\272\372\002\037I2\265\277\211vC\216e\227\030\002`\271\004\002\003\003@B"
        end = 0x5611f5003201 "\021\342VT\327\210\034}\273\026\203\243\313^\230\272\372\002\037I2\265\277\211vC\216e\227\030\002`\271\004\002\003\003@B"
#4  0x00005611f2844252 in qc_treat_rx_pkts (qc=qc@entry=0x5611f48c05b0, cur_el=cur_el@entry=0x5611f48c0be8, next_el=next_el@entry=0x0) at src/quic_conn.c:4576
        pkt = 0x5611f3eca5b0
        ret = 0
        largest_pn = -1
        largest_pn_time_received = 0
        qel = 0x5611f48c0be8
        node = 0x5611f3eca650
        out = <optimized out>
#5  0x00005611f284ac86 in quic_conn_app_io_cb (t=t@entry=0x5611f3e4a6b0, context=context@entry=0x5611f48c05b0, state=<optimized out>) at src/quic_conn.c:5025
        qel = <optimized out>
        qc = <optimized out>
#6  0x00005611f29daa8d in run_tasks_from_lists (budgets=budgets@entry=0x7fffb08e38a0) at src/task.c:596
        _ = {
          func = 0x5611f2d0b6a7 "run_tasks_from_lists",
          file = 0x5611f2d0b612 "src/task.c",
          line = 658,
          what = 5 '\005',
          arg8 = 0 '\000',
          arg32 = 0
        }
        tl_queues = 0x5611f30f5f90 <ha_thread_ctx+144>
        budget_mask = 15 '\017'
        profile_entry = 0x0
        done = 1
        queue = 0
        t = 0x5611f3e4a6b0
        process = 0x5611f284abb0 <quic_conn_app_io_cb>
        ctx = 0x5611f48c05b0
        state = 4075266320
#7  0x00005611f29db3fa in process_runnable_tasks () at src/task.c:876
        max = {91, 0, 0, 0}
        tt = 0x5611f30f5f00 <ha_thread_ctx>
        default_weights = {64, 48, 16, 1}
        heavy_queued = 1
        max_processed = 93
        max_total = <optimized out>
        queue = 4
        budget = 0
        grq = <optimized out>
        lrq = <optimized out>
        gpicked = <optimized out>
        lpicked = <optimized out>
        t = <optimized out>
        tmp_list = <optimized out>
#8  0x00005611f29a65c2 in run_poll_loop () at src/haproxy.c:2954
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = <optimized out>
#9  0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#10 0x00005611f29a98f4 in main (argc=<optimized out>, argv=0x7fffb08e3db8) at src/haproxy.c:3808
        limit = {
          rlim_cur = 18446744073709551615,
          rlim_max = 18446744073709551615
        }
        pidfd = 65
        retry = <optimized out>
        err = <optimized out>
        intovf = <optimized out>

Thread 1 (Thread 0x7f4fea75b640 (LWP 467494)):
#0  0x00005611f285bf9c in qc_do_build_pkt (pos=0x7f4fd8eaed2b "\363\021V", end=0x7f4fd8eaf1fe "-ce41-4c63-9370-88c263b4a2bb\r\n\r\nc-979194f8af05&chapter[]=e81a1853-a50b-46a1-9f39-e293c0d714d6&chapter[]=82e0ad48-d6c2-4eaa-a06c-651560950afa&chapter[]=bde3b7c4-d99c-4198-bff3-e4fefc308173&chapter[]=c2cca9d7-f169-434c-971b-be1825a03185&chapter[]=0622715b-35aa-44cd-b0f2-b1a9e47ff2d4 HTTP/1.1\r\nsec-fetch-site: same-site\r\naccept: */*\r\norigin: https://mangadex.org\r\nsec-fetch-dest: empty\r\naccept-language: en-GB,en-US;q=0.9,en;q=0.8\r\nsec-fetch-mode: cors\r\nuser-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_4_1 like M"..., dglen=<optimized out>, pkt=0x7f4fd851a140, pn=148, padding=<optimized out>, probe=<optimized out>, qel=0x7f4fd97ea998, qc=0x7f4fd97ea360, ver=<optimized out>, frms=0x7f4fea750080, pn_len=<optimized out>, buf_pn=<optimized out>, must_ack=<optimized out>, cc=<optimized out>) at src/quic_conn.c:7858
        arngs = <optimized out>
        frm = {
          list = {
            n = 0x0,
            p = 0x0
          },
          pkt = 0x0,
          type = 6 '\006',
          {
            padding = {
              len = 0
            },
            ack = {
              largest_ack = 0,
              ack_delay = 0,
              ack_range_num = 0,
              first_ack_range = 0
            },
            tx_ack = {
              ack_delay = 0,
              arngs = 0x0
            },
            crypto = {
              offset = 0,
              len = 0,
              qel = 0x0,
              data = 0x0
            },
            reset_stream = {
              id = 0,
              app_error_code = 0,
              final_size = 0
            },
            stop_sending = {
              id = 0,
              app_error_code = 0
            },
            new_token = {
              len = 0,
              data = 0x0
            },
            stream = {
              id = 0,
              stream = 0x0,
              buf = 0x0,
              offset = {
                node = {
                  branches = {
                    b = {0x0, 0x0}
                  },
                  node_p = 0x0,
                  leaf_p = 0x0,
                  bit = 0,
                  pfx = 0
                },
                {<No data fields>},
                key = 0
              },
              len = 0,
              data = 0x0,
              dup = 0 '\000'
            },
            max_data = {
              max_data = 0
            },
            max_stream_data = {
              id = 0,
              max_stream_data = 0
            },
            max_streams_bidi = {
              max_streams = 0
            },
            max_streams_uni = {
              max_streams = 0
            },
            data_blocked = {
              limit = 0
            },
            stream_data_blocked = {
              id = 0,
              limit = 0
            },
            streams_blocked_bidi = {
              limit = 0
            },
            streams_blocked_uni = {
              limit = 0
            },
            new_connection_id = {
              seq_num = 0,
              retire_prior_to = 0,
              cid = {
                len = 0 '\000',
                data = 0x0
              },
              stateless_reset_token = 0x0
            },
            retire_connection_id = {
              seq_num = 0
            },
            path_challenge = {
              data = "\000\000\000\000\000\000\000"
            },
            path_challenge_response = {
              data = "\000\000\000\000\000\000\000"
            },
            connection_close = {
              error_code = 0,
              frame_type = 0,
              reason_phrase_len = 0,
              reason_phrase = '\000' <repeats 63 times>
            },
            connection_close_app = {
              error_code = 0,
              reason_phrase_len = 0,
              reason_phrase = '\000' <repeats 63 times>
            }
          },
          origin = 0x0,
          reflist = {
            n = 0x0,
            p = 0x0
          },
          ref = {
            n = 0x0,
            p = 0x0
          },
          flags = 0,
          loss_count = 0
        }
        ack_frm = {
          list = {
            n = 0x0,
            p = 0x0
          },
          pkt = 0x0,
          type = 2 '\002',
          {
            padding = {
              len = 0
            },
            ack = {
              largest_ack = 0,
              ack_delay = 0,
              ack_range_num = 0,
              first_ack_range = 0
            },
            tx_ack = {
              ack_delay = 0,
              arngs = 0x0
            },
            crypto = {
              offset = 0,
              len = 0,
              qel = 0x0,
              data = 0x0
            },
            reset_stream = {
              id = 0,
              app_error_code = 0,
              final_size = 0
            },
            stop_sending = {
              id = 0,
              app_error_code = 0
            },
            new_token = {
              len = 0,
              data = 0x0
            },
            stream = {
              id = 0,
              stream = 0x0,
              buf = 0x0,
              offset = {
                node = {
                  branches = {
                    b = {0x0, 0x0}
                  },
                  node_p = 0x0,
                  leaf_p = 0x0,
                  bit = 0,
                  pfx = 0
                },
                {<No data fields>},
                key = 0
              },
              len = 0,
              data = 0x0,
              dup = 0 '\000'
            },
            max_data = {
              max_data = 0
            },
            max_stream_data = {
              id = 0,
              max_stream_data = 0
            },
            max_streams_bidi = {
              max_streams = 0
            },
            max_streams_uni = {
              max_streams = 0
            },
            data_blocked = {
              limit = 0
            },
            stream_data_blocked = {
              id = 0,
              limit = 0
            },
            streams_blocked_bidi = {
              limit = 0
            },
            streams_blocked_uni = {
              limit = 0
            },
            new_connection_id = {
              seq_num = 0,
              retire_prior_to = 0,
              cid = {
                len = 0 '\000',
                data = 0x0
              },
              stateless_reset_token = 0x0
            },
            retire_connection_id = {
              seq_num = 0
            },
            path_challenge = {
              data = "\000\000\000\000\000\000\000"
            },
            path_challenge_response = {
              data = "\000\000\000\000\000\000\000"
            },
            connection_close = {
              error_code = 0,
              frame_type = 0,
              reason_phrase_len = 0,
              reason_phrase = '\000' <repeats 63 times>
            },
            connection_close_app = {
              error_code = 0,
              reason_phrase_len = 0,
              reason_phrase = '\000' <repeats 63 times>
            }
          },
          origin = 0x0,
          reflist = {
            n = 0x0,
            p = 0x0
          },
          ref = {
            n = 0x0,
            p = 0x0
          },
          flags = 0,
          loss_count = 0
        }
        cc_frm = {
          list = {
            n = 0x0,
            p = 0x0
          },
          pkt = 0x0,
          type = 0 '\000',
          {
            padding = {
              len = 0
            },
            ack = {
              largest_ack = 0,
              ack_delay = 0,
              ack_range_num = 0,
              first_ack_range = 0
            },
            tx_ack = {
              ack_delay = 0,
              arngs = 0x0
            },
            crypto = {
              offset = 0,
              len = 0,
              qel = 0x0,
              data = 0x0
            },
            reset_stream = {
              id = 0,
              app_error_code = 0,
              final_size = 0
            },
            stop_sending = {
              id = 0,
              app_error_code = 0
            },
            new_token = {
              len = 0,
              data = 0x0
            },
            stream = {
              id = 0,
              stream = 0x0,
              buf = 0x0,
              offset = {
                node = {
                  branches = {
                    b = {0x0, 0x0}
                  },
                  node_p = 0x0,
                  leaf_p = 0x0,
                  bit = 0,
                  pfx = 0
                },
                {<No data fields>},
                key = 0
              },
              len = 0,
              data = 0x0,
              dup = 0 '\000'
            },
            max_data = {
              max_data = 0
            },
            max_stream_data = {
              id = 0,
              max_stream_data = 0
            },
            max_streams_bidi = {
              max_streams = 0
            },
            max_streams_uni = {
              max_streams = 0
            },
            data_blocked = {
              limit = 0
            },
            stream_data_blocked = {
              id = 0,
              limit = 0
            },
            streams_blocked_bidi = {
              limit = 0
            },
            streams_blocked_uni = {
              limit = 0
            },
            new_connection_id = {
              seq_num = 0,
              retire_prior_to = 0,
              cid = {
                len = 0 '\000',
                data = 0x0
              },
              stateless_reset_token = 0x0
            },
            retire_connection_id = {
              seq_num = 0
            },
            path_challenge = {
              data = "\000\000\000\000\000\000\000"
            },
            path_challenge_response = {
              data = "\000\000\000\000\000\000\000"
            },
            connection_close = {
              error_code = 0,
              frame_type = 0,
              reason_phrase_len = 0,
              reason_phrase = '\000' <repeats 63 times>
            },
            connection_close_app = {
              error_code = 0,
              reason_phrase_len = 0,
              reason_phrase = '\000' <repeats 63 times>
            }
          },
          origin = 0x0,
          reflist = {
            n = 0x0,
            p = 0x0
          },
          ref = {
            n = 0x0,
            p = 0x0
          },
          flags = 0,
          loss_count = 0
        }
        frm_list = {
          n = 0x7f4fea74f880,
          p = 0x7f4fea74f880
        }
        ret = 0
        len_frms = 0
        beg = <optimized out>
        ack_frm_len = 0
        len = <optimized out>
        add_ping_frm = <optimized out>
        padding_len = <optimized out>
        len_sz = <optimized out>
        cf = <optimized out>
        payload = <optimized out>
        head_len = <optimized out>
        rx_largest_acked_pn = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        path_room = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        arngs = <optimized out>
        room = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        tmp_cf = <optimized out>
        room = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __ret = <optimized out>
        __n = <optimized out>
        __p = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#1  qc_build_pkt (pos=pos@entry=0x7f4fea74fde0, end=<optimized out>, qel=qel@entry=0x7f4fd97ea998, tls_ctx=tls_ctx@entry=0x7f4fd97ea9a0, frms=frms@entry=0x7f4fea750080, qc=qc@entry=0x7f4fd97ea360, ver=0x0, dglen=0, pkt_type=4, must_ack=0, padding=0, probe=0, cc=0, err=0x7f4fea74fde8) at src/quic_conn.c:8118
        ret_pkt = 0x0
        pkt = 0x7f4fd851a140
        first_byte = <optimized out>
        pn_len = 0
        buf_pn = 0x0
        pn = 148
        last_byte = <optimized out>
        payload = <optimized out>
        payload_len = <optimized out>
        aad_len = <optimized out>
#2  0x00005611f2849371 in qc_prep_app_pkts (qc=0x7f4fd97ea360, buf=0x0, frms=0x7f4fea750080) at src/quic_conn.c:3487
        err = 0
        probe = 83
        cc = <optimized out>
        must_ack = <optimized out>
        ret = -1
        dg_headlen = 10
        qel = 0x7f4fd97ea998
        total = 0
        pos = 0x7f4fd8eaed2a "@\363\021V"
        pkt = <optimized out>
        end = 0x7f4fea74f000 "  call trace(1):\n  | 0x7f4fda3f7608 [6d 6c 3e 0a 62 66 30 70]: main+0x293de7a50b58\n"
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        err = <optimized out>
        probe = <optimized out>
        cc = <optimized out>
        must_ack = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
        _a = <optimized out>
        _b = <optimized out>
        __x = <optimized out>
        __x = <optimized out>
#3  qc_send_app_pkts (qc=qc@entry=0x7f4fd97ea360, frms=frms@entry=0x7f4fea750080) at src/quic_conn.c:4749
        ret = <optimized out>
        status = 0
        buf = 0x7f4fd97eaef8
#4  0x00005611f284874c in qc_send_mux (qc=0x7f4fd97ea360, frms=frms@entry=0x7f4fea750080) at src/quic_conn.c:4818
        ret = <optimized out>
#5  0x00005611f2867a80 in qc_send_frames (qcc=qcc@entry=0x7f4fd877d020, frms=frms@entry=0x7f4fea750080) at src/mux_quic.c:1679
No locals.
#6  0x00005611f2866548 in qc_send (qcc=qcc@entry=0x7f4fd877d020) at src/mux_quic.c:1963
        _ = {
          func = 0x5611f2cbd312 "qc_send",
          file = 0x5611f2cbbfc3 "src/mux_quic.c",
          line = 2003,
          what = 3 '\003',
          arg8 = 0 '\000',
          arg32 = 0
        }
        frms = {
          n = 0x7f4fd80670d0,
          p = 0x7f4fd9612250
        }
        qcs_failed = {
          n = 0x7f4fea750040,
          p = 0x7f4fea750040
        }
        first_qcs = <optimized out>
        total = 0
        qcs = <optimized out>
        qcs_tmp = <optimized out>
        ret = <optimized out>
#7  0x00005611f28657ce in qc_io_cb (t=<optimized out>, ctx=ctx@entry=0x7f4fd877d020, status=<optimized out>) at src/mux_quic.c:2250
        qcc = <optimized out>
#8  0x00005611f29daa8d in run_tasks_from_lists (budgets=budgets@entry=0x7f4fea7502d0) at src/task.c:596
        _ = {
          func = 0x5611f2d0b6a7 "run_tasks_from_lists",
          file = 0x5611f2d0b612 "src/task.c",
          line = 658,
          what = 5 '\005',
          arg8 = 0 '\000',
          arg32 = 0
        }
        tl_queues = 0x5611f30f6890 <ha_thread_ctx+2448>
        budget_mask = 15 '\017'
        profile_entry = 0x0
        done = 2
        queue = 0
        t = 0x7f4fd80709c0
        process = 0x5611f28657a0 <qc_io_cb>
        ctx = 0x7f4fd877d020
        state = 83
#9  0x00005611f29db3fa in process_runnable_tasks () at src/task.c:876
        max = {90, 0, 0, 0}
        tt = 0x5611f30f6800 <ha_thread_ctx+2304>
        default_weights = {64, 48, 16, 1}
        heavy_queued = 1
        max_processed = 93
        max_total = <optimized out>
        queue = 4
        budget = 0
        grq = <optimized out>
        lrq = <optimized out>
        gpicked = <optimized out>
        lpicked = <optimized out>
        t = <optimized out>
        tmp_list = <optimized out>
#10 0x00005611f29a65c2 in run_poll_loop () at src/haproxy.c:2954
        _ = {
          func = 0x5611f2d00ea7 "run_poll_loop",
          file = 0x5611f2d00eb5 "src/haproxy.c",
          line = 2985,
          what = 1 '\001',
          arg8 = 0 '\000',
          arg32 = 0
        }
        wake = <optimized out>
        next = <optimized out>
#11 0x00005611f29aa89f in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3153
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0,
            __count = 0,
            __owner = 0,
            __nusers = 0,
            __kind = 0,
            __spins = 0,
            __elision = 0,
            __list = {
              __prev = 0x0,
              __next = 0x0
            }
          },
          __size = '\000' <repeats 39 times>,
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 31,
              __wseq32 = {
                __low = 31,
                __high = 0
              }
            },
            {
              __g1_start = 17,
              __g1_start32 = {
                __low = 17,
                __high = 0
              }
            },
            __g_refs = {0, 0},
            __g_size = {0, 0},
            __g1_orig_size = 28,
            __wrefs = 0,
            __g_signals = {0, 0}
          },
          __size = "\037\000\000\000\000\000\000\000\021", '\000' <repeats 23 times>, "\034", '\000' <repeats 14 times>,
          __align = 31
        }
        warn_fail = 0
        warn_fail = 0
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
#12 0x00007f4ff3b0db43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#13 0x00007f4ff3b9fa00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

Additional Information

I don't have traces alas.

The pkt response buffer contains a portion of another HTTP request though, so it looks a lot like reading out of bounds

haproxyFred commented 1 year ago

Again a similar backtrace for https://github.com/haproxy/haproxy/issues/2120 with exactly the same value for buf : 0x0 and probe : 83 in qc_prep_app_pkts(). Could you provide us the coredump please?

Tristan971 commented 1 year ago

Sent it your way! 👍

The matching binaries and symbols are here:

haproxyFred commented 1 year ago

After analyzing the coredump, I think this time the BUG_ON() is right. I have to try to reproduce this issue, but not today.

haproxyFred commented 1 year ago

I have managed to reproduce this issue, acknowledging two times the same ACK frame sent in the same packet. Here is a fix: https://github.com/haproxytech/quic-dev/commit/00f448693a0af73b10f21bfb4f69c70cc96f2129

a-denoyelle commented 1 year ago

@haproxyFred thanks for your quick fix. @Tristan971 we plan to release a 2.7 really soon to fix the other issue. Do you have the time to test Fred fix so that we may integrate it on the new release ? If it's not possible, no worry, we will release the 2.7 with only the fix for #2141.

Tristan971 commented 1 year ago

oh wow nice one @haproxyFred!

@a-denoyelle even after merging the patch on my side I won’t be able to confirm that it fixes it, because I have never experienced that crash again :/ it’s many times rarer than the other one. I’ll let you know if it causes a new crash, but that’s all I could probably

based on that I’d just release 2.7 without it for now, since it’s absurdly rare anyway

a-denoyelle commented 1 year ago

Ok thanks for your feedback.

haproxyFred commented 1 year ago

All this does not explain why the stack seems corrupted (buf = 0x0, probe = 83). I guess the next time haproxy will crash building the first frame in such a state.

Tristan971 commented 1 year ago

As I said in my comment I can't confirm that it fixed the issue (since I didn't run into it again in a whole week without the patch), but at least the patch didn't induce any new crash in about 10 hours across our whole fleet, so if it makes sense from a logic standpoint it can probably be safely backported.

Tristan971 commented 1 year ago

Just chiming in after 36 hours; still have no crash with this patch so it is at least not making anything worse 👍

haproxyFred commented 1 year ago

All this does not explain why the stack seems corrupted (buf = 0x0, probe = 83). I guess the next time haproxy will crash building the first frame in such a state.

Wrong comment... In fact this had already been checked during a gdb debugging session.

Tristan971 commented 1 year ago

Have you decided to not backport the patch in the end or was it just forgotten for 2.8-dev11? cc @wtarreau (not that it matters a lot, since things were fine without it too afaict, but just in case it was indeed forgotten)

wtarreau commented 1 year ago

No, it's just that we found that this code is particularly complex and deserved at least a comment, and as you said it was hard to trigger it we preferred to wait for Fred to be back this week, and since then he's been busy. Maybe we can get it updated and merged today.

Tristan971 commented 1 year ago

Makes sense 👍

a-denoyelle commented 1 year ago

In fact Fred already updated its patch but I forgot to merge it. Sorry for this, and thanks for the reminder. @Tristan971 just to know, is it possible for you to test the master branch ? It would be useful to ensure we do not have introduce last minute regression with the coming release soon.

Tristan971 commented 1 year ago

just to know, is it possible for you to test the master branch ? It would be useful to ensure we do not have introduce last minute regression with the coming release soon.

Yeah sure; I'll try and do that sometime today

a-denoyelle commented 1 year ago

As always thank you very much :)

Tristan971 commented 1 year ago

is it possible for you to test the master branch ? It would be useful to ensure we do not have introduce last minute regression with the coming release soon.

Fwiw I've been running e279f59 for a week without any issue to report (besides #2147 but that one is not QUIC related and definitely not 2.8-exclusive either)

Either way, I just updated to ffdf6a3 now

Tristan971 commented 1 year ago

More or less leaves #2095 as the only known QUIC "issue" as far as I'm concerned, and I can't imagine that you'd want to block 2.8 final release on it anyway, as it will probably take many small steps over time to get through it.

Tristan971 commented 1 year ago

Well, ignore my comment here https://github.com/haproxy/haproxy/issues/2140#issuecomment-1554082998; turns out I'd been running 9de10ce (+ the patch) since and didn't realize... Must have forgotten to deploy... Now running ffdf6a3 (I double-checked...)

Tristan971 commented 1 year ago

That bug didn't trigger again in 6 days for me. Imo we can close the issue.

Tristan971 commented 1 year ago

never reproduced after months