nginx 1.25.3.1 http2 segfault with ja3 patch added

I cannot reproduce the error with curl, I can't see it in the logs(segfaults before the log) and there are like 4k rps on a single server, so debug logs are not really an option :( In the core I see the orignal request url with gdb. But thats about it. If I call the url directly there is no error.

Today I tested with and without ja3 patch. Same build process (official openresty builder). With ja3, there is a segfault in every 10 minutes, without it no segfault at all.

The core contains sensitive information so I cannot share it publicly,

built with OpenSSL 1.1.1w 11 Sep 2023 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.3 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.26 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.37 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.20 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.14 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib -Wl,-rpath,' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --add-dynamic-module=/tmp/modules/brotli --add-dynamic-module=/tmp/modules/geoip2 --add-module=/tmp/modules/nginx-ssl-fingerprint --with-stream --without-pcre2 --with-http_ssl_module

I know this is not much:

(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007eff33311220  0x00007eff33312179  Yes         /lib/x86_64-linux-gnu/libdl.so.2
0x00007eff332f3ae0  0x00007eff33303535  Yes         /lib/x86_64-linux-gnu/libpthread.so.0
0x00007eff332b4040  0x00007eff332c8c80  Yes (*)     /lib/x86_64-linux-gnu/libcrypt.so.1
0x00007eff3322e670  0x00007eff3329423d  Yes         /usr/local/openresty/luajit/lib/libluajit-5.1.so.2
0x00007eff331b62c0  0x00007eff33205e5d  Yes (*)     /usr/local/openresty/pcre/lib/libpcre.so.1
0x00007eff3313d750  0x00007eff3318974a  Yes (*)     /usr/local/openresty/openssl111/lib/libssl.so.1.1
0x00007eff32eca000  0x00007eff3305b260  Yes (*)     /usr/local/openresty/openssl111/lib/libcrypto.so.1.1
0x00007eff32e36280  0x00007eff32e4765b  Yes (*)     /usr/local/openresty/zlib/lib/libz.so.1
0x00007eff32c63630  0x00007eff32dd84bd  Yes         /lib/x86_64-linux-gnu/libc.so.6
0x00007eff3331f100  0x00007eff33341684  Yes         /lib64/ld-linux-x86-64.so.2
0x00007eff32aff3c0  0x00007eff32ba5fa8  Yes         /lib/x86_64-linux-gnu/libm.so.6
0x00007eff32ada5e0  0x00007eff32aeb055  Yes (*)     /lib/x86_64-linux-gnu/libgcc_s.so.1
0x00007eff32ac15c0  0x00007eff32ac7a1c  Yes         /lib/x86_64-linux-gnu/libnss_files.so.2
0x00007eff333173e0  0x00007eff3331864c  Yes         /usr/local/openresty/nginx/modules/ngx_http_geoip2_module.so
0x00007eff32ab2300  0x00007eff32ab4e5c  Yes (*)     /lib/x86_64-linux-gnu/libmaxminddb.so.0
0x00007eff271766e0  0x00007eff2717a0c1  Yes         /usr/local/openresty/lualib/cjson.so
0x00007eff270fefa0  0x00007eff27123300  Yes (*)     /lib/x86_64-linux-gnu/libnss_systemd.so.2
(*): Shared library is missing debugging information.

Core was generated by `nginx: worker process                                                 '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ngx_palloc_large (pool=0x55d85c102a00, size=<optimized out>) at src/core/ngx_palloc.c:228
228 src/core/ngx_palloc.c: No such file or directory.
(gdb) bt
#0  ngx_palloc_large (pool=0x55d85c102a00, size=<optimized out>) at src/core/ngx_palloc.c:228
#1  0x000055d85a2614ff in ngx_palloc (pool=<optimized out>, size=<optimized out>) at src/core/ngx_palloc.c:131
#2  0x000055d85a2d8a75 in ngx_http_v2_add_header (h2c=h2c@entry=0x55d85c18a500, header=header@entry=0x55d85c18a5a0) at src/http/v2/ngx_http_v2_table.c:212
#3  0x000055d85a2d6c4b in ngx_http_v2_state_process_header (h2c=0x55d85c18a500, 
    pos=0x55d85bf57d56 "A\217%\225ȕ\350\067r\364\032)5K\220\364\377S\260I|\245\211\323M\037C\256\272\fA\244ǩ\217\063\246\232?ߚh\372\035u\320b\r&=Ly\246\217\276\320\001w\376\276X\371\373", <incomplete sequence \355>, 
    end=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1778
#4  0x000055d85a2d736f in ngx_http_v2_state_field_huff (h2c=<optimized out>, pos=<optimized out>, end=<optimized out>) at src/http/v2/ngx_http_v2.c:1617
#5  0x000055d85a2d7608 in ngx_http_v2_state_field_len (h2c=h2c@entry=0x55d85c18a500, pos=<optimized out>, 
    end=end@entry=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1575
#6  0x000055d85a2d7886 in ngx_http_v2_state_header_block (h2c=0x55d85c18a500, pos=<optimized out>, 
    end=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1491
#7  0x000055d85a2d5145 in ngx_http_v2_read_handler (rev=0x7eff26bb8d70) at src/http/v2/ngx_http_v2.c:432
#8  0x000055d85a28a596 in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901
#9  0x000055d85a280978 in ngx_process_events_and_timers (cycle=cycle@entry=0x55d85bd276d0) at src/event/ngx_event.c:258
#10 0x000055d85a288690 in ngx_worker_process_cycle (cycle=cycle@entry=0x55d85bd276d0, data=data@entry=0x5) at src/os/unix/ngx_process_cycle.c:793
#11 0x000055d85a286f8d in ngx_spawn_process (cycle=cycle@entry=0x55d85bd276d0, proc=proc@entry=0x55d85a288610 <ngx_worker_process_cycle>, data=data@entry=0x5, name=name@entry=0x55d85a3a3ceb "worker process", respawn=respawn@entry=-3)
    at src/os/unix/ngx_process.c:199
#12 0x000055d85a288ba4 in ngx_start_worker_processes (cycle=cycle@entry=0x55d85bd276d0, n=8, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:382
#13 0x000055d85a289442 in ngx_master_process_cycle (cycle=0x55d85bd276d0) at src/os/unix/ngx_process_cycle.c:135
#14 0x000055d85a25ed7a in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:387

f 0
(gdb) info local
p = 0x55d85bfe8990
n = 2
large = 0x9f5d259b3eb91b2

(gdb) p *pool
$38 = {d = {last = 0x55d85c102bfc "\330U", end = 0x55d85c102c00 "\020\002", next = 0x55d85c002ef0, failed = 1}, max = 432, current = 0x55d85c102a00, chain = 0x0, large = 0x55d85c0030d8, cleanup = 0x55d85c002fa0, log = 0x55d85c102a60}

f 2
#2  0x000055d85a2d8a75 in ngx_http_v2_add_header (h2c=h2c@entry=0x55d85c18a500, header=header@entry=0x55d85c18a5a0) at src/http/v2/ngx_http_v2_table.c:212
$12 = {connection = 0x7eff26d3e180, http_connection = 0x55d85c102ac0, total_bytes = 130, payload_bytes = 0, processing = 1, frames = 0, idle = 3, new_streams = 1, refused_streams = 0, priority_limit = 256, send_window = 65535, 
  recv_window = 2147483647, init_window = 65535, frame_size = 16384, waiting = {prev = 0x55d85c18a570, next = 0x55d85c18a570}, state = {sid = 1, length = 282, padding = 0, flags = 5, incomplete = 0, keep_pool = 1, parse_name = 0, 
    parse_value = 0, index = 1, header = {name = {len = 5, data = 0x55d85a3abcbe ":path"}, value = {len = 165, 
        data = 0x55d85c0ea340 "***SENSITIVE INFO (url)***"}}, header_limit = 32576, 
    field_state = 0 '\000', field_start = 0x55d85c0ea340 "***SENSITIVE INFO (url)***", 
    field_end = 0x55d85c0ea3e5 "", field_rest = 0, pool = 0x55d85c0ea2f0, stream = 0x55d85bfc9ee0, buffer = '\000' <repeats 15 times>, buffer_used = 0, handler = 0x55d85a2d77e0 <ngx_http_v2_state_header_block>}, hpack = {
    entries = 0x55d85c204dc0, added = 0, deleted = 0, reused = 0, allocated = 64, size = 4096, free = 4096, storage = 0x0, pos = 0x0}, pool = 0x55d85c032990, free_frames = 0x0, free_fake_connections = 0x0, 
  streams_index = 0x55d85c002fb8, last_out = 0x0, dependencies = {prev = 0x55d85bfc0398, next = 0x55d85bfc0398}, closed = {prev = 0x55d85c18a698, next = 0x55d85c18a698}, closed_nodes = 0, last_sid = 1, lingering_time = 0, 
  settings_ack = 1, table_update = 1, blocked = 1, goaway = 0, fp_fingerprinted = 0, fp_settings = {len = 35, data = 0x55d85c0030b8 "\001"}, fp_windowupdate = 0, fp_priorities = {len = 4, data = 0x55d85bfc02f0 "\001"}, 
  fp_pseudoheaders = {len = 2, data = 0x55d85bfc0310 "ce\374[\330U"}, fp_str = {len = 0, data = 0x0}}

(gdb) p header
$13 = (ngx_http_v2_header_t *) 0x55d85c18a5a0
(gdb) p header->name
$14 = {len = 5, data = 0x55d85a3abcbe ":path"}
(gdb) p header->value
$15 = {len = 165, data = 0x55d85c0ea340 "***SENSITIVE INFO (url)***"}
(gdb) p h2c->hpack
$17 = {entries = 0x55d85c204dc0, added = 0, deleted = 0, reused = 0, allocated = 64, size = 4096, free = 4096, storage = 0x0, pos = 0x0}
(gdb) p **h2c->hpack->entries
$20 = {name = {len = 94387745532352, data = 0x55d85bd08010 "\a"}, value = {len = 0, data = 0x0}}
(gdb) p *h2c->connection
$24 = {data = 0x55d85c18a500, read = 0x7eff26bb8d70, write = 0x7eff26a37d70, fd = 49, recv = 0x55d85a28fc60 <ngx_ssl_recv>, send = 0x55d85a290100 <ngx_ssl_write>, recv_chain = 0x55d85a290040 <ngx_ssl_recv_chain>, 
  send_chain = 0x55d85a290530 <ngx_ssl_send_chain>, listening = 0x55d85bd27f10, sent = 58, log = 0x55d85c102a60, pool = 0x55d85c102a00, type = 1, sockaddr = 0x55d85c102a50, socklen = 16, addr_text = {len = 12, 
    data = 0x55d85c102ab0 "84.*IP****"}, proxy_protocol = 0x0, ssl = 0x55d85c102b18, udp = 0x0, local_sockaddr = 0x55d85bf44430, local_socklen = 16, buffer = 0x0, queue = {prev = 0x7eff26d38748, next = 0x7eff26d38fb8}, 
  number = 737234, start_time = 84136899431, requests = 1, buffered = 0, log_error = 2, timedout = 0, error = 0, destroyed = 0, pipeline = 0, idle = 1, reusable = 0, close = 0, shared = 0, sendfile = 0, sndlowat = 0, tcp_nodelay = 1, 
  tcp_nopush = 0, need_last_buf = 0, need_flush_buf = 0, sendfile_task = 0x0}
(gdb) p *h2c->connection->pool
$25 = {d = {last = 0x55d85c102bfc "\330U", end = 0x55d85c102c00 "\020\002", next = 0x55d85c002ef0, failed = 1}, max = 432, current = 0x55d85c102a00, chain = 0x0, large = 0x55d85c0030d8, cleanup = 0x55d85c002fa0, log = 0x55d85c102a60}
(gdb) 

(gdb) f 3
#3  0x000055d85a2d6c4b in ngx_http_v2_state_process_header (h2c=0x55d85c18a500, 
    pos=0x55d85bf57d56 "A\217%\225ȕ\350\067r\364\032)5K\220\364\377S\260I|\245\211\323M\037C\256\272\fA\244ǩ\217\063\246\232?ߚh\372\035u\320b\r&=Ly\246\217\276\320\001w\376\276X\371\373", <incomplete sequence \355>, 
    end=0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h") at src/http/v2/ngx_http_v2.c:1778
1778    src/http/v2/ngx_http_v2.c: No such file or directory.
(gdb) p pos
$39 = (u_char *) 0x55d85bf57d56 "A\217%\225ȕ\350\067r\364\032)5K\220\364\377S\260I|\245\211\323M\037C\256\272\fA\244ǩ\217\063\246\232?ߚh\372\035u\320b\r&=Ly\246\217\276\320\001w\376\276X\371\373", <incomplete sequence \355>
(gdb) p end
$42 = (u_char *) 0x55d85bf57e70 "\177\222\235)\255\027\030cǏ\v\216\241\321!\252_ϟ?@\212AH\264\245I'Y\006I\177\207%\207B\026A\222_@\212AH\264\245I'Z\223\310_\203!\354G@\212AH\264\245I'ZB\241?\204-5\247\327s\223\235)\255\027\030cǏ\v\216\241\321!\252_ϟ,\177P\222\233٫\372RB\313@\322_\245#\263\351OhL\237Q\234\">\213W\337h"
(gdb) p *h2c->connection->log
$47 = {log_level = 4, file = 0x55d85bd27a70, connection = 737234, disk_full_time = 0, handler = 0x55d85a2a1a30 <ngx_http_log_error>, data = 0x55d85c102b00, writer = 0x0, wdata = 0x0, 
  action = 0x55d85a3aaaec "processing HTTP/2 connection", next = 0x0}
(gdb) p **h2c->streams_index
$51 = {id = 1, index = 0x0, parent = 0xffffffffffffffff, queue = {prev = 0x55d85c18a688, next = 0x55d85c18a688}, children = {prev = 0x55d85bfc03a8, next = 0x55d85bfc03a8}, reuse = {prev = 0x0, next = 0x0}, rank = 1, weight = 16, 
  rel_weight = 0.0625, stream = 0x55d85bfc9ee0}

(gdb) p *ngx_cycle
$57 = {conf_ctx = 0x55d85bd28978, pool = 0x55d85bd27680, log = 0x55d85bd276e8, new_log = {log_level = 4, file = 0x55d85bd27a70, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, 
    next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x7eff26d384b0, free_connection_n = 16278, modules = 0x55d85bd290e0, modules_n = 107, modules_used = 1, reusable_connections_queue = {prev = 0x7eff26d39468, 
    next = 0x7eff26d3c528}, reusable_connections_n = 93, connections_reuse_time = 0, listening = {elts = 0x55d85bd27de8, nelts = 7, size = 296, nalloc = 10, pool = 0x55d85bd27680}, paths = {elts = 0x55d85bd27a08, nelts = 6, size = 8, 
    nalloc = 10, pool = 0x55d85bd27680}, config_dump = {elts = 0x55d85bd27a58, nelts = 0, size = 24, nalloc = 1, pool = 0x55d85bd27680}, config_dump_rbtree = {root = 0x55d85bd27820, sentinel = 0x55d85bd27820, 
    insert = 0x55d85a2664c0 <ngx_str_rbtree_insert_value>}, config_dump_sentinel = {key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x55d85bd27850, part = {
      elts = 0x55d85bd27a70, nelts = 1, next = 0x0}, size = 40, nalloc = 20, pool = 0x55d85bd27680}, shared_memory = {last = 0x55d85bd4c108, part = {elts = 0x55d85bd27d90, nelts = 1, next = 0x55d85bd4bfd0}, size = 88, nalloc = 1, 
    pool = 0x55d85bd27680}, connection_n = 16384, files_n = 0, connections = 0x7eff26d37010, read_events = 0x7eff26bb6010, write_events = 0x7eff26a35010, old_cycle = 0x0, conf_file = {len = 42, 
    data = 0x55d85bd279ba "/usr/local/openresty/nginx/conf/nginx.conf"}, conf_param = {len = 29, data = 0x55d85bd279e5 "daemon on; master_process on;\032\063\377~"}, conf_prefix = {len = 32, 
    data = 0x55d85bd27970 "/usr/local/openresty/nginx/conf//usr/local/openresty/nginx/logs/error.log"}, prefix = {len = 27, data = 0x55d85bd27990 "/usr/local/openresty/nginx/logs/error.log"}, error_log = {len = 14, 
    data = 0x55d85bd279ab "logs/error.log"}, lock_file = {len = 43, data = 0x55d85bf446fc "/usr/local/openresty/nginx/logs/nginx.lock.accept"}, hostname = {len = 24, data = 0x55d85bd290c8 "*************\200?Z\330U"}, 
  intercept_error_log_handler = 0x0, intercept_error_log_data = 0x0, entered_logger = 0}

phuslu / nginx-ssl-fingerprint

nginx 1.25.3.1 http2 segfault with ja3 patch added #51