Crash in Asio SSL handshake

feliwir commented 3 years ago

Hello, i'm using LibreSSL with asio to perform a SSL handshake. However cbb_buffer_add is crashing for me. The issue is occuring randomly and i cannot really reproduce it reliably. Here is my callstack (the important parts of it)_

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7ae8535 in __GI_abort () at abort.c:79
#2  0x00007ffff7b3f508 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7c4a28d "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff7b45c1a in malloc_printerr (str=str@entry=0x7ffff7c4843b "free(): invalid pointer") at malloc.c:5341
#4  0x00007ffff7b4742c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:4165
#5  0x0000555555bcee2f in recallocarray (ptr=0x7fffe4072e80, oldnmemb=<optimized out>, newnmemb=newnmemb@entry=512, size=size@entry=1)
    at /home/mbits/Development/mes/Externals/libressl/crypto/compat/recallocarray.c:77
#6  0x0000555555b5a667 in cbb_buffer_add (base=0x7fffe4078dc0, out=out@entry=0x7ffff57833b8, len=len@entry=256) at /home/mbits/Development/mes/Externals/libressl/ssl/bs_cbb.c:114
#7  0x0000555555b5ac1e in CBB_add_space (cbb=0x7ffff5783410, out_data=0x7ffff57833b8, len=256) at /home/mbits/Development/mes/Externals/libressl/ssl/bs_cbb.c:374
#8  0x0000555555b5ac56 in CBB_add_bytes (cbb=cbb@entry=0x7ffff5783410, 
    data=data@entry=0x7fffe4069790 "\203l`6n\355\063\062,|W\345\236\377\t\\\352\177)A+(<o\274\071\242&\356\271\026\243\063\005r!Bk\036|\252T5\270\"\006\030k\303s\tF\267\354ggI\":\357\004|AD\204\064\024\035\376\270\nuw7?7v\227N\020\346\246\355h\031\202\261=y\336\274\252\336\305|\373\067\241;\264\224\374\304\337~;\035\067\370\320\312\002=q`\215\344\203U+\267=-\320\327\031\003\"\350\215\343̬߱\202M \310\025\345\037\375\276\067\362\350\260{\343\355\060҂\312\312\036\340\005\203oۯ\377\346\354\351Y|\244\230JJIX\343\230\324D1\210l\262\252\061\r\035\330u\273\232\252^=n \254!\224{"..., len=256) at /home/mbits/Development/mes/Externals/libressl/ssl/bs_cbb.c:364
#9  0x0000555555b59c33 in tls13_server_certificate_verify_send (ctx=0x7fffe4001460, cbb=0x7ffff5783480) at /home/mbits/Development/mes/Externals/libressl/ssl/tls13_server.c:724
#10 0x0000555555b6f0dd in tls13_handshake_send_action (action=0x555555e8f200 <state_machine+256>, ctx=0x7fffe4001460)
    at /home/mbits/Development/mes/Externals/libressl/ssl/tls13_handshake.c:417
#11 tls13_handshake_perform (ctx=ctx@entry=0x7fffe4001460) at /home/mbits/Development/mes/Externals/libressl/ssl/tls13_handshake.c:373
#12 0x0000555555b58f3b in tls13_server_accept (ctx=ctx@entry=0x7fffe4001460) at /home/mbits/Development/mes/Externals/libressl/ssl/tls13_server.c:61
#13 0x0000555555b566d7 in tls13_legacy_accept (ssl=0x7fffe4018520) at /home/mbits/Development/mes/Externals/libressl/ssl/tls13_legacy.c:392
#14 0x00005555555f8f53 in asio::ssl::detail::engine::do_accept (this=0x7fffe4001250) at /home/mbits/Development/mes/Code/glue-asio/external/asio/ssl/detail/impl/engine.ipp:302
#15 0x00005555555f8d06 in asio::ssl::detail::engine::perform (this=0x7fffe4001250, op=
    (int (asio::ssl::detail::engine::*)(class asio::ssl::detail::engine * const, void *, std::size_t)) 0x5555555f8f30 <asio::ssl::detail::engine::do_accept(void*, unsigned long)>, 
    data=0x0, length=0, ec=..., bytes_transferred=0x0) at /home/mbits/Development/mes/Code/glue-asio/external/asio/ssl/detail/impl/engine.ipp:234
#16 0x00005555555f8888 in asio::ssl::detail::engine::handshake (this=0x7fffe4001250, type=asio::ssl::stream_base::server, ec=...)
    at /home/mbits/Development/mes/Code/glue-asio/external/asio/ssl/detail/impl/engine.ipp:134

I'm using the latest LibreSSL version (3.2.2)

4a6f656c commented 3 years ago

Looking at the code, I do not see how we can end up with the CBB having an invalid pointer in normal use (and we'd likely have multiple reports if that was the case). This is basically suggesting that the base->buf pointer is invalid, however the only time that pointer changes is when recallocarray() is called or when it is freed (at which point it is set to NULL and we'd be seeing a NULL pointer dereference as a result).

Can you confirm that the trace is always the same (including tls13_server_certificate_verify_send) each time?

Without a reproducable test case it is going to be rather difficult to investigate further - you could add debugging to print/log the pointer when it is allocated (via the call to tls13_handshake_msg_start() on line 414 of tls13_handshake.c) and see how that compares to the pointer in the recallocarray() call that fails.

feliwir commented 3 years ago

The trace is indeed always the same, but i can't really produce a testcase. I only found this issue while testing our software, and unfortunaly i don't have the resources to track this down to the root

libressl / portable

Crash in Asio SSL handshake #633