search

OpenSIPS / opensips

OpenSIPS is a GPL implementation of a multi-functionality SIP Server that targets to deliver a high-level technical solution (performance, security and quality) to be used in professional SIP server platforms.
https://opensips.org
Other
1.28k stars 580 forks source link

[CRASH] 3.1 segfault in TLS #1799

Closed danpascu closed 4 years ago

danpascu commented 5 years ago

OpenSIPS version you are running

version: opensips 3.1.0-dev (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, CC_O0, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 6884b3d17
main.c compiled on  with gcc 8

Crash Core Dump

(gdb) bt
#0  EVP_DecryptUpdate (inl=16, in=0x7f901d14bd80 "\217\034|G\277\332:7o|\220\373\071\001\036\063xq\232)\234\225(\214\005o}:\370\230\330\n\341\212\350\330/\274\325\020\345\206\v\220\071", outl=0x7ffdba100a34, 
    out=0x7f901d14bd90 "xq\232)\234\225(\214\005o}:\370\230\330\n\341\212\350\330/\274\325\020\345\206\v\220\071", ctx=0x7f901d14be38) at ../crypto/evp/evp_enc.c:455
#1  EVP_DecryptUpdate (ctx=0x7f901d14be38, out=0x7f901d14bd90 "xq\232)\234\225(\214\005o}:\370\230\330\n\341\212\350\330/\274\325\020\345\206\v\220\071", outl=0x7ffdba100a34, 
    in=0x7f901d14bd80 "\217\034|G\277\332:7o|\220\373\071\001\036\063xq\232)\234\225(\214\005o}:\370\230\330\n\341\212\350\330/\274\325\020\345\206\v\220\071", inl=16) at ../crypto/evp/evp_enc.c:443
#2  0x00007f901cb760f3 in ctr_df (in3len=16, in3=0x7f901d14c5b8 "@W\375<\220\177", in2len=0, in2=0x0, in1len=<optimized out>, 
    in1=0x7f901e2f4ff8 "\f\017\357@A3\217\207\360)-X\372\357\355GL\036\330\177^j,\311\376Q=\266\333:\322", <incomplete sequence \356>, ctr=0x7f901d14bd18) at ../crypto/rand/drbg_ctr.c:210
#3  ctr_update (drbg=0x7f901d14bc70, in1=0x7f901e2f4ff8 "\f\017\357@A3\217\207\360)-X\372\357\355GL\036\330\177^j,\311\376Q=\266\333:\322", <incomplete sequence \356>, in1len=32, in2=0x7f901d14c5b8 "@W\375<\220\177", 
    in2len=<optimized out>, nonce=nonce@entry=0x0, noncelen=<optimized out>) at ../crypto/rand/drbg_ctr.c:264
#4  0x00007f901cb76423 in drbg_ctr_reseed (adinlen=<optimized out>, adin=<optimized out>, entropylen=<optimized out>, entropy=<optimized out>, drbg=<optimized out>) at ../crypto/rand/drbg_ctr.c:304
#5  drbg_ctr_reseed (drbg=<optimized out>, entropy=<optimized out>, entropylen=<optimized out>, adin=<optimized out>, adinlen=<optimized out>) at ../crypto/rand/drbg_ctr.c:298
#6  0x00007f901cb76dd1 in RAND_DRBG_reseed (drbg=drbg@entry=0x7f901d14bc70, adin=adin@entry=0x7f901d14c5b8 "@W\375<\220\177", adinlen=adinlen@entry=16, prediction_resistance=prediction_resistance@entry=0)
    at ../crypto/rand/drbg_lib.c:450
#7  0x00007f901cb7726a in RAND_DRBG_generate (drbg=drbg@entry=0x7f901d14bc70, out=out@entry=0x7f901e1f8650 "", outlen=outlen@entry=32, prediction_resistance=prediction_resistance@entry=0, adin=0x7f901d14c5b8 "@W\375<\220\177", 
    adinlen=adinlen@entry=16) at ../crypto/rand/drbg_lib.c:630
#8  0x00007f901cb77481 in RAND_DRBG_bytes (drbg=0x7f901d14bc70, out=0x7f901e1f8650 "", outlen=32) at ../crypto/rand/drbg_lib.c:679
#9  0x00007f901ccdd6fd in ssl_fill_hello_random (s=s@entry=0x7f901e2117a8, server=server@entry=1, result=0x7f901e1f8650 "", len=len@entry=32, dgrd=DOWNGRADE_TO_1_2) at ../ssl/s3_lib.c:4589
#10 0x00007f901cd0ea99 in tls_early_post_process_client_hello (s=0x7f901e2117a8) at ../ssl/statem/statem_srvr.c:1902
#11 tls_post_process_client_hello (s=0x7f901e2117a8, wst=<optimized out>) at ../ssl/statem/statem_srvr.c:2226
#12 0x00007f901ccfe55b in read_state_machine (s=0x7f901e2117a8) at ../ssl/statem/statem.c:664
#13 state_machine (s=0x7f901e2117a8, server=1) at ../ssl/statem/statem.c:434
#14 0x00007f901ccea264 in SSL_do_handshake (s=0x7f901e2117a8) at ../ssl/ssl_lib.c:3599
#15 0x00007f901cd6702a in tls_accept (c=0x7f901e1fd250, poll_events=0x0) at ../tls_mgm/tls_conn_server.h:270
#16 0x00007f901cd685c7 in tls_fix_read_conn (c=0x7f901e1fd250, t_dst=0x0) at ../tls_mgm/tls_conn_server.h:529
#17 0x00007f901cd6c39c in tls_read_req (con=0x7f901e1fd250, bytes_read=0x7ffdba100e6c) at proto_tls.c:546
#18 0x00005580191c1194 in handle_io (fm=0x7f90191f4bd0, idx=1, event_type=1) at net/net_tcp_proc.c:301
#19 0x00005580191bf842 in io_wait_loop_epoll (h=0x5580192e0880 <_worker_io>, t=2, repeat=0) at net/../io_wait_loop.h:280
#20 0x00005580191c1a6c in tcp_worker_proc_loop () at net/net_tcp_proc.c:411
#21 0x00005580191d1fdb in tcp_start_processes (chd_rank=0x5580192a9f3c <chd_rank>, startup_done=0x0) at net/net_tcp.c:2076
#22 0x00005580190fc4f0 in main_loop () at main.c:801
#23 0x00005580190ff86c in main (argc=15, argv=0x7ffdba1011b8) at main.c:1480
(gdb) frame 0
#0  EVP_DecryptUpdate (inl=16, in=0x7f901d14bd80 "\217\034|G\277\332:7o|\220\373\071\001\036\063xq\232)\234\225(\214\005o}:\370\230\330\n\341\212\350\330/\274\325\020\345\206\v\220\071", outl=0x7ffdba100a34, 
    out=0x7f901d14bd90 "xq\232)\234\225(\214\005o}:\370\230\330\n\341\212\350\330/\274\325\020\345\206\v\220\071", ctx=0x7f901d14be38) at ../crypto/evp/evp_enc.c:455
455     ../crypto/evp/evp_enc.c: No such file or directory.
(gdb) print ctx
$7 = (EVP_CIPHER_CTX *) 0x7f901d14be38
(gdb) print ctx->cipher
$8 = (const EVP_CIPHER *) 0x7f901e2f8058
(gdb) print ctx->cipher->block_size 
$9 = 1279593435
(gdb) print *ctx->cipher
$10 = {nid = -1022239884, block_size = 1279593435, key_len = 1806233390, iv_len = 1994392236, flags = 13227913398160867695, init = 0x4f617c2e4b125fe0, do_cipher = 0xc0c0c0c0c0c0c0c0, cleanup = 0xabcdefedabcdefed, 
  ctx_size = 505481600, set_asn1_parameters = 0x500000000, get_asn1_parameters = 0x20, ctrl = 0x7f901d14be08, app_data = 0x20}
(gdb) print /x *ctx->cipher
$11 = {nid = 0xc311db74, block_size = 0x4c450bdb, key_len = 0x6ba8ef2e, iv_len = 0x76e002ac, flags = 0xb792fd26b102416f, init = 0x4f617c2e4b125fe0, do_cipher = 0xc0c0c0c0c0c0c0c0, cleanup = 0xabcdefedabcdefed, ctx_size = 0x1e210980, 
  set_asn1_parameters = 0x500000000, get_asn1_parameters = 0x20, ctrl = 0x7f901d14be08, app_data = 0x20}
(gdb)

it crahses at line 455 in libssl in crypto/evp/evp_enc.c which reads:

   443  int EVP_DecryptUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl,
   444                        const unsigned char *in, int inl)
   445  {
   446      int fix_len, cmpl = inl;
   447      unsigned int b;
   448  
   449      /* Prevent accidental use of encryption context when decrypting */
   450      if (ctx->encrypt) {
   451          EVPerr(EVP_F_EVP_DECRYPTUPDATE, EVP_R_INVALID_OPERATION);
   452          return 0;
   453      }
   454  
   455      b = ctx->cipher->block_size;
   456  
   457      if (EVP_CIPHER_CTX_test_flags(ctx, EVP_CIPH_FLAG_LENGTH_BITS))
   458          cmpl = (cmpl + 7) / 8;
   459

The segfault is a bit weird, because while ctx->cipher->block_size contains an absurdly high value suggesting that ctx->cipher is corrupted, it can be accessed just fine in the debugger, i.e. there doesn't seem to be an illegal memory access on line 455, unless there was in this process at the time, but then some other process overwrote ctx->cipher during shutdown.

Also the value for cipher->do_cipher and cipher->cleanup look suspiciously like memory boundary markers from qmalloc, but I've seen no memory corruption being reported by qmalloc despite the fact that I run with Q_MALLOC_DBG.

To Reproduce

Wish I knew.

Relevant System Logs

While this particular process crashed, another process reported some errors about some resource being temporarily unavailable in TLS code:

Aug 12 02:33:34 node15 ./opensips[24070]: CRITICAL:core:sig_usr: segfault in process pid: 24070, id: 13
Aug 12 02:33:34 node15 ./opensips[24069]: ERROR:proto_tls:tls_accept: New TLS connection from 1.2.3.4:34454 failed to accept
Aug 12 02:33:34 node15 ./opensips[24069]: ERROR:proto_tls:tls_accept: TLS error: (ret=-1, err=1, errno=11/Resource temporarily unavailable):
Aug 12 02:33:34 node15 ./opensips[24069]: ERROR:proto_tls:tls_print_errstack: TLS errstack: error:24067044:random number generator:rand_pool_add:internal error
Aug 12 02:33:34 node15 ./opensips[24069]: ERROR:proto_tls:tls_read_req: failed to do pre-tls reading
Aug 12 02:33:36 node15 ./opensips[24055]: INFO:core:handle_sigs: child process 24070 exited by a signal 11

OS/environment information

bcnewlin commented 4 years ago

These fixes are working for me in 2.4 for SIP TLS, but appear to have broken HTTPS using the rest_client. See latest update in #1905.

razvancrainea commented 4 years ago

There seem to be no other crahses in TLS, so I believe we're good to close this ticket. If there's any other problems with TLS, please open a new bug report.