Memory leak after DTLS request

[vwout]

I'm using libnyoci in a library for Lua (to make coap available in Lua). When a lot of requests are made with DTLS activated, memory is getting lost.

It looks like somehow the tls data is not cleaned up. The data is received using the same loop that is in the cmd_get code of nyocictl:

  while (ERRORCODE_INPROGRESS == gRet) {
    if (nyoci) {
      nyoci_plat_wait(nyoci, 1000);
      nyoci_plat_process(nyoci);
    }
  }

After all response data from the GET request is received, I do call SSL_CTX_free on the SSL context and eventually nyoci_release. This however does not seem to cause the internal TLS functions SSL_shutdown or SSL_free to be called.

I reproduced this with nyocictl. A valgrind trace with a GET request shows that the memory allocated using SSL_new and BIO_new in openssl/nyoci-plat-tls.c are never free'd.

How can I make sure that all resources related to a specific DTLS ssl session are released after use?

[vwout]

To add more detail, this is the command used to reproduce:

# valgrind --leak-check=full --show-leak-kinds=all ./nyoci-prefix/bin/nyocictl -p 5684 -i id -P psk coaps://192.168.0.2
coaps://192.168.0.2> get /15001/65543
coaps://192.168.0.2> exit

The following leaks (amongst others) are reported:

==6218== 1 bytes in 1 blocks are indirectly lost in loss record 2 of 120
==6218==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==6218==    by 0x4CE9E24: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CE6CFF: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CF39D4: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CF66C4: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CF0152: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CDBF43: SSL_do_handshake (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x48529C3: handle_ssl_outbound_traffic (nyoci-plat-tls.c:399)
==6218==    by 0x4852D9A: nyoci_plat_tls_inbound_packet_process (nyoci-plat-tls.c:884)
==6218==    by 0x4851E25: nyoci_plat_process (nyoci-plat-net.c:931)
==6218==    by 0x10EB64: tool_cmd_get (cmd_get.c:377)
==6218==    by 0x10C955: exec_command (main.c:227)
==6218==    by 0x10C955: process_input_line (main.c:347)

==6218== 16 bytes in 1 blocks are indirectly lost in loss record 12 of 120
==6218==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==6218==    by 0x4EC4558: CRYPTO_zalloc (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==6218==    by 0x4DF9038: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==6218==    by 0x4DF3EE0: BIO_new (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==6218==    by 0x4853319: nyoci_plat_tls_outbound_packet_process (nyoci-plat-tls.c:934)
==6218==    by 0x484CBFC: nyoci_outbound_send (nyoci-outbound.c:790)
==6218==    by 0x10E785: resend_get_request (cmd_get.c:199)
==6218==    by 0x10E785: resend_get_request (cmd_get.c:185)
==6218==    by 0x484E545: nyoci_internal_resend (nyoci-transaction.c:262)
==6218==    by 0x484E545: nyoci_internal_transaction_timeout_ (nyoci-transaction.c:314)
==6218==    by 0x4851E37: nyoci_plat_process (nyoci-plat-net.c:942)
==6218==    by 0x10EB64: tool_cmd_get (cmd_get.c:377)
==6218==    by 0x10C955: exec_command (main.c:227)
==6218==    by 0x10C955: process_input_line (main.c:347)
==6218==    by 0x10C9C8: process_input_readline (main.c:378)

==6218== 16 bytes in 1 blocks are indirectly lost in loss record 13 of 120
==6218==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==6218==    by 0x4EC4558: CRYPTO_zalloc (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==6218==    by 0x4CC46E9: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CC4AC6: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CC21B4: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4CDE560: SSL_new (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
==6218==    by 0x4853355: nyoci_plat_tls_outbound_packet_process (nyoci-plat-tls.c:939)
==6218==    by 0x484CBFC: nyoci_outbound_send (nyoci-outbound.c:790)
==6218==    by 0x10E785: resend_get_request (cmd_get.c:199)
==6218==    by 0x10E785: resend_get_request (cmd_get.c:185)
==6218==    by 0x484E545: nyoci_internal_resend (nyoci-transaction.c:262)
==6218==    by 0x484E545: nyoci_internal_transaction_timeout_ (nyoci-transaction.c:314)
==6218==    by 0x4851E37: nyoci_plat_process (nyoci-plat-net.c:942)
==6218==    by 0x10EB64: tool_cmd_get (cmd_get.c:377)

The allocated memory is released in the function remove_ssl_object. This function only seems to be called in case of several different error situations in nyoci_plat_tls_outbound_packet_process.

[vwout]

It looks like the leakage is caused by the SSL sessions. This btree is not cleaned up when the nyoci object closes, this in contrary to the transactions btree.