[DTLS] Fail to handshake on server if server uses `SSL_CTX_set_max_send_fragment`

[nak3]

description

• When SSL_CTX_set_max_send_fragment(ctx, 512) is used on DTLS server side, server failed due to SSL_accept error = 5 when client tried to connect.

• After investigating the issue, I figured out the error returned from the code blow:

https://github.com/libressl/openbsd/blob/3d60073121c9fed2d9a86b0ec752999b75409e21/src/lib/libssl/d1_both.c#L292-L305

            if (BIO_ctrl(SSL_get_wbio(s),
                BIO_CTRL_DGRAM_MTU_EXCEEDED, 0, NULL) > 0)
                s->d1->mtu = BIO_ctrl(SSL_get_wbio(s),
                    BIO_CTRL_DGRAM_QUERY_MTU, 0, NULL);
            else
                return (-1);

• BIO_CTRL_DGRAM_MTU_EXCEEDED check retruned 1 due to exceeding the MTU and returned -1.

Reproducer

• It depends on the environment, but my Mac OS can 100% produce the issue by SSL_CTX_set_max_send_fragment(ctx, 512) on DTLS server side.

Proposal patch

• Make sure that setting len less than max_send_fragment.
• I verified that the issue could be solved by the patch.

diff --git src/lib/libssl/d1_both.c src/lib/libssl/d1_both.c
index b5c68a173..13f4baaf9 100644
--- src/lib/libssl/d1_both.c
+++ src/lib/libssl/d1_both.c
@@ -263,6 +263,10 @@ dtls1_do_write(SSL *s, int type)
                else
                        len = s->init_num;

+               if (len > s->max_send_fragment) {
+                       len = s->max_send_fragment;
+               }
+
                /* XDTLS: this function is too long.  split out the CCS part */
                if (type == SSL3_RT_HANDSHAKE) {
                        if (s->init_off != 0) {
@@ -274,6 +278,10 @@ dtls1_do_write(SSL *s, int type)
                                        len = curr_mtu;
                                else
                                        len = s->init_num;
+
+                               if (len > s->max_send_fragment) {
+                                       len = s->max_send_fragment;
+                               }
                        }

                        dtls1_fix_message_header(s, frag_off,