Over-large packet makes long ciphertext and causes the stack overflow at dtls_handle_message()

[Mrsdwang]

In the master branch (53a0d97), DTLS server allows to receive the over-large packets. By sending over-large packets, the stack will overflow and attacker can arrive to the address he want.

0 is_record (msglen=8484, msg=0x6196d9 <error: Cannot access memory at address 0x6196d9>)

at dtls.c:361

1 dtls_handle_message (ctx=0x61b00001f180, session=0x7fffffffdf60,

msg=0x6196d9 <error: Cannot access memory at address 0x6196d9>, msglen=8484) at dtls.c:3779

2 0x0000000000402165 in dtls_handle_read (ctx=0x61b00001f180) at dtls-server.c:183

3 0x0000000000402ee5 in main (argc=1, argv=0x7fffffffe408) at dtls-server.c:351

If packets is over large, the variable "rlen" in is_record function will be very large but it still lower than packet length and "rlen" not be set to "0" , so when program carries to the dtls.c:3916 , "msg += rlen", the value of "msg "is larger ,so the stack overflows, and the BUG is triggered.

In my test case, the packet's length is 28073, and the "rlen" will be 19545, "rlen" is lower than 28073,so it won't set to be "0", when pointer "msg" adds the "rlen" , huge value of "msg" causes the stack overflow.

[Mrsdwang]

the problem of "rlen" is caused by converting the type of "msg" to dtls_record_header_t *, so "msg" is cut out 13 bytes,and the variable "length" get the 12th and 13th bytes . In my test case the 12th and 13th bytes are 4c59, so the "rlen" is 19545.