Is it possible to reduce mbedtls incoming buffer size to save RAM?

[jhnlmn]

It appears that iot-device-sdk-embedded-c consumes about 45 KB or RAM and significant chunk of it is wasted on a huge input buffer: MBEDTLS_SSL_IN_BUFFER_LEN 16717 MBEDTLS_SSL_OUT_BUFFER_LEN 4429 I see in https://tls.mbed.org/kb/how-to/reduce-mbedtls-memory-and-storage-footprint " By default, Mbed TLS uses a 16 KB frame buffer to hold data for incoming and outgoing frames. This is a TLS standard requirement. If you control both sides of a connection (server and client), you can reduce the maximum frame size to reduce the buffers needed to store the data. The size of this frame is determined by MBEDTLS_SSL_MAX_CONTENT_LEN. You can safely reduce this to a more appropriate size (such as 2 KB) if:

Both sides support the max_fragment_length SSL extension (allowing reduction to under 1 KB for the buffers). You know the maximum size that will ever be sent in a single SSL/TLS frame (whether or not you control both sides of the connection). " So, my question is: do you know whether google IOT server supports max_fragment_length SSL extension? Or do you know the maximum size that will ever be sent in a single SSL/TLS frame, assuming my IOT message payload is less than 1 KB? Can these small messages be somehow combined to form a larger frame?

Note that I already tried to reduce CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN from 16384 to 4096 and it seem to work OK, but I want to be sure. Thank you

[atigyi]

I've ended up in this openssl command parameter -maxfraglen, trying to set the max fragment lenght:

$ openssl s_client -tlsextdebug -maxfraglen 512 -state mqtt.googleapis.com:8883                                                 
CONNECTED(00000003)                                                                         
SSL_connect:before SSL initialization                                                       
SSL_connect:SSLv3/TLS write client hello                                                    
SSL_connect:SSLv3/TLS write client hello                                                    
TLS server extension "key share" (id=51), len=36                
0000 - 00 1d 00 20 48 38 c1 07-a1 a0 5d bc 49 22 ca 83   ... H8....].I"..
0010 - 11 aa f7 d7 a9 33 5b a6-b8 06 3f f9 2e b9 d2 1e   .....3[...?.....
0020 - 94 04 b6 2d                                       ...-   
TLS server extension "supported versions" (id=43), len=2        
0000 - 03 04                                             ..     
SSL_connect:SSLv3/TLS read server hello                                                     
SSL_connect:TLSv1.3 read encrypted extensions                                               
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1 
verify return:1                               
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1                               
depth=0 CN = mqtt.googleapis.com
verify return:1                                                                             
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:TLSv1.3 read server certificate verify
SSL_connect:SSLv3/TLS read finished
SSL_connect:SSLv3/TLS write change cipher spec 
SSL_connect:SSLv3/TLS write finished
---
...                                           

_"do you know whether google IOT server supports max_fragmentlength SSL extension?" Unfortunately I don't see a TLS server extension response (in the log above) which would validate the negotiation was successful. Additionally this TLS feature is primarily for resource constrained clients, IoT's TLS setup does not differ much from any other Google service TLS, so the best would be to explicitly test it.

"Or do you know the maximum size that will ever be sent in a single SSL/TLS frame, assuming my IOT message payload is less than 1 KB?" The ceiling of MQTT message size is larger than 16 KB, the default TLS fragment size. So there is no explicit MQTT layer limit which would prevent sending 16 KB+ payloads.

A test - I imagine - would be negotiating the max fragment length (e.g. for 512 KB) in the TLS handshake and try to send an 512 KB+ config message from Cloud IoT to the device, meanwhile watching the device traffic with Wireshark if the "large" MQTT message is split up into <=512 KB fragments.

_"Note that I already tried to reduce CONFIG_MBEDTLS_SSL_IN_CONTENTLEN from 16384 to 4096 and it seem to work OK, but I want to be sure." This sounds good, but unfortunately this isn't a guarantee the negotiation was successful. My guess here is either of these two happens: 1) the 4 KB max fragment length negotiation was successful OR 2) the negotiation was silently dropped and the server didn't send 4 KB+ message so far, thus the buffer was enough to store incoming messages.

The test mentioned above would tell the truth.

[jhnlmn]

Additionally this TLS feature is primarily for resource constrained clients, IoT's TLS setup does not differ much from any other Google service TLS

This is ridiculous. IoT devices are usually resource constrained and google simply cannot offer exactly the same service for huge desktop PCs and tiny embedded devices. How hard it would be for them to enable max_fragment_length extension on the server to reduce cost of IOT devices?

A test - I imagine - would be negotiating the max fragment length (e.g. for 512 KB) in the TLS handshake and try to send an 512 KB+ config message from Cloud IoT to the device, meanwhile watching the device traffic with Wireshark if the "large" MQTT message is split up into <=512 KB fragments.

1. There is no real negotiation: Client Hello does contain max_fragment_length field, Server Hello does not contain it, but client proceeds anyway and fails only when it receives a large message.

2: Attempt to set: CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=512 mbedtls_ssl_conf_max_frag_len( conf, MBEDTLS_SSL_MAX_FRAG_LEN_512); or CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=1024 mbedtls_ssl_conf_max_frag_len( conf, MBEDTLS_SSL_MAX_FRAG_LEN_1024);

does not work at all - client disconnects immediately after Hello: <= fetch input input record: msgtype = 22, version = [3:3], msglen = 1522 bad message length <= handshake => free <= free I (5492) IOT: ERROR! Connection has failed reason 21

3. This seem to work: CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=2048 mbedtls_ssl_conf_max_frag_len( conf, MBEDTLS_SSL_MAX_FRAG_LEN_2048); (note that the second line is not really needed because google appears to ignore max_fragment_length)

Then I tried to send config payload > 2048 bytes and message was divided into chunks no more than 1425 bytes: input record: msgtype = 23, version = [3:3], msglen = 1425 input record: msgtype = 23, version = [3:3], msglen = 721

So, google ignores max_fragment_length, but it seem to divide messages anyway. Is this behavior documented? Can I rely on it? Can 1425 be a limitation of a lower level link and be different depending on route?

Also, if I am certain that I will never send config or a command larger than 500 bytes, am I safe to use 2048 input buffer? Can server combine several queued small MQTT messages into one large TLS message?

[atigyi]

tl;dr Google does not support the max_fragment_length extension. RFC6066

Although a device connection can remain lucky and experiencing desired message lengths. E.g. currently 2KB might be enough to connect to mqtt.2030.ltsapis.goog but currently 8KB is required to connect to mqtt.googleapis.com due to different trusted root CA cert set. But this is pure luck and NOT GUARANTEED at all and can even change by time.

Q: "...but it seem to divide messages anyway. Is this behavior documented? Can I rely on it?" A: Unfortunately this cannot be relied on. Google does not make any special guarantees beyond the spec about message lengths.

Q: "Can 1425 be a limitation of a lower level link and be different depending on route?" Q: "Can server combine several queued small MQTT messages into one large TLS message?" A: We need more time to answer these questions. Although don't these questions loose their relevance given that the max_fragment_length isn't supported?

[jhnlmn]

OK, I understand that I cannot rely on any buffer smaller than standard 16 KB, even if 2 KB seem to work. There is no way I can squeeze these 16 KB into my existing ESP32 WROOM, so my only choice is to switch to a more expensive WROVER, which has an extra 8 MB PSRAM. Could you imagine something more stupid that having to pay for 8 MB PSRAM, when I only need 16 KB and all of that because google was so lazy to enable maxfraglen support on their servers? For google it is just a config option, but for IOT device makers it is an extra $1 in BOM, which is huge cost for many.

Although don't these questions loose their relevance given that the max_fragment_length isn't supported?

Well, remember this quote from https://tls.mbed.org/kb/how-to/reduce-mbedtls-memory-and-storage-footprint " You can safely reduce this to a more appropriate size (such as 2 KB) if: Both sides support the max_fragment_length SSL extension (allowing reduction to under 1 KB for the buffers). You know the maximum size that will ever be sent in a single SSL/TLS frame (whether or not you control both sides of the connection). "

So, we know that max_fragment_length is not enabled, but do we know the maximum size that will ever be sent?

currently 8KB is required to connect to mqtt.googleapis.com due to different trusted root CA cert set

Where this 8KB came from? Is it for incoming or outgoing buffer? Of course if I go with 8 MB PSRAM, then it is moot point, but may be this info will be useful for somebody else.

By the way, did you see my other issue: https://github.com/GoogleCloudPlatform/iot-device-sdk-embedded-c/issues/128 ? Could you imagine that enabling support for "%lld" increases code/flash size by 69 KB on ESP32 and it cannot fit into 4 MB flash and requires buying 8 MB flash?

Thank you