esp8266: TLS buffer overflow in axtls

[cemox]

Hi, I wrote a program that reads content from a news api request, everything works fine (actually I orijinally built my esp8266 projest with nodemcu but it is useless whit https connection, micropython is great).

in my program I have ssl.wrap_socket (called s) s = ssl.wrap_socket(s, server_hostname=TS_HOST) and it reads 2048 bytes from stream data = s.read(2048)

it works most of the time, but at times, it gives me buffer overflow error like this

TLS buffer overflow, record size: 5296 (+5)

when the incoming data is huge. Isn't the size parameter of read method supposed to force the wrapper to stop reading data when it reaches 2048 bytes and ignore the rest and continue execution without throwing an exception?

[dpgeorge]

Isn't the size parameter of read method supposed to force the wrapper to stop reading data when it reaches 2048 bytes and ignore the rest and continue execution without throwing an exception?

With TLS (https) it's a bit more complicated than that: the underlying TLS communication stream has a maximum packet size of 16k and it's possible that your device is receiving such a packet. This packet cannot be split up into chunks, it must be recieved, stored and processed (by the axtls library) all at once. The esp8266 must therefore have enough buffer space to store 16k at once. This is independent from your request for 2048 bytes from the stream, which is a request for unencrypted data.

You can try to work around the issue by making sure the incoming data is not that large.

Otherwise it's possible to recompile the esp8266 firmware to allow for a larger buffer size. Edit line 231 of esp8266/Makefile and change the -DRT_EXTRA=4096 to something like -RT_EXTRA=5120, then do make clean and make.

[cemox]

Very informative answer, thank you.

[cefn]

Is there support in Axtls/Micropython for RFC6606 - negotiating lesser max_fragment_size values for a TLS connection?

With this approach the esp8266 build would attempt to negotiate with the server a maximum TLS record size corresponding to its buffer, (though of course this only works if the esp8266 is a client and the server could always refuse).

Do you know if this negotiation already takes place, or if perhaps it could be added? I struggled to find any part of the micropython build which attempts to negotiate TLS record sizes. There is an indication that the protocol support is there, (perhaps just to ignore these TLS negotiating headers, though), for example in tls1.h ...

/* SSL extension types */
enum
{
    SSL_EXT_SERVER_NAME = 0,
    SSL_EXT_MAX_FRAGMENT_SIZE,
    SSL_EXT_SIG_ALG = 0x0d,
};

This is all predicated on whether server support for RFC6606 is widespread enough to bother, for example I am required to use TLS to use the Twitter API, but not sure yet if the Twitter server respects max_fragment_size in TLS negotiation.

[balajisraghavan]

Otherwise it's possible to recompile the esp8266 firmware to allow for a larger buffer size. Edit line 231 of esp8266/Makefile and change the -DRT_EXTRA=4096 to something like -RT_EXTRA=5120

@dpgeorge Could you please point out the place to increase the TLS buffer size in the latest codebase? Can't find -DRT_EXTRA=4096 in the make file anymore

[cefn]

Looks to me like it's at https://github.com/micropython/micropython/blob/master/ports/esp8266/Makefile#L8 but maybe I missed something.

[dpgeorge]

Yes, line 8 of the esp8266/Makefile is the new location.

[balajisraghavan]

Thanks @cefn and @dpgeorge. I tried changing it but looks like google cloud certs are about 8k and the esp8266 cant cope with that. Planning to try with the esp32 now.

[VinzSpring]

Isn't the size parameter of read method supposed to force the wrapper to stop reading data when it reaches 2048 bytes and ignore the rest and continue execution without throwing an exception?

With TLS (https) it's a bit more complicated than that: the underlying TLS communication stream has a maximum packet size of 16k and it's possible that your device is receiving such a packet. This packet cannot be split up into chunks, it must be recieved, stored and processed (by the axtls library) all at once. The esp8266 must therefore have enough buffer space to store 16k at once. This is independent from your request for 2048 bytes from the stream, which is a request for unencrypted data.

You can try to work around the issue by making sure the incoming data is not that large.

Otherwise it's possible to recompile the esp8266 firmware to allow for a larger buffer size. Edit line 231 of esp8266/Makefile and change the -DRT_EXTRA=4096 to something like -RT_EXTRA=5120, then do make clean and make.

Hi, I am trying to build the ROM with the specified buffer size for use with the telegram bot API (4MB ESP8266). However I can't get it to compile successfully since the esp-open-sdk fails to build. Would someone mind to provide a link to a valid binary with increased buffersize? I really need to be able to use the telegram API. Thank you so much.

[dpgeorge]

However I can't get it to compile successfully since the esp-open-sdk fails to build.

You can download a compiled version of the esp-open-sdk from https://github.com/jepler/esp-open-sdk/releases/download/2018-06-10/xtensa-lx106-elf-standalone.tar.gz

[YoruTen]

Have anyone managed to create a build with a larger buffer? I tried to do a 8192 buffer but got "fatal exception 29(storeprohibited cause)" as a new error. I would guess that 8192 is way to large for the esp8266, but my TLS buffer overflow error reported a size of 7500.

[cefn]

A pre-configured image (based on version 5d0d12c9) which works around the ESP8266 limitations noted below is available at http://shrimping.it/project/medea/ . It is based on the latest github at the time of writing with the medea library and examples onboard as frozen modules and a 8192byte TLS buffer to allow larger HTTPS payloads (e.g. at least 10 tweets from a Twitter timeline, at least a day's worth of 3-hourly OpenWeatherMap forecasts). Quote from the README.md of https://github.com/ShrimpingIt/medea

Possibly so old as to be useless though.

[flokain]

@YoruTen i was able to set the buffersize to 8192b as the packages i recieved were 7515b. It worked without a problem.

[YoruTen]

Which board are you using and could you post the image?

On Thu, 22 Oct 2020 at 00:27, flokain notifications@github.com wrote:

@YoruTen https://github.com/YoruTen i was able to set the buffersize to 8192,b as the packages i recieved were 7515b. It worked without a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/micropython/micropython/issues/3279#issuecomment-713912691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF7BHCCGIIR53W2N4XYMPQ3SL5N6RANCNFSM4DXRAA5Q .

-- //Andreas

[LLllIIUU]

Which board are you using and could you post the image? … On Thu, 22 Oct 2020 at 00:27, flokain @.***> wrote: @YoruTen https://github.com/YoruTen i was able to set the buffersize to 8192,b as the packages i recieved were 7515b. It worked without a problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3279 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF7BHCCGIIR53W2N4XYMPQ3SL5N6RANCNFSM4DXRAA5Q . -- //Andreas

Hi, have you successfully set the buffer size to 8192b now? What if it was successful?

[flokain]

@YoruTen Sorry for the late reply. I was using a wemos d2 mini board. the image consist a custom firmware so it would not work for you. but find the script to build it at the end of the post.

@LLllIIUU used the following make command to create the image:

 make AXTLS_DEFS_EXTRA="-Dabort=abort_ -DRT_MAX_PLAIN_LENGTH=1024 -DRT_EXTRA=8192"

bonus: in case you need more memory i recommend sth like the following tweak BEFORE you compile the image:

sed -i 's/STATIC char heap.*/STATIC char heap[44 * 1024];/g'  micropython/ports/esp8266/main.c

bonus: in case you need more firmware storage i recommend sth like the following tweak BEFORE you compile the image:

sed -i 's/    irom0_0_seg :  org = 0x40209000, len = .*/    irom0_0_seg :  org = 0x40209000, len = 0xa7000/g' micropython/ports/esp8266/boards/esp8266.ld

if you are interested how i solved to automate those tedious tasks for building a custom firmware for my own project take a look at this script

hope that helps