search

boblemaire / asyncHTTPrequest

asynchronous HTTP for ESP using ESPasyncTCP. Works like XMLHTTPrequest in JS.
GNU General Public License v3.0
65 stars 31 forks source link

Allow handling huge responses via ackLate #40

Closed mwalliczek closed 3 years ago

mwalliczek commented 3 years ago

To handle huge responses (bigger than the free heap) it is necessary to implement some kind of flow control. This PR addresses this issue using the ackLater function of ESPAsyncTCP.

boblemaire commented 3 years ago

Can you tell me more about why this is needed? I regularly get files that are 1 MB and transfer to SD with no flow control issues, so trying to understand what problem you are having that this solves. Educate me on the acklater function.

mwalliczek commented 3 years ago

Hi @boblemaire, I'm using an ESP 8266 in a complex scenario to play streamed music and display a web based image at a display, and without this fix I always got exceptions because of full memory. Apparently the received data took to long to be processed and so this library keeps receiving data and tried to allocate memory to store this data until all memory was reserved, which results in an exception and a crash of the ESP 8266.

So I need to stop the sender (web server) from sending any more data until all received data is processed and the memory is freed. The way to do so is stop sending ACK (acknowledges) messages for the received data, which will tell the sender to slow down the transmission speed. As described in RFC 1122, a host may delay sending an ACK response by up to 500 ms.

In fact the synchronous version of the HTTP request library (https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/WiFiClient.cpp) is doing the same, because it will also postpone sending acks until the synchronous receive function returns.

boblemaire commented 3 years ago

Hello @mwalliczek , I buy into the rational, there are just a couple of areas of concern:

First, is there a downside to just doing the ackLater all the time? Could there be a strategy to only acklater when there is already data in the buffer and keep track of the not-acked data?

I see that in _onData, you don't do it if there is an onData callback, but that callback may not result in a read of all the received data. It's unlikely, but why not just issue the ackLater and let the ack in read close the circle?

Lastly, I don't see any provision for ack when data is read into a String in responseText(). Pretty much the same ack(avail) as in the read, but needed.

boblemaire commented 3 years ago

No response from submitter. Closing PR. Will reconsider if concerns are addressed.