search

opendata-stuttgart / sensors-software

sourcecode for reading sensor data
574 stars 313 forks source link

Crash and reboot when API reply is too big #912

Open DeeKey opened 3 years ago

DeeKey commented 3 years ago

Recently there were complains from users who activated AirCMS API. Devices started to reboot all the time and disapeared from the map of sensor.community. The problem was related to AirCMS server hosting. Automatic DDoS protection was somehow activated and every time sensors were reading replay from AirCMS they received very large HTML page in reply (with text about DDoS measures). Device was not expecting it and crashes which leads to constant reboot.

To reproduce: 1) Activate API (in our case it was AirCMS) 2) On the server side of API create some sort of a large HTML reply (in the specific case of AirCMS it was DDoS guard warning message)

Expected behavior: 3) data will not be sent to this "faulty" API server and still will be sent to rest API endpoints.

Actual behavior: 3) The loop crashed, data not sent anywhere and device reboots.

DeeKey commented 3 years ago

Per discussion this bug is considered LOW priority and wont be fixed by core developers. This happened to the single API so far and not many users very affected. That is more server related problem then software. But still if someone would propose a pull request with solution which is not "memory heavy" it might be accepted mainstream!

ricki-z commented 3 years ago

There was a misconfiguration as the AirCMS API shouldn't use a DDOS protection. This was changed and the transmission should work again.

DeeKey commented 3 years ago

Adding information (user comments) from other sources which might help to fix this problem.

"HTTPClient::sendRequest() calls handleHeaderResponse() to parse the header lines and extract the status code and a few other items, but stops processing at the end of the header. I don't see anything else touching the payload until getString() is called. It looks like just checking that getString() has returned a valid string before using it ought to be sufficient."

"The underlying WiFi lib is receiving the complete response (the Stream object the HTTP client is working with). handleHeaderResponse() is reading this stream until the first blank line. But the stream object is consuming RAM until destroyed. There is even a debug message that there may be some data left in the buffer." DEBUG_HTTPCLIENT("[HTTP-Client][end] still data in buffer (%d), clean up.\n", _client->available());