roelandkluit commented 2 years ago

Describe the bug

When utilizing the AsyncHTTPS library there is a decrease of the available heap space with about 100 to about 200 bytes per https request until it runs out of heap space and crashes.

Steps to Reproduce

I modified the included example slightly to demonstrate the problem;

https://github.com/khoih-prog/AsyncHTTPSRequest_Generic/blob/main/examples/AsyncHTTPSRequest_ESP/AsyncHTTPSRequest_ESP.ino

Add a global variable;

uint previousHeapSize = ESP.getFreeHeap();

Modified void sendRequest() function;

void sendRequest()
{
  static bool requestOpenResult;

///***Add this
  uint heapSize = ESP.getFreeHeap();
  Serial.print("Current HeapFree: "); Serial.print(heapSize);
  if(previousHeapSize > heapSize )
  {
    Serial.print(" decrease: ");
    Serial.println(previousHeapSize - heapSize);  
  }else{
    Serial.print(" increase: ");
    Serial.println(heapSize - previousHeapSize);      
  }
  previousHeapSize = heapSize;  
///***End Add this

  if (request.readyState() == readyStateUnsent || request.readyState() == readyStateDone)

Commented out the Serial.print within requestCB to prevent printing of output


if (readyState == readyStateDone)
{
/*Serial.println(F("\n**************************************"));
Serial.println(request->responseText());
Serial.println(F("**************************************"));*/

request->setDebug(false);
}

5. Commented out the ticker1.attach within setup()
`//ticker1.attach(HEARTBEAT_INTERVAL, heartBeatPrint);`

7. Decreased the HTTPS_REQUEST_INTERVAL to 10 seconds to faster show the problem, although it also happens when you leave the interval to the example's default

// 300s = 5 minutes to not flooding

define HTTPS_REQUEST_INTERVAL 10 //300

### Expected behavior

Expected for the free heap size to be remain stable after initial initialization with minor variations both in increase and decrease of available free heap space.

### Actual behavior

The modified demo shows an ongoing decrease of available heap space:

11:05:07.402 -> Starting AsyncHTTPSRequest_ESP using ESP32_DEV 11:05:07.402 -> AsyncTCP_SSL v1.2.0 11:05:07.448 -> AsyncHTTPSRequest_Generic v2.0.1 11:05:07.529 -> Connecting to WiFi SSID: Wireless 11:05:08.036 -> ..... 11:05:10.006 -> AsyncHTTPSRequest @ IP : 172.16.16.241 11:05:10.053 -> Current HeapFree: 264648 increase: 102796 11:05:20.034 -> Current HeapFree: 199900 decrease: 64748 11:05:30.020 -> Current HeapFree: 199548 decrease: 352 11:05:40.037 -> Current HeapFree: 199376 decrease: 172 11:05:50.048 -> Current HeapFree: 199248 decrease: 128 11:06:00.037 -> Current HeapFree: 199060 decrease: 188 11:06:10.022 -> Current HeapFree: 198892 decrease: 168 11:06:20.021 -> Current HeapFree: 198764 decrease: 128 11:06:30.003 -> Current HeapFree: 198592 decrease: 172 11:06:40.039 -> Current HeapFree: 198408 decrease: 184 11:06:50.032 -> Current HeapFree: 198084 decrease: 324 11:07:00.034 -> Current HeapFree: 197932 decrease: 152 11:07:10.009 -> Current HeapFree: 197760 decrease: 172 11:07:20.017 -> Current HeapFree: 197604 decrease: 156 11:07:30.048 -> Current HeapFree: 197428 decrease: 176 11:07:40.021 -> Current HeapFree: 197272 decrease: 156 11:07:50.020 -> Current HeapFree: 197080 decrease: 192 11:08:00.015 -> Current HeapFree: 196956 decrease: 124 11:08:10.047 -> Current HeapFree: 196788 decrease: 168 11:08:20.016 -> Current HeapFree: 196612 decrease: 176 11:08:30.029 -> Current HeapFree: 196460 decrease: 152 11:08:40.022 -> Current HeapFree: 196300 decrease: 160 11:08:50.037 -> Current HeapFree: 196132 decrease: 168 11:09:00.032 -> Current HeapFree: 196016 decrease: 116 11:09:10.025 -> Current HeapFree: 195824 decrease: 192 11:09:20.011 -> Current HeapFree: 195656 decrease: 168 11:09:30.003 -> Current HeapFree: 195512 decrease: 144 11:09:40.040 -> Current HeapFree: 195320 decrease: 192 11:09:50.007 -> Current HeapFree: 195168 decrease: 152 11:10:00.013 -> Current HeapFree: 195008 decrease: 160 11:10:10.015 -> Current HeapFree: 194892 decrease: 116 11:10:20.002 -> Current HeapFree: 194700 decrease: 192 11:10:30.032 -> Current HeapFree: 194556 decrease: 144 11:10:40.003 -> Current HeapFree: 194380 decrease: 176 11:10:50.002 -> Current HeapFree: 194228 decrease: 152 11:11:00.011 -> Current HeapFree: 194056 decrease: 172 11:11:10.017 -> Current HeapFree: 193916 decrease: 140 11:11:20.025 -> Current HeapFree: 193744 decrease: 172 11:11:30.005 -> Current HeapFree: 193572 decrease: 172 11:11:40.001 -> Current HeapFree: 193424 decrease: 148 11:11:50.007 -> Current HeapFree: 193244 decrease: 180 11:12:00.002 -> Current HeapFree: 193108 decrease: 136 11:12:10.003 -> Current HeapFree: 192928 decrease: 180 11:12:20.018 -> Current HeapFree: 192796 decrease: 132 11:12:30.035 -> Current HeapFree: 192608 decrease: 188 11:12:40.003 -> Current HeapFree: 192452 decrease: 156 11:12:50.010 -> Current HeapFree: 192296 decrease: 156



### Debug and AT-command log (if applicable)

N/A

### Screenshots

N/A

### Information

> - Arduino IDE version 1.8.16) 
> - ESP32 Wrover device using Board manager: ESP32 v3.02
> - OS  Windows 11

khoih-prog commented 2 years ago

Hi @roel80

Thanks for your bug report and the detailed investigation. I'm able to duplicate the bug, and will send time to work on and fix that bug.

I'd appreciate it if you can help investigate, locate, fix the bug, post a PR, if possible, to speed up squash the bug much faster.

Regards,

nilber commented 2 years ago

I have the same problem, each Ticker cycle the memory is not released, until it finishes and all other requests are cashed.

How to clear memory ?

AsyncHTTPSRequest request;

pwbecker commented 2 years ago

+1 I am having the same issue. Finally tracked it down to AsyncHTTPSRequest. It leaves me a bit stuck - I cannot deploy the project with this issue. I see it has been an issue here for a while... does anyone know of alternative libraries which do async HTTPS requests?

Here is some debug output

08:53:45.074 -> Free memory minimum: 12196
08:53:45.329 -> [ATCP] _connected: error => closing
08:53:45.329 -> Servercomms: Callback with state readyStateDone, response -4: NOT_CONNECTED, len 3
08:53:47.064 -> Free memory minimum: 11920
08:53:48.347 -> [ATCP] _connected: error => closing
08:53:48.347 -> Servercomms: Callback with state readyStateDone, response -4: NOT_CONNECTED, len 3
08:53:49.054 -> Free memory minimum: 11784
08:53:51.045 -> Free memory minimum: 11784
08:53:51.335 -> [ATCP] _connected: error => closing
08:53:51.335 -> Servercomms: Callback with state readyStateDone, response -4: NOT_CONNECTED, len 3
08:53:53.068 -> Free memory minimum: 11784

pwbecker commented 2 years ago

Looking at the code, I notice there are 22 calls to "new" in AsyncHTTPSRequest_Impl_Generic.h. (e.g. see below) Could this be the issue? I am not sure if this is good programming practice...

I also notice that the non-SSL version, AsyncHTTPRequest_Generic, does not call new at all, and also does not have the same memory leak ...

_URL = new URL;
_URL->scheme = new char[8];
_URL->host = new char[hostEnd - hostBeg + 1];
_URL->path = new char[queryBeg - pathBeg + 1];
_URL->query = new char[url.length() - queryBeg + 1];

khoih-prog commented 2 years ago

I also notice that the non-SSL version, AsyncHTTPRequest_Generic, does not call new at all, and also does not have the same memory leak ...

I hope you know where to read the code

https://github.com/khoih-prog/AsyncHTTPRequest_Generic/blob/master/src/AsyncHTTPRequest_Impl_Generic.h#L1085

https://github.com/khoih-prog/AsyncHTTPRequest_Generic/blob/master/src/AsyncHTTPRequest_Impl_Generic.h#L1089

https://github.com/khoih-prog/AsyncHTTPRequest_Generic/blob/master/src/AsyncHTTPRequest_Impl_Generic.h#L1122

https://github.com/khoih-prog/AsyncHTTPRequest_Generic/blob/master/src/AsyncHTTPRequest_Impl_Generic.h#L1134

https://github.com/khoih-prog/AsyncHTTPRequest_Generic/blob/master/src/AsyncHTTPRequest_Impl_Generic.h#L1141

...

roelandkluit commented 2 years ago

Just submitted a proposed fix for memory leak. Issue seems to be caused by known crash of deletion of _URL object. This is caused by a to small buffer for the scheme in the URL object, that would not have the terminating null at the end of the string for HTTPS://

ref; https://github.com/khoih-prog/AsyncHTTPSRequest_Generic/blob/main/src/AsyncHTTPSRequest_Impl_Generic.h#L1280 https://github.com/khoih-prog/AsyncHTTPSRequest_Generic/blob/main/src/AsyncHTTPSRequest_Impl_Generic.h#L1106

roelandkluit commented 2 years ago

AsyncHTTPSRequest @ IP : 172.16.16.191 Current HeapFree: 264236 increase: 100224 09:08:19:923 -> Current HeapFree: 198972 decrease: 65264 09:08:29:842 -> Current HeapFree: 198800 decrease: 172 09:08:39:843 -> Current HeapFree: 198784 decrease: 16 09:08:49:874 -> Current HeapFree: 198784 no change 09:08:59:865 -> Current HeapFree: 198784 no change 09:09:09:891 -> Current HeapFree: 198800 increase: 16 09:09:19:889 -> Current HeapFree: 198796 decrease: 4 09:09:29:842 -> Current HeapFree: 198800 increase: 4 09:09:39:921 -> Current HeapFree: 198800 no change 09:09:49:864 -> Current HeapFree: 198808 increase: 8 09:09:59:884 -> Current HeapFree: 198808 no change 09:10:09:893 -> Current HeapFree: 198772 decrease: 36 09:10:19:875 -> Current HeapFree: 198756 decrease: 16 09:10:29:840 -> Current HeapFree: 198800 increase: 44 09:10:39:869 -> Current HeapFree: 198784 decrease: 16 09:10:49:845 -> Current HeapFree: 198780 decrease: 4 09:10:59:918 -> Current HeapFree: 198780 no change 09:11:09:888 -> Current HeapFree: 198752 decrease: 28 09:11:19:855 -> Current HeapFree: 198768 increase: 16 09:11:29:849 -> Current HeapFree: 198800 increase: 32 09:11:39:904 -> Current HeapFree: 198820 increase: 20 09:11:49:858 -> Current HeapFree: 198800 decrease: 20 09:11:59:856 -> Current HeapFree: 198820 increase: 20 09:12:09:889 -> Current HeapFree: 198800 decrease: 20 09:12:19:878 -> Current HeapFree: 198820 increase: 20 09:12:29:882 -> Current HeapFree: 198604 decrease: 216 09:12:39:928 -> Current HeapFree: 198604 no change 09:12:49:913 -> Current HeapFree: 198604 no change 09:12:59:854 -> Current HeapFree: 198612 increase: 8 09:13:09:895 -> Current HeapFree: 198604 decrease: 8 09:13:19:914 -> Current HeapFree: 198600 decrease: 4 09:13:29:853 -> Current HeapFree: 198604 increase: 4 09:13:39:895 -> Current HeapFree: 198592 decrease: 12 09:13:49:918 -> Current HeapFree: 198580 decrease: 12 09:13:59:844 -> Current HeapFree: 198616 increase: 36 09:14:09:861 -> Current HeapFree: 198608 decrease: 8 09:14:19:933 -> Current HeapFree: 198604 decrease: 4 09:14:29:845 -> Current HeapFree: 198624 increase: 20 09:14:39:876 -> Current HeapFree: 198604 decrease: 20 09:14:49:891 -> Current HeapFree: 198624 increase: 20 09:14:59:865 -> Current HeapFree: 198604 decrease: 20 09:15:09:833 -> Current HeapFree: 198624 increase: 20 09:15:19:857 -> Current HeapFree: 198584 decrease: 40 09:15:29:902 -> Current HeapFree: 198568 decrease: 16 09:15:39:886 -> Current HeapFree: 198576 increase: 8 09:15:49:903 -> Current HeapFree: 198584 increase: 8 09:15:59:886 -> Current HeapFree: 198588 increase: 4 09:16:09:834 -> Current HeapFree: 198608 increase: 20 09:16:19:933 -> Current HeapFree: 198588 decrease: 20 09:16:29:931 -> Current HeapFree: 198588 no change 09:16:39:901 -> Current HeapFree: 198600 increase: 12 09:16:49:934 -> Current HeapFree: 198580 decrease: 20 09:16:59:917 -> Current HeapFree: 196784 decrease: 1796 09:17:09:905 -> Current HeapFree: 198576 increase: 1792 09:17:19:861 -> Current HeapFree: 198580 increase: 4 09:17:29:904 -> Current HeapFree: 198596 increase: 16 09:17:39:935 -> Current HeapFree: 198564 decrease: 32 09:17:49:934 -> Current HeapFree: 198568 increase: 4 09:17:59:921 -> Current HeapFree: 198576 increase: 8 09:18:09:853 -> Current HeapFree: 198588 increase: 12 09:18:19:917 -> Current HeapFree: 198588 no change 09:18:29:848 -> Current HeapFree: 198588 no change 09:18:39:835 -> Current HeapFree: 198588 no change 09:18:49:924 -> Current HeapFree: 198584 decrease: 4 09:18:59:840 -> Current HeapFree: 198588 increase: 4 09:19:09:845 -> Current HeapFree: 198592 increase: 4 09:19:19:856 -> Current HeapFree: 198588 decrease: 4 09:19:29:941 -> Current HeapFree: 198588 no change 09:19:39:837 -> Current HeapFree: 198588 no change 09:19:49:879 -> Current HeapFree: 198588 no change 09:19:59:839 -> Current HeapFree: 198580 decrease: 8 09:20:09:846 -> Current HeapFree: 198568 decrease: 12 09:20:19:841 -> Current HeapFree: 198580 increase: 12 09:20:29:846 -> Current HeapFree: 198576 decrease: 4 09:20:39:880 -> Current HeapFree: 198588 increase: 12 09:20:49:830 -> Current HeapFree: 198584 decrease: 4 09:20:59:891 -> Current HeapFree: 198584 no change 09:21:09:828 -> Current HeapFree: 198580 decrease: 4 09:21:19:941 -> Current HeapFree: 198576 decrease: 4 09:21:29:905 -> Current HeapFree: 198588 increase: 12 09:21:39:902 -> Current HeapFree: 198576 decrease: 12 09:21:49:831 -> Current HeapFree: 198588 increase: 12 09:21:59:852 -> Current HeapFree: 198576 decrease: 12 09:22:09:866 -> Current HeapFree: 198600 increase: 24 09:22:19:909 -> Current HeapFree: 198588 decrease: 12 09:22:29:838 -> Current HeapFree: 198584 decrease: 4 09:22:39:912 -> Current HeapFree: 198584 no change

khoih-prog commented 2 years ago

Hi @roel80

The AsyncHTTPSRequest_Generic v2.2.0 has just been released. Your contribution is noted in Contributions and Thanks.

Please help with more rigorous tests to see if there is any more memory-related issues. As you know, there is so much a guy can do, especially with too many libraries to develop and take care, without the help of many more people.

Still waiting for more bug reports and fixes from you ;=}}

The library is better and better thanks to contribution users like you.

Best Regards,

Releases v2.2.0

Fix crash when turn-on debug or start
Fix memory leak. Check Having problemen with AsyncHTTPS memory leak #11 and Update for memleak in AsyncHTTPSRequest_Impl_Generic.h #13

khoih-prog / AsyncHTTPSRequest_Generic

Having problemen with AsyncHTTPS memory leak #11

Describe the bug

Steps to Reproduce

define HTTPS_REQUEST_INTERVAL 10 //300

Releases v2.2.0