HttpClient crashes after HTTP request

KenthJ commented 4 years ago

Board: ESP-WROOM-32 IDE name: Arduino IDE version 1.0.4 Flash Frequency: 80Mhz PSRAM enabled: No Upload Speed: 921600 Computer OS: Windows 10

Hi

Please bear with me, I'm completely new in this ESP32 world.

I'm trying to make a very simple HTTP request from my ESP32 towards a webserver which is running on a Raspberry Pi running Win 10 IOT, I have been banging my head for several hours and days trying to solve my problem below.

In most cases the code succeeds getting a response as expected, but when I call http.end() it crashes, sometimes with Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited), other times with (LoadProhibited).

If I instead use any other URLs instead of my own webserver, everything works fine. When I test the URL of my Raspberry PI webserver with a normal internet browser on my PC, or using Fiddler everything works fine too.

The strange thing is that it only crashes when it sends a http req to my homemade webserver, it works fine with all other webservers.

I double checked my http header from the server, and also added "Content-Type: text/html" "Date:....." and "Server:..." to the header.

Further more I checked the header and html response using a program called Fiddler and using FireFoxs Inspector tool, and I really can't see any problem with it, but there must be something in the response from my webserver causing the crash

#include <WiFi.h>
#include <HTTPClient.h>

const char* ssid = "mySSID";
const char* password =  "myPassword";

void setup(void)
{
  Serial.begin(9600);
  delay(4000);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Waiting for Wifi");
  }

  Serial.println("Wifi connected");
  delay(1000);
}

void loop()
{ 
  if ((WiFi.status() == WL_CONNECTED)) //Check the current connection status
  {  
    HTTPClient http;;

    bool httpInitResult = http.begin("http://192.168.0.105:485/esp32.html");

    http.addHeader("Content-Type", "text/html");

    if( httpInitResult == false )
    {
      Serial.println("http.begin() failed!");
    }
    else
    {
      int httpCode = http.GET();

      Serial.print("httpCode: ");
      Serial.println(httpCode);

      if (httpCode > 0) //Check for the returning code
      { 
        String payload = http.getString();

        Serial.print("Payload: ");
        Serial.println(payload);
      }
      else 
      {
        Serial.println("Error on HTTP request");
      }
    }
    Serial.println("Trace just before http.end()");
    http.end(); //Free the resources
    Serial.println("Trace just after http.end()");
  }
  delay(1000);
}

Output from Serial monitor with debug traces enabled:

15:36:22.978 -> Waiting for Wifi
15:36:22.978 -> Wifi connected
15:36:23.934 -> [V][HTTPClient.cpp:235] beginInternal(): url: http://192.168.0.105:485/esp32.html
15:36:24.034 -> [D][HTTPClient.cpp:276] beginInternal(): host: 192.168.0.105 port: 485 url: /esp32.html
15:36:24.135 -> [D][HTTPClient.cpp:1025] connect():  connected to 192.168.0.105:485
15:36:24.382 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
15:36:24.471 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Content-Length: 148'
15:36:24.552 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Content-Type: text/html'
15:36:24.633 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Server: rpi'
15:36:24.715 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Date: Thu, 23 Jan 2020 14:36:23 GMT'
15:36:24.796 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Connection: close'
15:36:24.877 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: ''
15:36:24.959 -> [D][HTTPClient.cpp:1158] handleHeaderResponse(): code: 200
15:36:25.000 -> [D][HTTPClient.cpp:1161] handleHeaderResponse(): size: 148
15:36:25.081 -> [D][HTTPClient.cpp:1295] writeToStreamDataBlock(): connection closed or file end (written: 148).
15:36:25.163 -> [D][HTTPClient.cpp:370] disconnect(): tcp stop
15:36:25.246 -> 
15:36:25.246 -> Payload: <!doctype html><html lang="en"><head><meta charset="utf-8"><title>Esp32</title></head><body><p>2020-01-23 15:36:23;23.13;4.63;2.3;</p></body></html>
15:36:25.399 -> Trace just before http.end()
15:36:25.399 -> [D][HTTPClient.cpp:383] disconnect(): tcp is closed
15:36:25.500 -> 
15:36:26.350 -> [V][HTTPClient.cpp:235] beginInternal(): url: http://192.168.0.105:485/esp32.html
15:36:26.450 -> [D][HTTPClient.cpp:276] beginInternal(): host: 192.168.0.105 port: 485 url: /esp32.html
15:36:26.504 -> [D][HTTPClient.cpp:1025] connect():  connected to 192.168.0.105:485
15:36:26.851 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
15:36:26.952 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Content-Length: 148'
15:36:27.005 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Content-Type: text/html'
15:36:27.106 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Server: rpi'
15:36:27.152 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Date: Thu, 23 Jan 2020 14:36:26 GMT'
15:36:27.253 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Connection: close'
15:36:27.353 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: ''
15:36:27.406 -> [D][HTTPClient.cpp:1158] handleHeaderResponse(): code: 200
15:36:27.453 -> [D][HTTPClient.cpp:1161] handleHeaderResponse(): size: 148
15:36:27.507 -> [D][HTTPClient.cpp:1295] writeToStreamDataBlock(): connection closed or file end (written: 148).
15:36:27.607 -> [D][HTTPClient.cpp:370] disconnect(): tcp stop
15:36:27.707 -> 
15:36:27.707 -> Payload: <!doctype html><html lang="en"><head><meta charset="utf-8"><title>Esp32</title></head><body><p>2020-01-23 15:36:25;23.13;4.63;2.3;</p></body></html>
15:36:27.839 -> Trace just before http.end()
15:36:27.886 -> Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
15:36:27.970 -> Core 1 register dump:
15:36:27.970 -> PC      : 0x4015d4d8  PS      : 0x00060430  A0      : 0x800d36db  A1      : 0x3ffb1e80  
15:36:28.109 -> A2      : 0x3ffb1f10  A3      : 0x00000002  A4      : 0x0000001c  A5      : 0x0000ff00  
15:36:28.155 -> A6      : 0x00ff0000  A7      : 0xff000000  A8      : 0x800d6591  A9      : 0x3ffb1e60  
15:36:28.256 -> A10     : 0x3ffbda34  A11     : 0x3f40240a  A12     : 0x00000002  A13     : 0x0000ff00  
15:36:28.356 -> A14     : 0x00ff0000  A15     : 0xff000000  SAR     : 0x00000004  EXCCAUSE: 0x0000001c  
15:36:28.456 -> EXCVADDR: 0x00000012  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xffffffff  
15:36:28.557 -> 
15:36:28.557 -> Backtrace: 0x4015d4d8:0x3ffb1e80 0x400d36d8:0x3ffb1ea0 0x400d37a9:0x3ffb1ec0 0x400d1ba6:0x3ffb1ee0 0x400d7775:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0
15:36:28.710 -> 
15:36:28.710 -> Rebooting...
15:36:28.710 ->     1⸮⸮⸮⸮VŌHT⸮⸮Xl@⸮[D][WiFiGeneric.cpp:337] _eventCallback(): Event: 0 - WIFI_READY
15:36:33.174 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 2 - STA_START
15:36:33.328 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 4 - STA_CONNECTED
15:36:33.629 -> [D][WiFiGeneric.cpp:337] _eventCallback(): Event: 7 - STA_GOT_IP
15:36:33.676 -> [D][WiFiGeneric.cpp:381] _eventCallback(): STA IP: 192.168.0.112, MASK: 255.255.255.0, GW: 192.168.0.1
15:36:34.130 -> Waiting for Wifi
15:36:34.130 -> Wifi connected
15:36:35.132 -> [V][HTTPClient.cpp:235] beginInternal(): url: http://192.168.0.105:485/esp32.html
15:36:35.232 -> [D][HTTPClient.cpp:276] beginInternal(): host: 192.168.0.105 port: 485 url: /esp32.html
15:36:37.038 -> [D][HTTPClient.cpp:1025] connect():  connected to 192.168.0.105:485
15:36:37.393 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
15:36:37.493 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Content-Length: 148'
15:36:37.540 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Content-Type: text/html'
15:36:37.641 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Server: rpi'
15:36:37.694 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Date: Thu, 23 Jan 2020 14:36:36 GMT'
15:36:37.794 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: 'Connection: close'
15:36:37.895 -> [V][HTTPClient.cpp:1123] handleHeaderResponse(): RX: ''
15:36:37.941 -> [D][HTTPClient.cpp:1158] handleHeaderResponse(): code: 200
15:36:37.995 -> [D][HTTPClient.cpp:1161] handleHeaderResponse(): size: 148
15:36:38.095 -> [D][HTTPClient.cpp:1295] writeToStreamDataBlock(): connection closed or file end (written: 148).
15:36:38.164 -> [D][HTTPClient.cpp:370] disconnect(): tcp stop
15:36:38.233 -> 
15:36:38.233 -> Payload: <!doctype html><html lang="en"><head><meta charset="utf-8"><title>Esp32</title></head><body><p>2020-01-23 15:36:36;23.13;4.63;2.3;</p></body></html>
15:36:38.370 -> Trace just before http.end()
15:36:38.404 -> Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
15:36:38.496 -> Core 1 register dump:
15:36:38.543 -> PC      : 0x54000000  PS      : 0x00060430  A0      : 0x800d36db  A1      : 0x3ffb1e80  
15:36:38.644 -> A2      : 0x3ffb1f10  A3      : 0x54000000  A4      : 0x0000001c  A5      : 0x0000ff00  
15:36:38.744 -> A6      : 0x00ff0000  A7      : 0xff000000  A8      : 0x8015d4de  A9      : 0x3ffb1e60  
15:36:38.797 -> A10     : 0x3ffbdaf0  A11     : 0x3f40240a  A12     : 0x00000002  A13     : 0x0000ff00  
15:36:38.898 -> A14     : 0x00ff0000  A15     : 0xff000000  SAR     : 0x00000004  EXCCAUSE: 0x00000000  
15:36:38.998 -> EXCVADDR: 0x00000000  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xffffffff  
15:36:39.098 -> 
15:36:39.098 -> Backtrace: 0x54000000:0x3ffb1e80 0x400d36d8:0x3ffb1ea0 0x400d37a9:0x3ffb1ec0 0x400d1ba6:0x3ffb1ee0 0x400d7775:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0
15:36:39.246 -> 
15:36:39.246 -> Rebooting...
15:36:39.299 ->

Output from ESP Exception decoder:

Decoding stack results
0x4015d4d8: HTTPClient::connected() at C:\Users\kfj41\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src\HTTPClient.cpp line 395
0x400d36d8: HTTPClient::disconnect(bool) at C:\Users\kfj41\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src\HTTPClient.cpp line 359
0x400d37a9: HTTPClient::end() at C:\Users\kfj41\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src\HTTPClient.cpp line 347
0x400d1ba6: loop() at C:\Users\kfj41\Documents\Arduino\Display SPI test\TFT_eSPI_WiFi_Http_req_5 without display\TFT_eSPI_WiFi_Http_req_4/TFT_eSPI_WiFi_Http_req_4.ino line 60
0x400d7775: loopTask(void*) at C:\Users\kfj41\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\cores\esp32\main.cpp line 19
0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

lbernstone commented 4 years ago

If you change the http.end to: if (http.connected()) http.end(); do you get an exception?

KenthJ commented 4 years ago

if (http.connected()) http.end();

Ibernstone, I tried your suggestion, but unfortunately it didn't solve the problem.

Adding more traces shown that the ESP32 often crashes before the http.end() is called at all

Could the error be related to that I am using a fixed IP address instead of a domain name, and no DNS lookup is done?

KenthJ commented 4 years ago

Anybody able to help here?

I would really appriciate some help with this.

Thanks in advance

liebman commented 4 years ago

I can't reproduce this (with the URL changed to https://jigsaw.w3.org/HTTP/300/Overview.html) Can you share how the server is configured so I can try to set that up here? (its a Raspberry Pi?)

KenthJ commented 4 years ago

Liebman, yes it is a Raspberry Pi running Windows 10 IoT

I inserted details below, do you need anything else?

Header from Fiddler HTTP/1.1 200 Connection Established Content-Length: 148 Content-Type: text/html Server: rpi Date: Sun, 02 Feb 2020 15:54:59 GMT Connection: close

Raw content from Fiddler: HTTP/1.1 200 Connection Established Content-Length: 148 Content-Type: text/html Server: rpi Date: Sun, 02 Feb 2020 15:54:59 GMT Connection: close <!doctype html>Esp32

2020-02-02 16:54:59;23.88;3.56;2.3;

liebman commented 4 years ago

OK, I won't be loading Windows IoT on my Pi. I'll set the web server on it up with something that looks similar-ish.

KenthJ commented 4 years ago

Great, I really appreciate that.

The raw content in my original post doesn't look correctly, since the html part isn't shown as I have written it, but I guess that is not so important, since I have validated the html content

liebman commented 4 years ago

I've been unable to duplicate the crash with current git version and apache2 configured on a Pi with: (/etc/apache2/sites-enabled/001-esp32.conf)

Listen 485
<VirtualHost *:485>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

and the html file: (/var/www/html/esp32.html)

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title>esp32 test</title>
  </head>
  <body>
    This is a test. 
  </body>
</html>

KenthJensen commented 4 years ago

Liebman

Thank you so much for your support, I really appreciate it.

I have an extra Raspberry PI laying around, so I will try your example above on that one, it looks fairly simple.

Have you run it on Rasbbian on your Raspberry PI, and can I just install it via NOOBS?

liebman commented 4 years ago

I used Raspbian.

KenthJ commented 4 years ago

After a lot of trial and error I managed to get the Apache webserver up and running on Raspbian, and I agree, things worked fine using this webserver, the ESP32 didn't crash here.

I noticed that my old server used the following in the http header: Connection: close

But the Apache webserver used: Keep-Alive: timeout=5, max=100 Connection: Keep-Alive

I tried using the same in the header from my original webserver, and voila, now the ESP32 didn't crash anymore.

This is a workaround I easily can live with permanently, but I guess it's something the httpCLient should be robust for, and not crash.

But thank you very much Liebman, so nice to get the problem solved.

liebman commented 4 years ago

I'll change the apache configuration I used to disable keep-alive and see if I can reproduce the issue.

liebman commented 4 years ago

Even with keep-alive disabled in apache I can't reproduce the crashes.

Zenock commented 4 years ago

Same issue. I'm not sure how to change the header from my server. :-( Mostly on http.end() but occasionally elsewhere.

Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled. Core 1 register dump: PC : 0x4015c69c PS : 0x00060630 A0 : 0x800d329a A1 : 0x3ffb1ef0
A2 : 0x3ffc106c A3 : 0x00000000 A4 : 0x00000004 A5 : 0x0000ff00
A6 : 0x00ff0000 A7 : 0xff000000 A8 : 0x800d50c1 A9 : 0x3ffb1ed0
A10 : 0x3ffba860 A11 : 0x3f4015f2 A12 : 0x00000002 A13 : 0x0000ff00
A14 : 0x00ff0000 A15 : 0xff000000 SAR : 0x0000001b EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000010 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xffffffff

Backtrace: 0x4015c69c:0x3ffb1ef0 0x400d3297:0x3ffb1f10 0x400d3309:0x3ffb1f30 0x400d1a32:0x3ffb1f50 0x400d1b5a:0x3ffb1f90 0x400d62bd:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0

Rebooting...

I'm using a hosted web server. Not sure I can change the header. Wouldn't know how if I could.

antonmeyer commented 4 years ago

Looks like I have a similar Problem. but in my case it triggers InstrFetchProhibited. the backtrace initial pointed to the display u8x8. but esp32 crashes also without display.

mst-at commented 4 years ago

Hello, I'm also experiencing similar issues. Board: DOIT ESP32 DEVKIT V1 IDE name: Arduino IDE version 1.8.12 Flash Frequency: 80Mhz Upload Speed: 921600 Computer OS: Ubuntu 18.04.4 LTS

My application is to POST a json file to a webpage. Everything works fine using arduino-eps32 v1.0.2 but when using v1.0.3 and 1.0.4 it fails with:

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x400d4bba  PS      : 0x00060430  A0      : 0x800d1cc0  A1      : 0x3ffb1e20  
A2      : 0x3ffb1f10  A3      : 0x00000000  A4      : 0x0000000a  A5      : 0x3ffc1670  
A6      : 0x3f40327e  A7      : 0xffffffff  A8      : 0x800d695c  A9      : 0x3ffb1e00  
A10     : 0x3ffcb6ac  A11     : 0x3ffbf448  A12     : 0x0000000a  A13     : 0x3f40239f  
A14     : 0x00000000  A15     : 0x0000002e  SAR     : 0x00000004  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000038  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xffffffff  

Backtrace: 0x400d4bba:0x3ffb1e20 0x400d1cbd:0x3ffb1e40 0x400d7739:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0

Sometimes it works without problem on first run but fails in the second loop (it also seems to depend on completely unrelated code executed after http.end()).

Minimum example of the code showing the issue is:

#include "WiFi.h"
#include <HTTPClient.h>
#include "base64.h"

// WIFI settings
const char* ssid = "MySSID";
const char* password = "MyPassword";

// website authentication
String authUsername = "MyUser";
String authPassword = "MyPassword";

void connectWiFi(){
  // Connect to Wi-Fi
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    Serial.println("Connecting to WiFi..");
    delay(1000);
  }
}

void setup() {
  // put your setup code here, to run once:
  // Serial port for debugging purposes
  Serial.begin(115200);

  // Connect to Wi-Fi
  connectWiFi();
}

void loop() {

  if(WiFi.status()== WL_CONNECTED){   //Check WiFi connection status
    Serial.println("++++++++++++++++++++++ Starting round ++++++++++++++++++++++");
    char buffer[100] = "{\"location\":\"esp32\",\"temperature\":42.42,\"humidity\":42.42}";
    Serial.println(buffer);

   // transfer
   HTTPClient http; 

   http.begin("http://sub.domain.com/writeData.php");
   http.addHeader("Content-Type", "application/json");
   String auth = base64::encode(authUsername + ":" + authPassword);
   http.addHeader("Authorization", "Basic " + auth);

   int httpResponseCode = http.POST(buffer);
   String response = http.getString();
   delay(5000);
   Serial.println(httpResponseCode);
   Serial.println(response);
   Serial.println("---------------- Done with round ---------------------");
   http.end(); 
   delay(5000);
   Serial.println(httpResponseCode);
   Serial.println("----------- Really done with round -----------------");

   if ( httpResponseCode == 200 ){
       Serial.println("Success!");
       delay(5000);
   }
  }
  Serial.println("Sleeping...");
  delay(5000);
  Serial.println("############# Totally done with round ############");
}

The response from the webserver to the POST request is: Header:

HTTP/1.1 200 OK
Date: Sat, 04 Apr 2020 18:27:54 GMT
Server: Apache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

Message:

New records created successfully
writeData.php

I would appreciate any hint on how this issue could be solved, I already spent hours on it with no outcome. Best regards, Martin

ThomasWaldmann commented 4 years ago

https://github.com/espressif/arduino-esp32/commit/7d7824701fe5e22f08555d3e1ce3180a922b2151#r39308827 maybe related?

update: that was changed again afterwards: https://github.com/espressif/arduino-esp32/commit/f4acac4c2bf83d76f49241489e24fc1d6bbb64e7#diff-39b6d5e36cff20acc96b42ef19ad4deb

recent fixes, needs checking: https://github.com/esp8266/Arduino/pull/6476

if have the impression that the esp8266 HTTPClient is getting fixes, but the esp32 incarnation misses some. @Jeroen88 could that be the root cause of this issue?

stale[bot] commented 4 years ago

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

ThomasWaldmann commented 4 years ago

Maybe usage of stalebot should be reconsidered. It is pointless and even harmful to close unfixed bugs (or like in this case, make people making pointless posts just to avoid closure).

stale[bot] commented 4 years ago

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

wrybread commented 3 years ago

I get the same crash whenever I connect to a webserver that sends the header "connection: close". The crash happens on the second iteration of the loop (the second time connecting). It doesn't happen when I connect to a webserver that doesn't send that header.

Does anyone have any ideas?

Unfortunately I don't have control over the webserver, it's the webserver built into a weather station.

Stripped down code:


// Import required libraries for wifi
#ifdef ESP32
  #include <WiFi.h>
  #include <WiFiMulti.h>
#else
  # these are untested, I've only tested with an ESP32
  #include <Arduino.h>
  #include <ESP8266WiFi.h>
  #include <Hash.h>
  #include <ESPAsyncTCP.h>
#endif

#include <HTTPClient.h>

// init wifimulti (lets us specify multiple ssid's)
WiFiMulti wifiMulti;

void setup(){
  Serial.begin(115200);
  delay(10);
  Serial.println("Starting up!");

  // specify SSIDs (these must be 2.4ghz networks)
  wifiMulti.addAP("put ssid here", "put password here");  

  // (Note that must run wifiMulti.run() here or it will reset in a loop...)
  Serial.println("Connecting Wifi...");
  if(wifiMulti.run() == WL_CONNECTED) {
      Serial.println("");
      Serial.print("WiFi connected! ");
      Serial.print( WiFi.SSID().c_str() );
      Serial.print(" ");
      Serial.print("IP address: ");
      Serial.println(WiFi.localIP());
  }
}

void loop(){ 

  delay(1000);

  if(wifiMulti.run() == WL_CONNECTED) {

    HTTPClient http;  

    // crashes! (because sends header "connection:close")
    String airlinkURL = "http://10.10.10.51/v1/current_conditions";      

    // works! (because doesn't send header "connection:close") (it sends "Connection: Keep-Alive")
    //String airlinkURL = "http://sinkingsensation.com/aqi/get_json.php";      

    Serial.print("Getting data from Airlink: ");
    Serial.println(airlinkURL);
    http.begin(airlinkURL.c_str());
    int httpResponseCode = http.GET();

    if (httpResponseCode>0) {
      Serial.print("HTTP Response code: ");
      Serial.println(httpResponseCode);

      String payload = http.getString();

      // causes crash?
      //Serial.println(payload);

    }
    else {
      Serial.print("Error connecting to the AirLink: ");
      Serial.println(httpResponseCode);
    }

    http.end();

    delay(5000);

  } // end if connected

}

wrybread commented 3 years ago

Aha, I found a workaround. It's ugly, but it seems to work.

I had to modify HTTPClient.cpp, which on my Windows system is in C:\Users\USERNAME\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src.

Comment out these lines, or change the conditional or comment out "_canReuse = false;":

                if(_canReuse && headerName.equalsIgnoreCase("Connection")) {
                    if(headerValue.indexOf("close") >= 0 && headerValue.indexOf("keep-alive") < 0) {
                        _canReuse = false;
                    }
                }

I'm glad to test any changes or alternative versions with this webserver, if anyone needs just let me know.

dev-techshlok commented 3 years ago

Thank you @wrybread. Your method worked for me.

wimjanse commented 3 years ago

Thank you @wrybread. I had problems with crashing http.end(), but your method solved the issue

apreb commented 3 years ago

Hi, was having the same exception issues and managed to solve it by using a different begin function:

const char *_url = "http://192.168.1.1/getstuff";
WiFiClient client;
HTTPClient http;
http.begin(client,_url);

if( http.GET() )
{
  String payload = http.getString();
  Serial.println(payload);
  payload = String();
} 
http.end();

ThomasWaldmann commented 3 years ago

@apreb that rather looks like some testing code than a modified begin function. Copy&Paste gone wrong?

apreb commented 3 years ago

not really, the code snippet is what is needed to request a GET on the resource and printing back results to the console, without crashing.

riraosan commented 3 years ago

Aha, I found a workaround. It's ugly, but it seems to work.

I had to modify HTTPClient.cpp, which on my Windows system is in C:\Users\USERNAME\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src.

Comment out these lines, or change the conditional or comment out "_canReuse = false;":
                if(_canReuse && headerName.equalsIgnoreCase("Connection")) {
                    if(headerValue.indexOf("close") >= 0 && headerValue.indexOf("keep-alive") < 0) {
                        _canReuse = false;
                    }
                }
I'm glad to test any changes or alternative versions with this webserver, if anyone needs just let me know.

I came across the same event today, so I searched and arrived at this issue. I confirmed that the reboot was 100%( request times =178/178) gone with the above fix. Thank you very much.

--- Devices esp32(Http Server and REST API(getting BME280 info)) <--> esp32 (HttpClient. http.GET() 178 times)

Acorn221 commented 3 years ago

Aha, I found a workaround. It's ugly, but it seems to work.

I had to modify HTTPClient.cpp, which on my Windows system is in C:\Users\USERNAME\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src.

Comment out these lines, or change the conditional or comment out "_canReuse = false;":
                if(_canReuse && headerName.equalsIgnoreCase("Connection")) {
                    if(headerValue.indexOf("close") >= 0 && headerValue.indexOf("keep-alive") < 0) {
                        _canReuse = false;
                    }
                }
I'm glad to test any changes or alternative versions with this webserver, if anyone needs just let me know.

Thankyou!

garageeks commented 3 years ago

Hi to all, I have this crash as well while sending a POST request to the wrong server, which returns a 404 error. I tried @wrybread hack but didn't solve the issue. Of course I will fix the server issue, but I wish the ESP32 side was more fault tolerant. The server headers output is:

HTTP/1.1 404 Not Found
Connection: Keep-Alive
Transfer-Encoding: chunked
Keep-Alive: timeout=20
X-Frame-Options: SAMEORIGIN
Content-Type: text/html
X-DNS-Prefetch-Control: off

The code is compiled in PlaftormIO, using framework ESP32 1.0.4

Here's the output:

[HTTP] begin...
http://192.168.1.1/table-update
[HTTP] POST...
[E][WiFiClient.cpp:436] read(): fail on fd 54, errno: 104, "Connection reset by peer"
Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x40155e84  PS      : 0x00060230  A0      : 0x8015624c  A1      : 0x3ffb1b90  
A2      : 0x00000000  A3      : 0x3ffb1bef  A4      : 0x00000001  A5      : 0x00000001  
A6      : 0x3ffc1904  A7      : 0x00000068  A8      : 0x00000000  A9      : 0x00000000  
A10     : 0x0022c4f3  A11     : 0x40085fe4  A12     : 0x00000050  A13     : 0x00000000  
A14     : 0x0022c4f3  A15     : 0x00000000  SAR     : 0x0000000a  EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000008  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff

Backtrace: 0x40155e84:0x3ffb1b90 0x40156249:0x3ffb1bb0 0x40185dce:0x3ffb1be0 0x400d6b9d:0x3ffb1c10 0x400d6c35:0x3ffb1c30 0x400d5911:0x3ffb1c50 0x400d60a3:0x3ffb1cd0 0x400d610f:0x3ffb1d10 0x400d6133:0x3ffb1d30 0x400d3121:0x3ffb1d50 0x400d3e1e:0x3ffb1f90 0x400d9531:0x3ffb1fb0 0x40089c6d:0x3ffb1fd0

and here's the decoded backtrace:

PC: 0x40155e84: WiFiClientRxBuffer::read(unsigned char*, unsigned int) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src\WiFiClient.cpp line 107
EXCVADDR: 0x00000008

Decoding stack results
0x40155e84: WiFiClientRxBuffer::read(unsigned char*, unsigned int) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src\WiFiClient.cpp line 107
0x40156249: WiFiClient::read(unsigned char*, unsigned int) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src\WiFiClient.cpp line 434
0x40185dce: WiFiClient::read() at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\WiFi\src\WiFiClient.cpp line 345
0x400d6b9d: Stream::timedRead() at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\cores\esp32\Stream.cpp line 36
0x400d6c35: Stream::readStringUntil(char) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\cores\esp32\Stream.cpp line 287
0x400d5911: HTTPClient::handleHeaderResponse() at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\HTTPClient\src\HTTPClient.cpp line 1118
0x400d60a3: HTTPClient::sendRequest(char const*, unsigned char*, unsigned int) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\HTTPClient\src\HTTPClient.cpp line 573
0x400d610f: HTTPClient::POST(unsigned char*, unsigned int) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\HTTPClient\src\HTTPClient.cpp line 493
0x400d6133: HTTPClient::POST(String) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\HTTPClient\src\HTTPClient.cpp line 498
0x400d3121: wifiTransmission() at C:/Users/proprietario/zzz/projects/Active Projects/xxx/Electronics/Firmware/yyy/Table-FW/src/main.ino line 400
0x400d3e1e: loop() at C:/Users/proprietario/zzz/projects/Active Projects/xxx/Electronics/Firmware/yyy/Table-FW/src/main.ino line 316
0x400d9531: loopTask(void*) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\cores\esp32\main.cpp line 19
0x40089c6d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

garageeks commented 3 years ago

Keep on crashing :( It's doing nothing fancy, just a HTTP POST request to a server every 30 seconds. It occurred twice today, one after 154 successful HTTP requests and another time with 39 successful requests, in both cases the backtrace is identical.

It seems to crash at this instruction at line 1125 if(headerLine.startsWith("HTTP/1.")) { just before there is a log_v that prints the received header, will enable verbose logging to see if it's a malformed header.

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x401708dc  PS      : 0x00060830  A0      : 0x800df992  A1      : 0x3ffb1130  
A2      : 0x3ffb1290  A3      : 0x00000000  A4      : 0x00000005  A5      : 0x00000020  
A6      : 0x00000020  A7      : 0xff000000  A8      : 0x00000000  A9      : 0x3ffc1d9c  
A10     : 0x3ffd1568  A11     : 0x00060023  A12     : 0x00060021  A13     : 0x0000ff00  
A14     : 0x00ff0000  A15     : 0xff000000  SAR     : 0x0000000a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000010  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffff6

PC: 0x401708dc
EXCVADDR: 0x00000010

Decoding stack results
0x400df98f: HTTPClient::handleHeaderResponse() at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\HTTPClient\src\HTTPClient.cpp line 1125
0x400dfa01: HTTPClient::writeToStreamDataBlock(Stream*, int) at C:\Users\proprietario\.platformio\packages\framework-arduinoespressif32\libraries\HTTPClient\src\HTTPClient.cpp line 1304
0x400d67de: wifiTransmission() at .pio/libdeps/esp32dev/ArduinoJson/src/ArduinoJson/Document/JsonDocument.hpp line 226
0x400d695a: mainThread() at C:/Users/proprietario/Documents/PlatformIO/Projects/PSC-Gen2-FirmwareESP32/src/PSC-Gen2-FirmwareESP32.ino line 538
0x400d69e9: mainThread() at C:/Users/proprietario/Documents/PlatformIO/Projects/PSC-Gen2-FirmwareESP32/src/PSC-Gen2-FirmwareESP32.ino line 565
0x400e44bd: size_of_encoded_value at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libgcc/unwind-pe.h line 75
0x40089a69: uxPortCompareSetExtram at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.inc.h line 105

Jeroen88 commented 3 years ago

Just a wild guess, could you increase the time out by calling HTTPClient::setTimeout(30000), this will increase the time out to 30 seconds

cpainchaud commented 3 years ago

I am being hit hard by this issue, stack trace

0 0x400e1235:0x3ffe7f00 in WiFiClient::connect(char const, unsigned short, int) at /framework-arduinoespressif32/libraries/WiFi/src/WiFiClient.cpp:572 1 0x400f0876:0x3ffe7f30 in HTTPClient::connect() at/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:925 2 0x400f08f7:0x3ffe7f60 in HTTPClient::sendRequest(char const, unsigned char, unsigned int) at /framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:925 3 0x400f0a73:0x3ffe7fb0 in HTTPClient::GET() at /framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:925 4 0x400d7777:0x3ffe7fd0 in RFLink::OTA::downloadFromUrl(char const) at RFLink/13_OTA.cpp:29 (discriminator 3)

This with version: Arduino IDE Version : 10805 ESP SDK version: v3.3.4-432-g7a85334d8

EDIT: my stack trace being different it may be another issue for which i am going to create a new issue

me-no-dev commented 3 years ago

@cpainchaud are you using board manager version of this repo or git? There were some changes recently to String and HTTP/Client

cpainchaud commented 3 years ago

@me-no-dev PIO framework update My pio entry: [ESP32_base] platform = espressif32@3.1.0

slim-bean commented 3 years ago

I was seeing this issue or a related issue, for me it happened after changing my web server configuration. However I made a few changes so I don't know which may have been the most impactful, but I suspect it was the change to use HTTPS instead of HTTP (I also changed the routing to go through a different set of reverse proxies)

I would see crashes every few minutes or every few hours, although the place it crashed wasn't consistent with the backtraces often completely different, but here are two examples:

0x401619c5: std::_Function_base::~_Function_base() at c:\users\ed\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-97-gc752ad5-5.2.0\xtensa-esp32-elf\include\c++\5.2.0/functional line 1830
0x400d304d: HTTPClient::setURL(String const&) at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\HTTPClient\src\HTTPClient.cpp line 1439
0x400d30d1: HTTPClient::connect() at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\HTTPClient\src\HTTPClient.cpp line 1083
0x400d1335: submitSensors(float, float, float) at \\wsl$\Ubuntu\home\ed\projects\timefidget\arduino\fidgobject/fidgobject.ino line 88
0x400d1445: _GLOBAL__sub_I_msa() at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\cores\esp32/IPAddress.h line 94
0x400d7ad5: setCpuFrequencyMhz at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\cores\esp32\esp32-hal-cpu.c line 136
0x4008a03e: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

and

0x40161c02: std::__find_if    > >(unsigned int *, unsigned int *, __gnu_cxx::__ops::_Iter_pred   >, std::random_access_iterator_tag) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/stl_algo.h line 153
0x400d3ec2: WiFiClient::connect(IPAddress, unsigned short, int) at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\WiFi\src\WiFiClient.cpp line 266
0x400d3f6d: postToSysQueue(system_prov_event_t*) at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\WiFi\src\WiFiGeneric.cpp line 57
0x400d4131: std::function ::function(std::function  const&) at c:\users\ed\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-97-gc752ad5-5.2.0\xtensa-esp32-elf\include\c++\5.2.0/functional line 2238
0x400d3a72: WiFiClient::setTimeout(unsigned int) at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\WiFi\src\WiFiClient.cpp line 295
0x400d341a: HTTPClient::HTTPClient() at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\HTTPClient\src\HTTPClient.cpp line 97
0x400d348e: HTTPClient::HTTPClient() at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\HTTPClient\src\HTTPClient.cpp line 97
0x400d3607: HTTPClient::begin(String, char const*) at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\HTTPClient\src\HTTPClient.cpp line 194
0x400d362b: HTTPClient::begin(String) at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\libraries\HTTPClient\src\HTTPClient.cpp line 224
0x400d130f: submitSensors(float, float, float) at \\wsl$\Ubuntu\home\ed\projects\timefidget\arduino\fidgobject/fidgobject.ino line 84
0x400d1445: _GLOBAL__sub_I_msa() at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\cores\esp32/IPAddress.h line 94
0x400d7ad5: setCpuFrequencyMhz at C:\Users\ed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5\cores\esp32\esp32-hal-cpu.c line 136
0x4008a03e: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

I wish I had saved more, they were often different :(

So for the interesting part (and sorry it looks like I didn't save this backtrace either), the last one I checked crashed in my code where I had this line (marked with >>>>>>):

if (httpCode > 0) {
        Serial.printf("timefidget [HTTP] POST...  Code: %d  Response: ", httpCode);
>>>>>>> https.writeToStream(&Serial);
        Serial.println();
      } else {
        Serial.printf("timefidget [HTTP] POST... Error: %s\n", https.errorToString(httpCode).c_str());
      }

So for kicks I commented out that line and I haven't had a crash in days 😮

Edit:

I wanted to comment that some of the other solutions I had seen around creating a new HTTPClient for every http operation did seem to work however this wouldn't work for me as it then forces a new connection negotiation on every request (instead of reusing the connection), this was too slow for the nature of what I'm trying to do so I really need to be able to keep the HTTPClient and reuse it between subsequent calls.

By removing that line to write to the serial port I've been able to use the same HTTPClient with no crashes now (at least that I've seen) for several days.

podaen commented 3 years ago

response-status

ppescher commented 3 years ago

@garageeks I had a very similar issue (same verbose error messages) that was caused by the server closing the connection before the HTTPClient class can read the response.

The problem is that the client read() function calls stop() when the socket is closed remotely and that invalidates the receive buffer object. I fixed it by checking the pointer before using it:

int WiFiClient::read(uint8_t *buf, size_t size)
{
    int res = -1;
    if (_rxBuffer) {
        res = _rxBuffer->read(buf, size);
        if(_rxBuffer->failed()) {
            log_e("fail on fd %d, errno: %d, \"%s\"", fd(), errno, strerror(errno));
            stop();
        }
    }
    return res;
}

The same "bug" is present inside peek() too, but not in available().

ThomasWaldmann commented 3 years ago

@ppescher so you tried the original code, reproduced the issue and the issue is definitely gone with your modified code?

If so, can you create a patch or open a pull request?

ppescher commented 3 years ago

@ThomasWaldmann Apparently yes!

I'm not sure this is the same problem as in the original post, but it looks definitely the same as @garageeks reported.

In my case it happened only with a custom server that was not replying correctly to the ESP32 client, but the "bug" is there and the crash may also happen in other occasions. Since it is related to a half-closed socket, it may have something to do with "Connection: close" or "keepalive" headers and connection reuse settings.

There is nothing stopping an application code from calling read() after the socket has been closed or has flagged an error, unless you first use available() or connected() to check. And if you use library code, as HttpClient, you may not know if this good behavior has been always implemented.

The proposed fix is certainly improving robustness to unexpected conditions, like a connection that breaks prematurely or a server with a less "standard" implementation.

Patch is attached.

0001-Fix-crash-due-to-WiFiClient-read-after-stop.patch.txt

garageeks commented 3 years ago

@ppescher Thanks to combination of server changes and less frequent HTTP transmissions my code seems to be fairly stable. But we only see the tip of the iceberg composed by all these libraries, it looks like a race condition (i.e. by using it with AsyncWebServer) that trigger crashes. Your change make totally sense, good job! I'll try it You should open a pull request and see what the maintainers think about it

stale[bot] commented 3 years ago

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

ThomasWaldmann commented 3 years ago

If nobody solved it, there is no reason for a stupid stale bot just closíng this.

@ppescher @me-no-dev can you have a look?

stale[bot] commented 3 years ago

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

ppescher commented 3 years ago

@ThomasWaldmann I'm not the OP, but I had a similar issue that I solved with the patch I submitted. Merged by #5197 Not sure if it fixes the original issue too.

stale[bot] commented 2 years ago

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

ThomasWaldmann commented 2 years ago

If nobody solved it, there is no reason for a stupid stale bot just closíng this.

stale[bot] commented 2 years ago

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

VojtechBartoska commented 2 years ago

Hello, adding this issue to Roadmap so we will investigate this more.

me-no-dev commented 2 years ago

Can someone please re-test this against the latest version (2.0.3 currently)

espressif / arduino-esp32

HttpClient crashes after HTTP request #3659