HTTP header timeout

[aessig]

Hi, Thanks for the library, it looks really promising. Unfortunately, I can't make it work. Using the provided example, I have the following issue: "HTTP header timeout" The ESP8266 (Wemos Pro) is not getting any info. Do you have any suggestion?

[Bodmer]

This suggests the library is not getting a response from the DarkSky website after 5 seconds (5000 milliseconds) of waiting.

You could try increasing the waiting time here to say 20 seconds if you have a very slow internet link.

[aessig]

Thanks for your reply. The issue is not with the internet link. I have tried the library with an ESP32 board and it works perfectly. As I understood the library should also work with the ESP8266 too. Is there anything to look at especially for the ESP8266?

[Bodmer]

Check which ESP8266 board package is loaded in the IDE, I use 2.4.2.

I just tried the examples on my NodeMCU board and they seem to work fine.

[Leonos]

Same here, ESP32 works flawlessly, ESP8266 does not. Using Adafruit ESP8266, board package 2.4.2. Changing timing does not help, Perhaps it is a memory shortage because when using the Apixu connector (instead of DarkSky) I get 2 of the 5 (MAX_DAYS) forecasts, the last 3 are 'empty' - date is 01 Jan 1970, no text, all numeric values are 0, no sun and moon data. I get all 5 using an ESP32.

[Leonos]

This suggests the library is not getting a response from the DarkSky website after 5 seconds (5000 milliseconds) of waiting.

You could try increasing the waiting time here to say 20 seconds if you have a very slow internet link.

I just tested the unaltered My_DarkSkyWeather_Test example on an ESP8266 and it comes back with a 'Connection failed" which is already in line 226 - that's well before the 5000 ms of waiting in line 270. It's very strange because the same URL in a browser gives immediate response, and so does an ESP32, but of course, that uses a different WifiClientSecure.

In any case, it's strange that you do get results on an ESP8266.

[Bodmer]

Three things to try:

1. Can you try the JSON_Decoder example called Space_Station and see if that works on an ESP8266.

2. In the DarkSkyWeather library here is an option to minimise the data point set gathered and hence reduce the memory needed, Uncomment the line here.

3. In the DarkSkyWeather library there is an option to use a different http secure library called BearSSL that is more memory efficient, to enable this line by commenting out line 25 and uncommenting line 26 here.

[Leonos]

1 Space_Station works flawlessly here on an ESP8266. (By the way, thanks again for this example, it really helped me understand the dynamics of JSON_Decoder et al.) 2 Uncommenting that line gives a lot of errors:

In file included from C:\Users\User\AppData\Local\Temp\arduino_modified_sketch_849556\My_DarkSkyWeather_Test.ino:18:0: C:\Users\User\Documents\Arduino\libraries\DarkSkyWeather/DarkSkyWeather.h:30:22: error: 'DSW_current' has not been declared bool getForecast(DSW_current current, DSW_hourly hourly, DSW_daily daily, ^ C:\Users\User\Documents\Arduino\libraries\DarkSkyWeather/DarkSkyWeather.h:30:44: error: 'DSW_hourly' has not been declared bool getForecast(DSW_current current, DSW_hourly hourly, DSW_daily daily, ^ C:\Users\User\Documents\Arduino\libraries\DarkSkyWeather/DarkSkyWeather.h:30:64: error: 'DSW_daily' has not been declared bool getForecast(DSW_current current, DSW_hourly hourly, DSW_daily daily, ^ C:\Users\User\Documents\Arduino\libraries\DarkSkyWeather/DarkSkyWeather.h:76:5: error: 'DSW_current' does not name a type DSW_current current; // pointer provided by sketch to the DSW_current struct ^ (etc)

3 Makes no difference, unfortunately.

=========== I am no longer thinking it is memory. The response is first Requesting weather information from DarkSky.net... Then, after a few minutes: Connection failed. Followed by the 'empty' hourly and daily listings. Perhaps something to do with not being able to verify the fingerprints / certificates?

[Bodmer]

OK, if the security certificate is the issue the it can be bypassed by commenting out both lines here, so they read:

//#define AXTLS       // For ESP8266 only: use older axTLS secure client instead of BearSSL
//#define SECURE_SSL  // For ESP8266 only: use SHA1 fingerprint with BearSSL

This will then use BearSSL for the connection but will not check the security certificate results.

I should have mentioned that option 2 was for the "TFT_eSPI_weather" example only as that example is more likely to have memory issues.

[Leonos]

After some experimenting I got some basic code that works for Apixu and DarkSky, on both an ESP32 as well as an ESP8266. Just line 9 (#define DarkSky) needs to be commented out or not.

#ifdef ESP8266
  #include "ESP8266WiFi.h"
#else // ESP32
  #include "WiFi.h"
#endif

#include "WiFiClientSecure.h"

#define DarkSky

#define SSID "ssidname"
#define SSID_PASSWORD "ssidpassword"

String DSapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
String APXapi_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
String latitude = "52.37";
String longitude = "4.89";
String latlon = latitude + "," + longitude;
String language = "en";
bool weatherUpdateSuccess = false;

void setup() { 
  Serial.begin(250000);
  delay(2000);
  Serial.printf("\nConnecting to %s\n", SSID);
  WiFi.begin(SSID, SSID_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }
  Serial.print("\nConnected.\n\n");
}

void loop() {
  weatherUpdateSuccess = getWeatherForecast();
  delay(30 * 60 * 1000);
}

bool getWeatherForecast() {
  uint32_t dt = millis();
  WiFiClientSecure client;

#ifdef DarkSky
  const char* host = "api.darksky.net";
  String url = "https://api.darksky.net/forecast/" + DSapi_key + "/" + latlon 
             + "?exclude=hourly,minutely,alerts,flags" + "&units=si" + "&lang=" + language;
#else  // Apixu
  const char* host = "api.apixu.com";
  String url = "http://api.apixu.com/v1/forecast.json?key=" + APXapi_key + "&q=" + latlon 
             + "&days=3" + "&hour=0&lang=" + language;
#endif // DarkSky or Apixu

  Serial.println(url);
  if (!client.connect(host, 443)) {
    Serial.println("Connection failed");
    return false;
  }
  Serial.println("Host connected to client");
  char c = 0;
  int ccount = 0;
  uint32_t readCount = 0;

  client.print(String("GET ") + url + " HTTP/1.1\r\n" + "Host: " + host + "\r\n" + "Connection: close\r\n\r\n");
  uint32_t timeout = millis();
  while (!client.available()) {
    if ((millis() - timeout) > 4000UL) {
      Serial.println("No response");
      client.stop();
      return false;
    }
    yield();
  }

  while (client.available() || client.connected()) {
    String line = client.readStringUntil('\n');
    Serial.println(line);
    if (line == "\r") {
      Serial.println("Header completed");
      break;
    }
  }
  timeout = millis();
  while (client.available() || client.connected()) {
    while (client.available()) {
      c = client.read();
      if (c == '{' || c == '[' || c == '}' || c == ']') { Serial.println(); }
      Serial.print(c);
      if ((ccount++ > 100 && c == ',') || c == ',') { ccount = 0; Serial.println(); }
      if ((millis() - timeout) > 20000UL) {
        Serial.println("Stream timed out");
        client.stop();
        return false;
      }
      yield();
    }
  }
  client.stop();
  Serial.print("\nDone in "); 
  Serial.print(millis() - dt); Serial.println(" ms\n");
  return true;
}

This no longer results in 'Connection failed'. Note that it contains no fingerprint or certificate code at all.

(Why doesn't this code indent like it is supposed to?)

[Bodmer]

OK, interesting because port 443 is normally for secure connections. I could not get DarkSky to accept a connection on that port without using a secure client but maybe I had a typo in the request.

Thanks for your input.

P.S. I've edited your post to remove your API keys and add the code formatting markup.

[Bodmer]

OK, I had a look at your code, essentially it is identical code up to the "Connection failed" point you experienced as is used by the DarkSky Weather library so I am not sure what is happening in your setup!

On my setup I had to change the #define SSID to #define WIFI_SSID (and where else used) as SSID is a reserved name and so the WiFi never connects.

Try the library in the default #define state again and see if it connects.

[Leonos]

P.S. I've edited your post to remove your API keys and add the code formatting markup.

I should have mentioned that the keys were largely made up random characters ;) Thanks for the code formatting markup. I'll try to remember it for next time.

On my setup I had to change the #define SSID to #define WIFI_SSID (and where else used) as SSID is a reserved name and so the WiFi never connects.

But that is strange, isn't it? The DarkSky example My_DarkSkyWeather_Test.ino uses #define SSID and the code worked on my ESP32 but not on my ESP8266. But above you mentioned it worked for you on your NodeMCU board?

But even stranger is the fact that I re-ran your original My_DarkSkyWeather_Test.ino example just now (unchanged #define SSID) and it works, even on an ESP8266.

So, now I am thinking it maybe was just a temporary hick-up on the DarkSky server side. That still does not explain why it did not work on the 8266 but did on an ESP32, and in the browser, but at least I am pretty convinced now that the code has nothing to do with it, apart from maybe a more time-critical WiFiClientSecure.h for the ESP8266? Actually, I am at a loss now.

[Bodmer]

I suspect these problems are down to cached code settings and the historical configuration content of the ESP8266 FLASH. In the end I made a complete FLASH erase of all settings of the ESP8266 and now it connects to WiFi with the name set to SSID. I guess the name change somehow forced a cached settings update. I will have a look at the ESP8266 core github issues list, I suspect that is where the problem is.

[Leonos]

Ok, when you find anything, let me know. Thanks again for all your support!!

[Bodmer]

I could not find a related issue for the ESP8266 core or thd Arduino IDE.

@aessig do you still have a problem with your setup?

[Nelson666]

Hi, I have the same issue with an bad WiFi Connection. 40% is too bad, and no SSL Header return's, but an local request to an internal LAN Server works.. With 60% RSSI all works fine.