HTTPClient redirect fails with http code -7 (guessed root cause included)

Basic Infos

[X] This issue complies with the issue POLICY doc.
[X] I have read the documentation at readthedocs and the issue is not addressed there.
[X] I have tested that the issue is present in current master branch (aka latest git).
[X] I have searched the issue tracker for a similar issue.
[X] If there is a stack dump, I have decoded it.
[X] I have filled out all fields below.

Platform

Hardware: ESP8266 NodeMCU
Core Version: 3.1.2
Development Env: Arduino IDE
Operating System: Windows

Settings in IDE

Module: NodeMCU 1.0 (ESP-12E Module)
Flash Mode: other
Flash Size: 2MB/1MB
lwip Variant: v2 Lower Memory
Reset Method: nodemcu
Flash Frequency: 40Mhz
CPU Frequency: 80Mhz
Upload Using: SERIAL
Upload Speed: 115200

Problem Description

When using HTTPClient setFollowRedirects(HTTPC_FORCE_FOLLOW_REDIRECTS) (follow redirect) on api.weather.gov (very popular weather source), redirect fails with http code -7 This is likely because when sending out 301, the server sends out not only headers, but http content as well (content length not 0). The guess is: the HTTPClient code itself, when making the second (redirected) request reuses connection, seemingly did not clear out the pipe enough (reason unknown), and caused subsequent reply to not look like an HTTP reply

From debug output: Reply has content, not just headers 02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 461' subsequent reply has gibberish in front, presumably left over from previous reply 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'est with the redundancy removed. If your client supports it, you will be redirected.",' 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: ' "instance": "https://api.weather.gov/requests/1aa0d75b"' 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: '}HTTP/1.1 200 OK'

Setting _reuse to false (new connection for redirect) solves the problem.

I am not sure api.weather.gov is violating standard by doing this, but "force follow" is supposed to throw standard out of the window, and consider api.weather.gov is very popular, this is worth fixing IMHO.

MCVE Sketch

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>
#include <WiFiClientSecureBearSSL.h>

#ifndef STASSID
#define STASSID "myapssid"
#define STAPSK "password"
#endif

ESP8266WiFiMulti WiFiMulti;

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println();
  Serial.println();
  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(STASSID, STAPSK);
  Serial.println("setup() done connecting to ssid '" STASSID "'");
}

void loop() {
  const char* headerkeys[] = {"Location"};
  if ((WiFiMulti.run() == WL_CONNECTED)) {
    std::unique_ptr<BearSSL::WiFiClientSecure> client(new BearSSL::WiFiClientSecure);
    client->setInsecure();
    HTTPClient https;
    https.collectHeaders(headerkeys, 1);
    https.setFollowRedirects(HTTPC_FORCE_FOLLOW_REDIRECTS);
    Serial.print("[HTTPS] begin...\n");
    if (https.begin(*client, "api.weather.gov", 443, "/points/40.697,-74.330"), true) {
      int httpCode = https.GET();
      if (httpCode > 0) {
        Serial.printf("[HTTPS] GET... returned code: %d\n", httpCode);
      } else {
        Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
      }
      https.end();
    } else {
      Serial.printf("[HTTPS] Unable to connect\n");
    }
  }
  Serial.println("Wait 10s before next round...");
  delay(10000);
}

Debug Messages

02:44:38.101 -> [HTTP-Client] connected to api.weather.gov:443 02:44:38.101 -> [HTTP-Client] sending request header 02:44:38.101 -> ----- 02:44:38.101 -> GET /points/40.697,-74.330 HTTP/1.1 02:44:38.101 -> Host: api.weather.gov 02:44:38.101 -> User-Agent: ESP8266HTTPClient 02:44:38.101 -> Accept-Encoding: identity;q=1,chunked;q=0.1,;q=0 02:44:38.101 -> Connection: keep-alive 02:44:38.101 -> Content-Length: 0 02:44:38.101 -> 02:44:38.101 -> ----- 02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 301 Moved Permanently ' 02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'Server: nginx/1.20.1 ' 02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/problem+json ' 02:44:38.133 -> [HTTP-Client][handleHeaderResponse] RX: 'Location: /points/40.697,-74.33 ' 02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Request-ID: ab8a3cd6-ea1d-49f1-9b4a-bdb160497cb7 ' 02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Correlation-ID: 1aa0d75b ' 02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Server-ID: vm-bldr-nids-apiapp13.ncep.noaa.gov ' 02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: ' 02:44:38.166 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Expose-Headers: X-Correlation-Id, X-Request-Id, X-Server-Id ' 02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Pragma: no-cache ' 02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 461 ' 02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: private, must-revalidate, max-age=83944 ' 02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Expires: Fri, 15 Mar 2024 09:03:43 GMT ' 02:44:38.199 -> [HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 14 Mar 2024 09:44:39 GMT ' 02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive ' 02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Edge-Request-ID: 293e95b7 ' 02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept,Feature-Flags,Accept-Language ' 02:44:38.232 -> [HTTP-Client][handleHeaderResponse] RX: 'Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload ' 02:44:38.265 -> [HTTP-Client][handleHeaderResponse] RX: ' ' 02:44:38.265 -> [HTTP-Client][handleHeaderResponse] code: 301 02:44:38.265 -> [HTTP-Client][handleHeaderResponse] size: 461 02:44:38.265 -> [HTTP-Client][sendRequest] following redirect (the same method): '/points/40.697,-74.33' redirCount: 0 02:44:38.265 -> [HTTP-Client][sendRequest] type: 'GET' redirCount: 0 02:44:38.265 -> [HTTP-Client] connect: already connected, reusing connection 02:44:38.265 -> [HTTP-Client] sending request header 02:44:38.302 -> ----- 02:44:38.302 -> GET /points/40.697,-74.33 HTTP/1.1 02:44:38.302 -> Host: api.weather.gov 02:44:38.302 -> User-Agent: ESP8266HTTPClient 02:44:38.302 -> Accept-Encoding: identity;q=1,chunked;q=0.1,;q=0 02:44:38.302 -> Connection: keep-alive 02:44:38.302 -> Content-Length: 0 02:44:38.302 -> 02:44:38.302 -> ----- 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'est with the redundancy removed. If your client supports it, you will be redirected.",' 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: ' "instance": "https://api.weather.gov/requests/1aa0d75b"' 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: '}HTTP/1.1 200 OK ' 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'Server: nginx/1.20.1 ' 02:44:38.333 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/geo+json ' 02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: ' 02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Expose-Headers: X-Correlation-Id, X-Request-Id, X-Server-Id ' 02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Request-ID: 2c631c31-ab57-4ec8-ac12-8207b452a909 ' 02:44:38.366 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Correlation-ID: 1a96a60d ' 02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Server-ID: vm-bldr-nids-apiapp16.ncep.noaa.gov ' 02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept-Encoding ' 02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'Cache-Control: public, max-age=83281, s-maxage=120 ' 02:44:38.399 -> [HTTP-Client][handleHeaderResponse] RX: 'Expires: Fri, 15 Mar 2024 08:52:40 GMT ' 02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 14 Mar 2024 09:44:39 GMT ' 02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 3098 ' 02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive ' 02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'X-Edge-Request-ID: 293e96aa ' 02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept,Feature-Flags,Accept-Language ' 02:44:38.433 -> [HTTP-Client][handleHeaderResponse] RX: 'Strict-Transport-Security: max-age=31536000 ; includeSubDomains ; preload ' 02:44:38.465 -> [HTTP-Client][handleHeaderResponse] RX: ' ' 02:44:38.465 -> [HTTP-Client][handleHeaderResponse] code: 0 02:44:38.465 -> [HTTP-Client][handleHeaderResponse] size: 3098 02:44:38.465 -> [HTTP-Client][handleHeaderResponse] Remote host is not an HTTP Server![HTTP-Client][returnError] error(-7): no HTTP server 02:44:38.465 -> [HTTP-Client][returnError] tcp stop 02:44:38.503 -> [HTTPS] GET... failed, error: no HTTP server 02:44:38.503 -> [HTTP-Client][end] tcp is closed

esp8266 / Arduino