HTTPClient doesn't send the URL in HTTPS

[J6B]

Basic Infos

Trying to make a GET to /something and receive a request for / only on web server side.

Hardware

Hardware: ESP-01 Core Version: 2.2.0

Description

My sketch do a GET using HTTPClient. I was in release 2.0.0 and I just updated to 2.2.0 release (did not try in 2.1.0) My sketch was able to send GET for HTTP and HTTPS URI using 2.0.0 release. Since I updated to 2.2.0, HTTP works almost fine (getting frequent -11 return code while my server answering bellow 5sec).

But the issue is while i'm trying to send same GET request using HTTPS, My web server receive a wrong header requesting for / only.

Exemple: I request for https://192.168.1.123/plugins/teleinfo/core/php/jeeTeleinfo.php?api=YQFMNcxGJ52XSrHYcF3U&ADCO=050522038502&HCHP=072686874&PAPP=00730 My apache2 log contains this: 192.168.1.104 - - [19/Apr/2016:11:57:21 +0200] "GET / HTTP/1.0" 400 0 "-" "-"

Settings in IDE

Module: Generic ESP8266 Module Flash Size: 1MB CPU Frequency: 80Mhz Flash Mode: dio Flash Frequency: 40Mhz Upload Using: OTA Reset Method: ck

Sketch

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

void setup() {
 //init Wifi
}

void loop() {

String completeURI = "https://192.168.1.123/plugins/teleinfo/core/php/jeeTeleinfo.php?api=YQFMNcxGJ52XSrHYcF3U&ADCO=050522038502&HCHP=072686874&PAPP=00730";

  //send HTTP request
  HTTPClient http;
  http.begin(completeURI);
  requestResult = http.GET();
  http.end();

}

<bountysource-plugin>

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/33106058-httpclient-doesn-t-send-the-url-in-https?utm_campaign=plugin&utm_content=tracker%2F14245935&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F14245935&utm_medium=issues&utm_source=github).
</bountysource-plugin>

[J6B]

I just tried with 2.1.0 and the same code works fine to reach server using HTTPS

[wouterds]

I can confirm that I can't do any calls to https endpoints using the ESP8266HTTPClient.h lib.

Eg this code:

  Serial.print("API Call to: https://www.reddit.com/iphone.json");
  Serial.println();

  HTTPClient http;

  int beginResult = http.begin("https://www.reddit.com/iphone.json");
  http.addHeader("Content-Type", "text/plain");

  int httpCode = http.POST("Message from ESP8266");
  String response = http.getString();

  Serial.print("beginResult: ");
  Serial.print(beginResult);
  Serial.println();

  Serial.print("http: ");
  Serial.print(httpCode);
  Serial.println();

  Serial.print("response: ");
  Serial.println(response);
  Serial.println();

  http.end();

Should return 404 for the status code and {"message": "Not Found", "error": 404} for the response body. Instead it returns -1 and no contents (connection refused). This happens for all https endpoints.

[J6B]

Hi, this is because you need now to provide website certificat thumbprint in http.begin for https website.

http.begin("https://www.reddit.com/iphone.json","f8d1965323111e86e6874aa93cc7c52969fb22bf"); (I put the thumbprint of www.reddit.com :-) )

[wouterds]

What if it changes (thinking about tools like Letsencrypt and Cloudflare that generate free certificates on the fly that are only valid for short periods of time)?

[igrr]

That is a valid concern! I am currently working on that as part of https://github.com/esp8266/Arduino/pull/3700. That will add ability to verify the certificate in one of the several ways (fingerprint, subjectPublicKeyInfo, root cert), when using HTTPClient.

On Thu, Oct 12, 2017, 08:12 Wouter De Schuyter notifications@github.com wrote:

What if it changes (thinking about tools like Letsencrypt and Cloudflare that generate free certificates on the fly)?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/esp8266/Arduino/issues/1941#issuecomment-336111483, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJcevAIAJiR4XB76RgVaHldluQ162dwks5srgIPgaJpZM4IKjjO .

[J6B]

Thanks igrr :-)

[wouterds]

Alright, good to know @igrr, thanks :-)!

Another question: is there a way to ignore it if you don't care about it? Eg like the unix command wget that has an option --no-check-certificate?

[igrr]

Currently there isn't a way to do that with HTTPClient. If you need this, you can copy the .cpp/.h files of HTTPClient into your sketch (renaming the .h file), and delete the line in .cpp file responsible for the check.

On Thu, Oct 12, 2017, 08:18 Wouter De Schuyter notifications@github.com wrote:

Alright, good to know @igrr https://github.com/igrr :-)!

Another question: is there a way to ignore it? If you don't care about? Eg like the unix command wget that has an option --no-check-certificate?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/esp8266/Arduino/issues/1941#issuecomment-336113309, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJceszokb9CDpTnDOW-t8y-NdaM86vUks5srgN4gaJpZM4IKjjO .

[wouterds]

I guess I'll do that for now, thanks!

[hutch120]

Just wondering if anyone got this going? If so, any chance you can publish the relevant bits of the code to change in HTTPClient.cpp? This is what I tried...

I can see the entry point for https is:

bool HTTPClient::begin(String url, String httpsFingerprint)

So I detect the url using url.startsWith("https:"), then call client.begin with a fake string param like so:

  if (url.startsWith("https:")) {
    client.begin(url, "INSECURE_HTTPS"); //Specify request destination for HTTPS
  } else{    
    client.begin(url);  //Specify request destination for HTTP
  }

And tried changing this:

_transportTraits = TransportTraitsPtr(new TLSTraits(httpsFingerprint));

to this:

_transportTraits = TransportTraitsPtr(new TransportTraits());

Few other things I tried, but that didn't work.

• return true on the verify function
• delete the whole verify code block.

Any tips?

[hutch120]

@wouterds Did you get this going? Any chance you can post the relevant changes? (See my attempts above)

[wouterds]

Hi! No, I did not get it working yet. I managed to copy the libs and include my custom libs, tried commenting all the verify methods out / make them early return true but without success. Still looking into it! Any help would be appreciated!

On 19 Oct 2017, at 02:19, Simon notifications@github.com wrote:

@wouterds https://github.com/wouterds Did you get this going? Any chance you can post the relevant changes? (See my attempts above)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/esp8266/Arduino/issues/1941#issuecomment-337764035, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJ5BLDsb3uDb1muYiWrlTTwY9vQ13YGks5stpWpgaJpZM4IKjjO.

[TTnsp]

@hutch120 : I have the same problem. So, to connect in https without certificat (fingerprint), i change the fonction "verify" in TLSTraits class in ESP8266HTTPClient.cpp. I made this fonction return always true, and that work fine.

[hutch120]

Hi @igrr, Any chance you could help @wouterds and I out on this one? We've both had a crack at modifying the source in ESP8266HTTPClient.cpp but are coming up short.

Any tips?

[zanechua]

@hutch120

Actually @TTnsp already provided the answer.

Here's the modified code portion in the ESP8266HTTPClient.cpp file

class TLSTraits : public TransportTraits
{
public:
    TLSTraits(const String& fingerprint) :
        _fingerprint(fingerprint)
    {
    }

    std::unique_ptr<WiFiClient> create() override
    {
        return std::unique_ptr<WiFiClient>(new WiFiClientSecure());
    }

    bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        //Always return true to never validate certificate.
        return true;
    }

[hutch120]

@zanechua Thanks... I'm pretty sure I tried returning true from the verify function, I even made a note that I tried it in my post.

I'll have another crack.... did you try returning true from the verify function @wouterds ?

[zanechua]

@hutch120

It works for me though. I'm using LetsEncrypt on my https server. It was a POST Request.

Did you modify the library directly? I actually copied them into my sketch folder and change the headers to use the local sketch libraries instead.

[TTnsp]

@zanechua

I'm in the same case of you and i copied the librairie into my sketch.

[iamalonso]

@zanechua Actually I'm trying to send a POST request from my Wemos d1 mini board to Mlab API via https and I have your same problem. I made a local copy of ESP8266HTTPClient.cpp into my sketch and I modified the verify function to return true but it didn't work.

[zanechua]

@alonsocarvajal

Not sure if I can help but what error are you getting?

[iamalonso]

@zanechua

Below I attach a portion of my code.

void loop() {

  // Get temperature and humidity
  float temperature = dht.readTemperature();
  float humidity = dht.readHumidity();

  // Creating JSON object
  JsonObject& root = jsonBuffer.createObject();
  root["temperature"] = temperature;
  root["humidity"] = humidity;
  size_t requestBodySize = root.measureLength() + 1;
  char requestBody[requestBodySize];
  root.printTo(requestBody, requestBodySize);

  // Starting  POST request 
  int httpBeginCode = http.begin("https://api.mlab.com/api/1/databases/MyDB/collections/MyColl?apiKey=XXXXXXXXXXXXXXXXXXXXXXXXX");
  http.addHeader("Content-Type", "application/json");
  int httpPOSTCode = http.POST(requestBody);

  // Printing responses
  Serial.println(httpBeginCode);
  Serial.println(httpPOSTCode);

  // Ending http request
  http.end();
  delay(3000);
}

The return codes that I'm receiving in httpBeginCode and httpPOSTCode variables are 0 and -1 respectively.

Remember that actually I modified the verify function located on ESP8266HTTPClient.cpp file to return true (see the next code).

bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        //return wcs.verify(_fingerprint.c_str(), host);
        return true;
    }

[zanechua]

Could you try this line with your post request instead?

int httpBeginCode = http.begin("https://api.mlab.com/api/1/databases/MyDB/collections/MyColl?apiKey=XXXXXXXXXXXXXXXXXXXXXXXXX", "imaginarykey");

[iamalonso]

@zanechua

I tried that alternative and now the return codes that I'm receiving in httpBeginCode and httpPOSTCode variables are 1 and -1 respectively. I suppose that now the http.begin() its ok but the http.POST() request stills fail.

Additionally I tried a GET request to get data from MLAB's and NASA's API but I received -1 as a return code. The code for this example is:

int httpBeginCode = http.begin("https://api.nasa.gov/planetary/apod?api_key=NNKOjkoul8n1CH18TWA9gwngW1s1SmjESPjNoUFo", "2E BD F3 9F C1 1D BC 49 96 25 6A DC 32 EB FE 2B F0 32 70 50");
int httpGETCode = http.GET();

// Print responses
Serial.println(httpBeginCode);
Serial.println(httpGETCode);

// Printing data
Serial.println(http.getString());

// Ending http request
http.end();
delay(3000);

In this occasion the return code httpBeginCode and httpGETCode are 1 and -1 respectively.

Therefore, in both cases (POST and GET request) the http.begin() is successful but the http.POST() and http.GET() methods are fail.

Note: I've also tried removing the return true in verify method into ESP8266HTTPClient.cpp file.

bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        //return wcs.verify(_fingerprint.c_str(), host);
        return true;
    }

bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        return wcs.verify(_fingerprint.c_str(), host);
        //return true;
    }

Note 2: The fingerprint of Nasa API was obtained from https://www.grc.com/fingerprints.htm.

[iamalonso]

Hello @igrr, I've been trying to get data from NASA's API using the example HTTPSRequest.ino but I can not succeed. The same happens when I send a POST request to MLAB's API (see the post above).

My code based on HTTPSRequest.ino is:

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

const char* ssid = "10";
const char* password = "XXXXXXX";
const char* host = "api.nasa.gov";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char* fingerprint = "2E BD F3 9F C1 1D BC 49 96 25 6A DC 32 EB FE 2B F0 32 70 50";

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

  String url = "/planetary/apod?api_key=NNKOjkoul8n1CH18TWA9gwngW1s1SmjESPjNoUFo";
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');
  if (line.startsWith("{\"state\":\"success\"")) {
    Serial.println("esp8266/Arduino CI successfull!");
  } else {
    Serial.println("esp8266/Arduino CI has failed");
  }
  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println("closing connection");
}

void loop() {
}

The serial monitor output is:

WiFi connected
IP address: 
192.168.160.10
connecting to api.nasa.gov
connection failed

It is possible that you can guide me regarding this problem?

[shannonzchanz]

Hello, I am experiencing the same problem as well. I attempted to change the return statement to true in my appdata, hardware folder.

` class TLSTraits : public TransportTraits { public: TLSTraits(const String& fingerprint) : _fingerprint(fingerprint) { }

std::unique_ptr<WiFiClient> create() override
{
    return std::unique_ptr<WiFiClient>(new WiFiClientSecure());

}

bool verify(WiFiClient& client, const char* host) override
{
    auto wcs = reinterpret_cast<WiFiClientSecure&>(client);
    //return wcs.verify(_fingerprint.c_str(), host);          //original code
    return true;
}`

[iamalonso]

Hello @shannonzchanz

1. What kind of request are you doing: GET or POST?
2. What website are you sending these request to?
3. Could you attach your Arduino code?

[liebman]

@shannonzchanz try this PR https://github.com/esp8266/Arduino/pull/3933

[shannonzchanz]

I have resolved it! It's just that my esp8266 doesn't reach the localhost. So, I would have to publish it online. Thanks for the help.

[jLynx]

I am still having the same issue with not being able to load an https website.

This is what I have currently tried

http.begin("https://myFAKEsite.org/api/test", "6AEE739DA4EE7A518C87E2E96691F46136C9F060");      
    http.addHeader("Content-Type", "application/x-www-form-urlencoded");

    int httpCode = http.POST(postData);   //Send the request
    String payload = http.getString();                  //Get the response payload

But I still get -1, even with passing the current SH1 key for that specific site.

What am I doing wrong here?

[hutch120]

Hi @theyogeshrathod Looks like maybe you are fairly new to Github, and just starting out with ESP boards? Please be aware that github issue lists are generally reserved for bugs and I'd suggest that a better place to ask random questions like this is on the StackOverflow website.

Also, if you do post a response to a thread please ensure you read and understand the issue first, or you just annoy people. This thread clearly has nothing to do with your problem, it is related to an issue with HTTPS as it says in the issue title.

Also, I suggest you check if a thread has been updated recently, this hasn't had an actual answer since 2017 and probably should be locked and closed.

@igrr please consider locking this issue.

[theyogeshrathod]

Oh, sorry @hutch120. I will just delete the above comment. Thanks!