HTTP module and SSL don't play nice on dev

[marcelstoer]

When I happened to see another (or a new) HTTP & SSL issue the other day I went "Grrr, why me again?". Some of you may remember 😉

TL;DR

Many HTTPS requests from the http module fail while connecting to the same resources with the net/TLS module usually succeeds.

Test code

function test(host, path)
  local url = "https://" .. host .. path;
  http.get(url, nil, function(code, data)
    if (code < 0) then
      print("HTTP request to " .. url .. " failed")
    else
      print("HTTP request to " .. url .. " succeeded")
    end
  end)
  local srv = tls.createConnection(net.TCP, 0)
  srv:on("receive", function(sck, c) print("net/TLS to " .. url .. " succeeded") end)
  srv:on("connection", function(sck, c)
    sck:send("GET " .. path .. " HTTP/1.1\r\nHost: " .. host .. "\r\nConnection: keep-alive\r\nAccept: */*\r\n\r\n")
  end)
  srv:connect(443, host)
end
test("raw.githubusercontent.com", "/espressif/esptool/master/MANIFEST.in")

Test result

net/TLS to https://raw.githubusercontent.com/espressif/esptool/master/MANIFEST.in succeeded
HTTP client: Connection timeout

Then I checked heap: 25312. That was suspiciously low, I started with ~44k, so I ran test("raw.githubusercontent.com", "/espressif/esptool/master/MANIFEST.in") again and got

E:M 528
E:M 272
HTTP client: Disconnected with error: 46
HTTP client: Connection timeout
HTTP client: Connection timeout

-> no successful feedback from net/TLS code anymore.

Does "HTTP client: Disconnected with error: 46" indicate that the client was still maintaining the previous (failed) connection which it tried to kill first? I have my doubts because I sometimes also see this when the test runs after a clean reboot.

I tested a few more URLs, each after a clean reboot with both http and net modules.

URL	http	net/TLS
https://raw.githubusercontent.com/espressif/esptool/master/MANIFEST.in	❌	✅
https://httpbin.org/ip	❌	❌ no output at all, not even error
https://nodemcu-build.com	❌	✅
https://clients5.google.com/pagead/drt/dn/	❌	✅

NodeMCU version

NodeMCU custom build by frightanic.com
    branch: dev
    commit: 5425adefff62f9ea2094e3e4581a79f1424e4433
    SSL: true
    modules: file,gpio,http,net,node,tmr,uart,wifi,tls
 build  built on: 2017-01-06 19:39
 powered by Lua 5.1.4 on SDK 2.0.0(656edbf)

Hardware

NodeMCU devkit v2

[heythisisnate]

I'm not sure if this is the same issue, but I'm unable to make any https requests at all on the 2.0.0 firmware. This worked on the previous version. Any time I try a GET or POST to a https URL I get HTTP client errors. Am I missing something here?

=http.get('https://google.com')
> HTTP client: Disconnected with error: 8
HTTP client: Connection timeout
HTTP client: Connection timeout

NodeMCU version:

NodeMCU custom build by frightanic.com
    branch: master
    commit: b96e31477ca1e207aa1c0cdc334539b1f7d3a7f0
    SSL: true
    modules: file,gpio,http,net,node,tmr,uart,wifi,tls
 build  built on: 2017-02-20 23:43
 powered by Lua 5.1.4 on SDK 2.0.0(656edbf)

[marcelstoer]

I'm not sure if this is the same issue

I'm quite convinced it is. My table with URLs also contains a Google URL.

[heythisisnate]

@marcelstoer I'm relieved that I'm not the only one having issues with the http library and HTTPS requests. I have not had any success at all on the 2.0.0 SDK (built from master) but it does work ok on 1.5.4.1. Do you have any indication where the problem may be? I'm happy to help as much as I can but I'm not sure of the bug reporting/fixing process for this project. It looks like you've contributed a ton to NodeMCU development (thanks for this!!) so can you let me know how else I can help get this resolved?

[marcelstoer]

It looks like you've contributed a ton to NodeMCU development (thanks for this!!) so can you let me know how else I can help get this resolved?

Sorry Nate, I do pretty much everything around here - except firmware coding 😞 The last substantial contributions to the module were from @pjsg and @luismfonseca.

I'm happy to help as much as I can but I'm not sure of the bug reporting/fixing process for this project.

Well, the Lua veneer for the HTTP library is here: https://github.com/nodemcu/nodemcu-firmware/blob/dev/app/modules/http.c. The library itself is here: https://github.com/nodemcu/nodemcu-firmware/tree/dev/app/http. The last commit that tinkered with HTTPS is a592af7ab1da5dab0b656f2365d60548df1e49b4 from @djphoenix but it still looks fine to me. If you want to dig into code and start building firmware have a look at http://nodemcu.readthedocs.io/en/latest/en/build/ (Linux/Linux VM or Docker from yours truly).

[heythisisnate]

@marcelstoer Thanks for the pointers. I looked around in the code and see that SSL support is flagged by the presence of the CLIENT_SSL_ENABLE constant. It seems like maybe this constant is not getting defined when building but I haven't verified this yet. It looks like the constant is commented out here: https://github.com/nodemcu/nodemcu-firmware/blob/master/app/include/user_config.h#L67

How does your cloud build service work? Is there some process that un-comments this definition when the SSL/TLS package is enabled? Are we sure that's working? I noticed in the tools/pr-build.sh script there's something that uncomments the line, but maybe that's only for dev builds?

[marcelstoer]

Is there some process that un-comments this definition when the SSL/TLS package is enabled?

Correct.

Are we sure that's working?

Yes, with near certainty. I'm quite convinced that I tested with images built from my cloud builder and with manually built images (Docker). I will test again though just to be sure.

I noticed in the tools/pr-build.sh script there's something that uncomments the line, but maybe that's only for dev builds?

Almost. This is triggered from .travis.yml when someone creates a pull request. Having an automatic CI build with all modules and SSL enabled gives us greater confidence that a PR won't break stuff even though we don't have any unit or integration tests.

[heythisisnate]

I tried building the firmware myself tonight with your docker build project and had the same problems. So it seems there is an actual bug somewhere (not in the build process).

even though we don't have any unit or integration tests.

😢

[heythisisnate]

I captured some output from debug mode that I wanted to share. My app is doing a https POST to a https-only API, https://graph-na02-useast1.api.smartthings.com. This is the result every time:

client handshake ok!
client's data invalid protocol
Reason:[-0x7880]
HTTP client: Disconnected with error: 8
HTTP client: Connection timeout

I'm able to reproduce the problem with a simple GET to https://google.com:

=http.get("https://google.com")
> client handshake start.
please start sntp first !
please start sntp first !
client handshake ok!
client's data invalid protocol
Reason:[-0x7880]
HTTP client: Disconnected with error: 8
HTTP client: Connection timeout
HTTP client: Connection timeout

Here you can see that error 0x7880 is "The peer notified us that the connection is going to be closed." I'm not really sure where to look from here. Any ideas?

[marcelstoer]

Is there a documentation somewhere about espconn error codes that are passed to http_error_callback?

[Kaiser442]

I'm afraid this isn't an appropriate place to ask this question, but I'm getting desperate. I'm having this same issue - https requests aren't working. I'd like to just downgrade and get a build using 1.5.1 if I can, because the scripts I'm using were developed under that SDK, but I am not sure how to do that. This product was originally made using a build from nodemcu-build.com, but I don't have access to that bin file and I'm not sure how to recreate it. It looks like that site doesn't let you specify anything other than the current master or dev branch. I found a "how to compile NodeMCU" page, but those instructions use Linux (which I have access to if required, but it's not one of my strengths). Help!

[heythisisnate]

@Kaiser442 The simplest thing to do to get an older firmware is use @marcelstoer's Docker build container: https://hub.docker.com/r/marcelstoer/nodemcu-build/

You can check out whatever branch or tag of the firmware that you want from git and then edit the app/include/user_modules.h and app/include/user_config.h files to choose the packages/options you want. Then build the firmware using the Docker image.

[jmattsson]

The client's data invalid protocol appears to come from the SDKs libssl.a. Does anyone have the ability to get a packet trace of an attempted connection? Ideally in a way that gives you the ephemeral keys so we can look inside the encrypted stream. It might also be an idea to double-check the available cipher suites against what the relevant servers are configured to use/accept.

I'm completely not in ESP{8266,32}-land at the moment, but if someone has traces I can probably find some to have a quick look.

@marcelstoer espconn error codes can be found in sdk/*/include/espconn.h, though they're not very informative. In this case, 8 = connection aborted. The Reason: -0x7880 is from mbedTLS, and is simply the notification from the remote side that it's going to terminate the connection.

Another idea would be for someone to play with openssl s_server in seriously verbose mode, and have NodeMCU connect to it - that might give some good clues.

[djphoenix]

@jmattsson there are debugging flags in user_mbedtls.h so you can enable each log level that you want. Can't explain it carefully right now. So I suspect there are fatal architecture issues in http module because it's "don't play nice" for plain requests too. I can debug it carefully some later (maybe in this weekend).

[Josepdal]

@djphoenix Have you found something interesting?

[Jonathan411]

This is a key issue for us. Can anyone help resolve this? I don't know if this is appropriate, but would a bounty help improve the priority of getting this fixed?

[dtran123]

I have been using the frozen 1.5.4.1 branch for now due to this problem. Even sending REST api calls via TCP connections will fail (in many cases) using this branch. So there isn't really an alternative. Both TCP and HTTP connections don't work well for secured connections (it's a hit & miss depending on the server you connect to)

[heythisisnate]

I would contribute $$ to a bounty to get this fixed. I'm also stuck on the 1.5.4.1 branch for my project due to this issue.

[Jonathan411]

I will put up a $500 bounty to get this fixed. This is really important to us. I have to believe that it is a serious issue for anyone who needs to send secure data.

[Jonathan411]

@heythisisnate will you add something to the bounty to see if we can get this issue resolved?

[heythisisnate]

Wow @Jonathan411 that's quite a generous bounty! I can't afford that much right now, but I'd be willing to chip in 0.035 BTC (~$40 USD) to the developer that gets a PR merged that fixes this issue.

[dtran123]

Hey guys, I will also contribute to the bounty with a 50$ USD for anyone who can solve this issue...though I am hoping that maybe the next SDK will address it. Refer to issue #1810.

[Jonathan411]

@heythisisnate , @dtran123 thanks for the bounty support, every bit helps!

[greg-szabo]

Hi,

I've looked into the issue a little bit because I was experiencing similar errors. In fact I couldn't use the tls module in a consistent fashion. I tried the dev branch of the firmware mostly (with some tests with 'master' branch and some tests with the 2.0.0.0 tag from February) and I used the docker build process. DEVELOP_VERSION was set to true I've added some extra os_printf's to the mbedtls_parse_internal function in mbedtls's espconn_mbedtls.c. I also commented out line 88 and 89 in mbedtls's debug.c (this enables verbose debugging for mbedtls, which seems to be badly implemented - or I didn't use MBEDTLS_DEBUG_C properly somehow.). I flashed the integer binary.

First of all, to do a decent SSL certificate chain verification, I needed the correct time. This requires the rtctime, rtcmem and sntp modules. After compiling and flashing, the below commands can set up time (with a 1000 second recurring sync according to the documentation):

sntp.sync("pool.ntp.org",function(sec,us,server,info) print ("Seconds: "..sec.." Server: "..server.." Stratum: "..info.stratum) end, function(errorcode,info) print ("SNTP errorcode: "..errorcode.." Info: "..info) end, true)

After running this command, you should see the "Seconds: ..." line filled with the current GMT time in EPOCH format. The "rtctime.get()" command should give the same result. (Use epochconverter.com to convert to a readable format.)

After this, I tried to add the certificate authorities to the trusted certificates list using tls.cert.verify, like this:

tls.cert.verify([[ -----BEGIN CERTIFICATE----- MIImyfirstcertificatedata... -----END CERTIFICATE----- ]],[[ -----BEGIN CERTIFICATE----- MIImysecondcertificatedata... -----END CERTIFICATE----- ]])

Note that the documentation states that multiple CA certificates can be added by comma-separated strings. This might be essential because I didn't know how mbedtls is validating the certificates. (Here's an interesting entry about this here.)

Afterwards you can use the tls module to open a secure connection. I hooked the "receive" and "connection" events, but none of them got called because the connection attempt fails at the SSL handshake. Namely the mbedtls_ssl_handshake(&TLSmsg->ssl) request in line 880 in espconn_mbedtls.c (within the mbedtls_parse_internal function) returns with an error message. Before I set up time, I usually received a MBEDTLS_ERR_SSL_INVALID_RECORD (0x7200) and when I realized that time might be essential and set up sntp and the rtc clock, it changed into a MBEDTLS_ERR_SSL_ALLOC_FAILED (0x7F00). So, by enabling all those required modules, I might have run out of memory... Savage...

At this point I simply moved over to MicroPython, to try to do SSL validation there. I can't believe there's no working example on the Internet about this critical feature (provided you want to develop something that hooks to a network). I might come back to this, because it intrigues me (and MicroPython doesn't have an ntp library for time-keeping or any examples either...)

From the top of my head, I might need to run the floating point firmware instead of the integer, but it's a weak argument at this point. (It should be noted in the docs if this was the case.) If I sprung any ideas for anyone, please share.

EDIT1: Notably, I'm interested if you have to add all intermediate certificates to the TLS trusted certificates store or if mbedtls will go through the chain received from the server and it's enough to store the root CA.

EDIT2: So my current suspect is the mbedtls library itself. The http module might have issues too, but unless mbedtls is consistent, there's no point in trying to fix the http module.

Regards, Greg

[dtran123]

@Greg-Szabo I agree with your EDIT2. that there is no point to look into http issue before we address secured TLS tcp socket connections. I have many projects that work well (socket connection is setup successfully) with branch 1.5.4.1 (based on previous SDK) but same code will not work on master or dev branches.

[djphoenix]

Another side of research (yay, hello there?)

I started work with #1700, and there are no issues with memory, but...

• TLS handshakes takes looong time, so I wrapped handshake steps with

  system_soft_wdt_stop();
  int res = mbedtls_ssl_handshake_step(&pcb->ctx);
  system_soft_wdt_restart();

• Now my wrapper works for handshake, reading, but some problems with write - searching for solution.

So I'll share some code in near time. Stay tuned.

[djphoenix]

So I have something working now (for tls module only). And I finally resolved why httpbin.org cannot be loaded - it doesn't work at all without SNI. A new module have fix for this.

For early pre-alpha source see my tlswrap branch

[djphoenix]

@dtran123 @Greg-Szabo about problems in mbedtls - I suspect (and sure) the key problem not in mbedtls library itself but wrapper layer, espconn. Without espconn all works better :)

Can you send here some examples for more accurate testing?

[greg-szabo]

Hi djphoenix,

Thanks for all the hard work on this module so far and thanks for looking into this problem. You might be right about the wrapper, but I couldn't get to the bottom of it. The domain I was trying is using SNI, so I guess that's why it didn't work.

Below you'll find the code I'm using. Specialities with this example:

1. Let's Encrypt certificate - good for 90 days only, so checking expiration is important
2. SNI
3. root CA + intermediate CA chain (This seems the default now on most web servers.)

I've skipped the code for wifi setup...

  sntp.sync("pool.ntp.org",function(sec,us,server,info)
    print ("Seconds: "..sec.." Server: "..server.." Stratum: "..info.stratum)
    end,
    function(errorcode,info)
    print ("SNTP errorcode: "..errorcode.." Info: "..info)
    end,
    true)

base="GET / HTTP/1.1\r\nHost: demo.philosobear.com\r\nConnection: keep-alive\r\nAccept: */*\r\n\r\n"

interca=[[
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
]]

rootca=[[
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
]]

print(tls.cert.verify(interca,rootca))

srv = tls.createConnection()

srv:on("receive", function(sck, c) print(c) end)
srv:on("connection", function(sck, c)
  -- Wait for connection before sending.
  sck:send(base.."\r\n")
end)

srv:connect(443,"demo.philosobear.com")
--srv:close()

I would be grateful if you gave any help with how you're troubleshooting the wrapper and mbedtls. As mentioned above, I've added extra os_printf messages to the wrapper which worked fairly well, but it didn't yield any results. In the mbedtls library I tried to turn on debugging, but for some reason it doesn't print anything. It's supposed to print a few arrows at least when the handshake starts and finishes, but nothing. Eventually I manually added import osapi.h and started adding extra os_printf() commands everywhere where I wanted to debug, but it's a hassle.

Regards, Greg

[djphoenix]

@Greg-Szabo OK, for my tlswrap (that replaces espconn layer at all) this test code works for me:

function test(host, path)
    local sck = tls.createConnection()
    sck:on('connection', function(s)
        print('CONN')
        s:send(
            'GET ' .. path .. ' HTTP/1.0\r\n' ..
            'Connection: close\r\n' ..
            'Host: ' .. host .. '\r\n' ..
            '\r\n'
        )
    end)
    sck:on('receive', function(s, d)
        print(d)
    end)
    sck:on('disconnection', function(s)
        print('CLOSE')
    end)
    sck:on('reconnection', function(s, e)
        print('ERR', e)
    end)
    sck:connect(443, host)
end

test('demo.philosobear.com', '/')

Two issues still exists:

• Now I haven't finished CA verification at all: all certs becomes valid. This moment is in my TODO-list.
• For 4K incoming buffer, there is problem with servers that sends large TLS records. TLS specifications sets buffer size as 16K, and it means that each connection should have two 16K buffers. Sad story for ESP... So it works for small messages (under 4K) anyway, and works with servers that support max_fragment_length TLS extension.

[marcelstoer]

In all of that let's not forget that actually verifying the server certificate is optional in establishing a connection to the server. It's an important security aspect but optional. So, I suggest we stick with the old "make it run, then make it secure, make it fast, and make it pretty" mantra 😉 As for SNI, yes axTLS doesn't support SNI but since Espressif switched to mbed TLS one should expect this to work fine.

[dtran123]

Agree. My secured tcp connections stopped working (for a number of servers such as IFTTT) between the previous 1.5.4.1 build and the current master & dev. The code does not use certificates. So yes we should address the basic scenario without certificates. It's a hit & miss depending on which server you connect to.

[dtran123]

Just to let everyone know that unfortunately the upgrade to SDK 2.1.0 did not fix this secure tcp socket issue. So I am still stuck with 1.5.4.1 branch till this issue is resolved as secure tcp is important for me.

[Josepdal]

@dtran123 is https working in 1.5.4.1?

//offtopic Would be ok if we create a Telegram group to share experiences with nodemcu, stay tuned with the last news, solve some problems, get interesting info about this project... Is anyone interested in? Just like this post and I will edit with some link.

Telegram NodeMcu Group: @NodeMcu

[dtran123]

I have found that making REST httpS requests via tcp socket to be more reliable than using the http module and this approach is less memory hungry. For my needs I don't necessarily need to use the http module. It's more code and less elegant but for now less troublesome. Until we have a solid http module that reliably supports https with or without certificates.

[dclaar]

The http module doesn't work for me at all. I tried the tls workaround, and it works...about 5 times. Then I get E:Med. I've verified that my timer-based loop does not leak memory by removing the call to my https function and seeing that the heap never changes. But with it included, every time my function is called, I've lost about 700 bytes of heap. I close the socket--and see the confirmation message before the timer fires again.

Heap at start of routine: send_data heap=29464 send_data heap=27952 send_data heap=27328 send_data heap=26792 send_data heap=26000 send_data heap=25472 -- Start getting E:M here send_data heap=24760 send_data heap=23736

send_data calls https():

function https(host, path, data)
    local sck = tls.createConnection()
    sck:on('connection', function(s)
        print('connect')
        local lines = 'PUT ' .. path .. ' HTTP/1.0\r\n' ..
            'Host: ' .. host .. '\r\n' .. 
            'Connection: close\r\n' ..
            'Accept: */*\r\n' ..
            'Content-Type: application/json\r\n' ..
            'Content-Length: ' .. string.len(data) .. '\r\n' ..
            '\r\n' ..
             data .. '\r\n'
       print(lines)
       s:send(lines)
    end)
    sck:on('receive', function(s, d)
        print(d)
    end)
    sck:on('sent', function(s)
        print('sent')
        print('before sck:close heap=' ..node.heap())
        sck:close()
        print('after sck:close heap=' ..node.heap())
    end)
    sck:on('disconnection', function(s)
        print('close')
    end)
    sck:on('reconnection', function(s, e)
        print('ERR', e)
    end)
    sck:connect(443, host)
end

NodeMCU custom build by frightanic.com branch: master commit: c8ac5cf SSL: true modules: dht,file,gpio,net,node,rtctime,sntp,tmr,uart,wifi,tls build built on: 2017-05-25 01:57 powered by Lua 5.1.4 on SDK 2.1.0(116b762)

[jmattsson]

@TerryE @pjsg @marcelstoer @devsaurus @djphoenix

Hey guys, this is a big issue. Does anyone have some time to spend on investigating and hopefully fixing this? I currently don't, much as I'd like to.

[marcelstoer]

A possibly related SDK issue just created by @davydnorris is https://github.com/espressif/ESP8266_NONOS_SDK/issues/10.

[davydnorris]

Oh interesting!

If you haven't already please head over to the espressif forum and add your input to the bug I have logged there, and link this thread as well.

There's someone who has hit the exact same issue as I have trying to use the AT command firmware, so now it's been seen and is affecting people in AT, NodeMCU and NonOS SDK builds

[davydnorris]

I have had a reply from Espressif - the bug I hit is known, and they recommend using the mbedTLS library instead of SSL. Apparently it's a drop in replacement for the ESP functions - remove libssl from your list and add mbedtls

http://www.espressif.com/en/support/download/sdks-demos

[djphoenix]

@davydnorris we have moved to mbedTLS since 2.0.0...

[funwun]

any update?

[dtran123]

I started having secured https problems when master and dev got upgraded to 2.0.0 SDK. Since we upgraded to mbedTLS since 2.0.0, it appears that possibly mbedTLS is breaking things that used to work when we had SSL lib. Either that or something in the new SDK version. That is why I am still stuck with 1.5.4.1.

[eth0lv]

Hello. Someone decided this problem (I use NodeMCU firmware)? I want connect to Telegram API, but have errors: client handshake start. please start sntp first ! please start sntp first ! client handshake failed! Reason:[-0x7200]

[georeb]

I am also having the same issue...

http.get("https://httpbin.org/ip", nil, function(code, data)
>> if (code < 0) then
>> print("HTTP request failed")
>> else
>> print(code, data)
end
end)

> client handshake start.
please start sntp first !
client handshake failed!
Reason:[-0x7780]
HTTP client: Disconnected with error: 9
HTTP client: Connection timeout

[pjsg]

Do you have sntp running? Is your local clock set correctly. TLS requires a reasonable degree of synchronization between the local and remote clock.

If you just build the firmware with rtctime and sntp, then you just need to do

sntp.sync()

once you have network connectivity. If you are going to be running for a long time, then you may want to do

sntp.sync(nil, nil, nil, 1)

which will continually keep the clock in sync. You might argue that this should be the default behavior (and I might well agree with that!)

[karrots]

The answer is in the error messages posted. HTTPS can't validate a connection without valid clock to verify certificates. You'll need to run a sntp sync before making the connection. Hence the error message

please start sntp first !

[georeb]

Thanks or the speedy response @pjsg and @karrots !

I did see the please start sntp first ! and assumed that it was a sync error. However, the problem still exists :(

I've tried this...

sntp.sync()
http.get("https://now.httpbin.org/", nil, function (code, resp) print(code, resp) end)

and this was the response...

> client handshake start.
please start sntp first !
client handshake failed!
Reason:[-0x7780]
HTTP client: Disconnected with error: 9
HTTP client: Connection timeout

[eth0lv]

@karrots, @pjsg Yes, I start sntp.sync() and then try tslconn:connect(443,"api.telegram.org") but I have this errors.

[karrots]

First time use of sntp.sync() requires a server to be specified.

https://nodemcu.readthedocs.io/en/master/en/modules/sntp/

[georeb]

@karrots

I've tried this and it still throws an error...

sntp.sync('pool.ntp.org',
  function(sec, usec, server)
    print("Clock Synced: "..sec..", "..usec..", "..server)
    end,
  function(error_code)
    print("Clock Sync Failed: "..sntp_connect_status_codes[error_code])
  end
)

-- wait for clock to sync

http.get("https://now.httpbin.org/", nil, function (code, resp) print(code, resp) end)