Error in BearSSL when attempting to connect to secure MQTT broker

[schnorea]

Basic Infos

• [x] This issue complies with the issue POLICY doc.
• [x] I have read the documentation at readthedocs and the issue is not addressed there.
• [x] I have tested that the issue is present in current master branch (aka latest git).
• [x] I have searched the issue tracker for a similar issue.
• [x] If there is a stack dump, I have decoded it.
• [x] I have filled out all fields below.

Platform

• Hardware: ESP-12

• Core Version: SDK:2.2.2-dev(38a443e)/Core:2.7.4=20704000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be and with straight from master.

• Development Env: Arduino IDE

• Operating System: Ubuntu & MacOS

Settings in IDE

• Module: Nodemcu
• Flash Mode: qio
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: nodemcu
• Flash Frequency: ?
• CPU Frequency: 160MHz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

Attempting to connect to AWS IoT. MQTT client sometime is rarely successful most of the times it errors out in BearSSL. I know the AWS IoT setup works as the credentials used on other platforms and as I said it sometimes works. Same setup used on ESP8266 with sometime success. When success works as expected. When it fails the wifi connects, the SNTP gets the time, the MQTT client gets an IP addess from AWS, then attempts to connect and fails which causes and exception. The exception is either a 28 or a 9 depending on the configuration but is always associated with the same line of code ( br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 1871).

This version uses the MQTT client but the PubSubClient has the same behavior so i don't think it has to do with these libraries.

I have built this on both my MAC Powerbook and Ubuntu 18.04 to the same effect.

I have attempted to use GDB with some success but tracing this problem usually results in a segfault on target and then GDB exits abruptly. This eliminates the ability to do a backtrace.

I have looked at the max stack usage and this doesn't look like the problem. One interesting thing is that when it fails the max has like three different levels. When successful it is always the same value.

When it fails it looks like this:

Max stack 4580, 4532, 4564, 4548 or something else.

When successful it is always:

Max stack 4596

When successful the code will print out messages sent via the AWS IOT Core test client as would be expected.

What i show is with 2.7.4 but i have pulled master and it has very similar behavior but doesn't print out the exception stack so i didn't include that.

MCVE Sketch

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>
#include <MQTT.h>
#include <ArduinoJson.h>
#include <time.h>
#include <StackThunk.h>

#define emptyString String()

#include "secrets.h"

const int MQTT_PORT = 8883;
const char MQTT_SUB_TOPIC[] = "robo/cmds";
const char MQTT_PUB_TOPIC[] = "$aws/things/" THINGNAME "/shadow/update";

#ifdef USE_SUMMER_TIME_DST
uint8_t DST = 1;
#else
uint8_t DST = 0;
#endif

WiFiClientSecure net;

BearSSL::X509List cert(cacert);
BearSSL::X509List client_crt(client_cert);
BearSSL::PrivateKey key(privkey);

MQTTClient client;

unsigned long lastMillis = 0;
time_t now;
time_t nowish = 1510592825;

void NTPConnect(void)
{
  Serial.print("Setting time using SNTP");
  configTime(TIME_ZONE * 3600, DST * 3600, "pool.ntp.org", "time.nist.gov");
  now = time(nullptr);
  while (now < nowish)
  {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("done!");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));
}

void messageReceived(String &topic, String &payload)
{
  Serial.println("Received [" + topic + "]: " + payload);
}

void lwMQTTErr(lwmqtt_err_t reason)
{
  if (reason == lwmqtt_err_t::LWMQTT_SUCCESS)
    Serial.print("Success");
  else if (reason == lwmqtt_err_t::LWMQTT_BUFFER_TOO_SHORT)
    Serial.print("Buffer too short");
  else if (reason == lwmqtt_err_t::LWMQTT_VARNUM_OVERFLOW)
    Serial.print("Varnum overflow");
  else if (reason == lwmqtt_err_t::LWMQTT_NETWORK_FAILED_CONNECT)
    Serial.print("Network failed connect");
  else if (reason == lwmqtt_err_t::LWMQTT_NETWORK_TIMEOUT)
    Serial.print("Network timeout");
  else if (reason == lwmqtt_err_t::LWMQTT_NETWORK_FAILED_READ)
    Serial.print("Network failed read");
  else if (reason == lwmqtt_err_t::LWMQTT_NETWORK_FAILED_WRITE)
    Serial.print("Network failed write");
  else if (reason == lwmqtt_err_t::LWMQTT_REMAINING_LENGTH_OVERFLOW)
    Serial.print("Remaining length overflow");
  else if (reason == lwmqtt_err_t::LWMQTT_REMAINING_LENGTH_MISMATCH)
    Serial.print("Remaining length mismatch");
  else if (reason == lwmqtt_err_t::LWMQTT_MISSING_OR_WRONG_PACKET)
    Serial.print("Missing or wrong packet");
  else if (reason == lwmqtt_err_t::LWMQTT_CONNECTION_DENIED)
    Serial.print("Connection denied");
  else if (reason == lwmqtt_err_t::LWMQTT_FAILED_SUBSCRIPTION)
    Serial.print("Failed subscription");
  else if (reason == lwmqtt_err_t::LWMQTT_SUBACK_ARRAY_OVERFLOW)
    Serial.print("Suback array overflow");
  else if (reason == lwmqtt_err_t::LWMQTT_PONG_TIMEOUT)
    Serial.print("Pong timeout");
}

void lwMQTTErrConnection(lwmqtt_return_code_t reason)
{
  if (reason == lwmqtt_return_code_t::LWMQTT_CONNECTION_ACCEPTED)
    Serial.print("Connection Accepted");
  else if (reason == lwmqtt_return_code_t::LWMQTT_UNACCEPTABLE_PROTOCOL)
    Serial.print("Unacceptable Protocol");
  else if (reason == lwmqtt_return_code_t::LWMQTT_IDENTIFIER_REJECTED)
    Serial.print("Identifier Rejected");
  else if (reason == lwmqtt_return_code_t::LWMQTT_SERVER_UNAVAILABLE)
    Serial.print("Server Unavailable");
  else if (reason == lwmqtt_return_code_t::LWMQTT_BAD_USERNAME_OR_PASSWORD)
    Serial.print("Bad UserName/Password");
  else if (reason == lwmqtt_return_code_t::LWMQTT_NOT_AUTHORIZED)
    Serial.print("Not Authorized");
  else if (reason == lwmqtt_return_code_t::LWMQTT_UNKNOWN_RETURN_CODE)
    Serial.print("Unknown Return Code");
}

void connectToMqtt(bool nonBlocking = false)
{
  Serial.print("MQTT connecting ");
  while (!client.connected())
  {
    if (client.connect(THINGNAME))
    {
      Serial.println("connected!");
      if (!client.subscribe(MQTT_SUB_TOPIC))
        lwMQTTErr(client.lastError());
    }
    else
    {
      Serial.print("SSL Error Code: ");
      Serial.println(net.getLastSSLError());
      Serial.print("failed, reason -> ");
      lwMQTTErrConnection(client.returnCode());
      if (!nonBlocking)
      {
        Serial.println(" < try again in 5 seconds");
        delay(5000);
      }
      else
      {
        Serial.println(" <");
      }
    }
    Serial.printf("Max stack %d.", stack_thunk_get_max_usage());
    if (nonBlocking)
      break;
  }
}

void connectToWiFi(String init_str)
{
  if (init_str != emptyString)
    Serial.print(init_str);
  while (WiFi.status() != WL_CONNECTED)
  {
    Serial.print(".");
    delay(1000);
  }
  if (init_str != emptyString)
    Serial.println("ok!");
}

void checkWiFiThenMQTT(void)
{
  connectToWiFi("Checking WiFi");
  connectToMqtt();
}

unsigned long previousMillis = 0;
const long interval = 5000;

void checkWiFiThenMQTTNonBlocking(void)
{
  connectToWiFi(emptyString);
  if (millis() - previousMillis >= interval && !client.connected()) {
    previousMillis = millis();
    connectToMqtt(true);
  }
}

void checkWiFiThenReboot(void)
{
  connectToWiFi("Checking WiFi");
  Serial.print("Rebooting");
  ESP.restart();
}

void sendData(void)
{
  DynamicJsonDocument jsonBuffer(JSON_OBJECT_SIZE(3) + 100);
  JsonObject root = jsonBuffer.to<JsonObject>();
  JsonObject state = root.createNestedObject("state");
  JsonObject state_reported = state.createNestedObject("reported");
  state_reported["value"] = random(100);
  Serial.printf("Sending  [%s]: ", MQTT_PUB_TOPIC);
  serializeJson(root, Serial);
  Serial.println();
  char shadow[measureJson(root) + 1];
  serializeJson(root, shadow, sizeof(shadow));
  if (!client.publish(MQTT_PUB_TOPIC, shadow, false, 0))
    lwMQTTErr(client.lastError());
}

void setup()
{
  Serial.begin(115200);
  //gdbstub_init();
  delay(5000);
  Serial.println();
  Serial.println();
  WiFi.hostname(THINGNAME);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, pass);
  connectToWiFi(String("Attempting to connect to SSID: ") + String(ssid));

  NTPConnect();

  net.setTrustAnchors(&cert);
  net.setClientRSACert(&client_crt, &key);
  client.begin(MQTT_HOST, MQTT_PORT, net);
  client.onMessage(messageReceived);

  connectToMqtt();
}

void loop()
{
  //now = time(nullptr);
  if (!client.connected())
  {
    Serial.print("Not Connected");
    checkWiFiThenMQTT();
    //checkWiFiThenMQTTNonBlocking();
    //checkWiFiThenReboot();
  }
  else
  {
    client.loop();
    if (millis() - lastMillis > 5000)
    {
      lastMillis = millis();
      sendData();
    }
  }
}

Debug Messages

SDK:2.2.2-dev(38a443e)/Core:2.7.4=20704000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 0 (4)
reconnect

scandone
Attempting to connect to SSID: ARSnet...scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 15
cnt 

connected with ARSnet, channel 6
dhcp client start...
ip:192.168.2.122,mask:255.255.255.0,gw:192.168.2.1
ok!
Setting time using SNTP..................pm open,type:2 0
...........done!
Current time: Thu Dec 31 00:24:09 2020
MQTT connecting BSSL:_connectSSL: start connection
BSSL:_run_until: Timeout
BSSL:_wait_for_handshake: failed
BSSL:Couldn't connect. Error = 'Unknown error code.'
SSL Error Code: 0
failed, reason -> Connection Accepted < try again in 5 seconds
Max stack 4580.Fatal exception 28(LoadProhibitedCause):
epc1=0x4022451e, epc2=0x00000000, epc3=0x00000000, excvaddr=0x0000000c, depc=0x00000000

Decoded Exception

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
PC: 0x4022309a: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 1871
EXCVADDR: 0x0000000c

Decoding stack results
0x402229e5: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 1528
0x40221120: jump_handshake at src/ssl/ssl_engine.c line 1081
0x40221605: br_ssl_engine_recvrec_ack at src/ssl/ssl_engine.c line 1206
0x402060de: BearSSL::WiFiClientSecure::_run_until(unsigned int, bool) at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 540
0x40206f08: HardwareSerial::write(unsigned char const*, unsigned int) at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266/HardwareSerial.h line 165
0x402072a1: Print::write(char const*) at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266/Print.h line 62
0x4020628d: BearSSL::WiFiClientSecure::available() at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 386
0x402053aa: BearSSL::WiFiClientSecure::connected() at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 260
0x40208b2d: __delay(unsigned long) at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_wiring.cpp line 57
0x4020a65d: MQTTClient::connected() at /home/ccdocker/Arduino/libraries/MQTT/src/MQTTClient.cpp line 398
0x40202330: connectToMqtt(bool) at /home/ccdocker/projects/2020/robot_iot_3/robot_iot_3.ino line 139
0x4020d7de: strdup at /home/earle/src/esp-quick-toolchain/repo/newlib/newlib/libc/string/strdup.c line 11
0x402024d1: setup() at /home/ccdocker/projects/2020/robot_iot_3/robot_iot_3.ino line 259
0x402085dc: loop_wrapper() at /home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 194

[earlephilhower]

Can you please enable full debugging in the IDE and get the logs? That often gives a better idea about handshake difficulties and such. There were also 3.0.0 changes to fix reporting of some errors that you may profit from, so consider trying the master, too.

Your crash is pretty obviously a nullptr deref inside MQTTClient (the execVADDR = 0x0000 + some offset). You would need to check the library code for that, it's not part of our stuff. Could be an OOM error on a new or a malloc not being handled properly (you need 26KB+ free, mostly contiguous for SSL, and the JSON libraries I've seen are all monster memory hogs).

[schnorea]

On the logs I will give it a go and post them. I tried 3.0.0 and the only difference was that I didn't get the exception print out so I need to find out where that get's enabled. I also tried multiple MQTTClients had the same issue pointing to BearSSL. I don't need JSON lib so i will remove that and try again.

Thanks for the pointers.

[schnorea]

Removing JSON didn't change anything except it loads a lot quicker.

[schnorea]

IDE Build Log

/home/ccdocker/bin/arduino-1.8.13/arduino-builder -dump-prefs -logger=machine -hardware /home/ccdocker/bin/arduino-1.8.13/hardware -hardware /home/ccdocker/.arduino15/packages -hardware /home/ccdocker/Arduino/hardware -tools /home/ccdocker/bin/arduino-1.8.13/tools-builder -tools /home/ccdocker/bin/arduino-1.8.13/hardware/tools/avr -tools /home/ccdocker/.arduino15/packages -built-in-libraries /home/ccdocker/bin/arduino-1.8.13/libraries -libraries /home/ccdocker/Arduino/libraries -fqbn=esp8266com:esp8266:nodemcuv2:xtal=160,vt=flash,exception=disabled,ssl=basic,eesz=4M,led=2,ip=lm2f,dbg=Serial,lvl=SSLTLS_MEMHTTP_CLIENTHTTP_SERVERCOREWIFIHTTP_UPDATEUPDATEROTAOOMMDNS,wipe=all,baud=115200 -vid-pid=10C4_EA60 -ide-version=10813 -build-path /tmp/arduino_build_649514 -warnings=all -build-cache /tmp/arduino_cache_141023 -prefs=build.warn_data_percentage=75 -verbose /home/ccdocker/projects/2020/robot_iot_3a/robot_iot_3a.ino
/home/ccdocker/bin/arduino-1.8.13/arduino-builder -compile -logger=machine -hardware /home/ccdocker/bin/arduino-1.8.13/hardware -hardware /home/ccdocker/.arduino15/packages -hardware /home/ccdocker/Arduino/hardware -tools /home/ccdocker/bin/arduino-1.8.13/tools-builder -tools /home/ccdocker/bin/arduino-1.8.13/hardware/tools/avr -tools /home/ccdocker/.arduino15/packages -built-in-libraries /home/ccdocker/bin/arduino-1.8.13/libraries -libraries /home/ccdocker/Arduino/libraries -fqbn=esp8266com:esp8266:nodemcuv2:xtal=160,vt=flash,exception=disabled,ssl=basic,eesz=4M,led=2,ip=lm2f,dbg=Serial,lvl=SSLTLS_MEMHTTP_CLIENTHTTP_SERVERCOREWIFIHTTP_UPDATEUPDATEROTAOOMMDNS,wipe=all,baud=115200 -vid-pid=10C4_EA60 -ide-version=10813 -build-path /tmp/arduino_build_649514 -warnings=all -build-cache /tmp/arduino_cache_141023 -prefs=build.warn_data_percentage=75 -verbose /home/ccdocker/projects/2020/robot_iot_3a/robot_iot_3a.ino
Using board 'nodemcuv2' from platform in folder: /home/ccdocker/Arduino/hardware/esp8266com/esp8266
Using core 'esp8266' from platform in folder: /home/ccdocker/Arduino/hardware/esp8266com/esp8266
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/python3/python3 /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/makecorever.py --build_path /tmp/arduino_build_649514 --platform_path /home/ccdocker/Arduino/hardware/esp8266com/esp8266 --version unix-2.7.4
Detecting libraries used...
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/include -I/tmp/arduino_build_649514/core -c -w -Os -g -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++11 -ffunction-sections -fdata-sections -fno-exceptions -DNEW_OOM_ABORT -DBEARSSL_SSL_BASIC -w -x c++ -E -CC -DNONOSDK22x_190703=1 -DF_CPU=160000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_OOM -DDEBUG_ESP_MDNS -DARDUINO=10813 -DARDUINO_ESP8266_NODEMCU -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU\"" -DLED_BUILTIN=2 -DFLASHMODE_DIO -DESP8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/variants/nodemcu /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp -o /dev/null -DARDUINO_LIB_DISCOVERY_PHASE
Alternatives for ESP8266WiFi.h: [ESP8266WiFi@1.0]
ResolveLibrary(ESP8266WiFi.h)
  -> candidates: [ESP8266WiFi@1.0]
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/include -I/tmp/arduino_build_649514/core -c -w -Os -g -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++11 -ffunction-sections -fdata-sections -fno-exceptions -DNEW_OOM_ABORT -DBEARSSL_SSL_BASIC -w -x c++ -E -CC -DNONOSDK22x_190703=1 -DF_CPU=160000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_OOM -DDEBUG_ESP_MDNS -DARDUINO=10813 -DARDUINO_ESP8266_NODEMCU -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU\"" -DLED_BUILTIN=2 -DFLASHMODE_DIO -DESP8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/variants/nodemcu -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp -o /dev/null -DARDUINO_LIB_DISCOVERY_PHASE
Alternatives for MQTT.h: [MQTT@2.4.8]
ResolveLibrary(MQTT.h)
  -> candidates: [MQTT@2.4.8]
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/include -I/tmp/arduino_build_649514/core -c -w -Os -g -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++11 -ffunction-sections -fdata-sections -fno-exceptions -DNEW_OOM_ABORT -DBEARSSL_SSL_BASIC -w -x c++ -E -CC -DNONOSDK22x_190703=1 -DF_CPU=160000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_OOM -DDEBUG_ESP_MDNS -DARDUINO=10813 -DARDUINO_ESP8266_NODEMCU -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU\"" -DLED_BUILTIN=2 -DFLASHMODE_DIO -DESP8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/variants/nodemcu -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src -I/home/ccdocker/Arduino/libraries/MQTT/src /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp -o /dev/null -DARDUINO_LIB_DISCOVERY_PHASE
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/BearSSLHelpers.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/CertStoreBearSSL.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFi.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiAP.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiGratuitous.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiMulti.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiSTA-WPS.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiScan.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClient.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureAxTLS.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiServer.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiServerSecureAxTLS.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiServerSecureBearSSL.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/WiFiUdp.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/libraries/MQTT/src/MQTTClient.cpp
Using cached library dependencies for file: /home/ccdocker/Arduino/libraries/MQTT/src/lwmqtt/client.c
Using cached library dependencies for file: /home/ccdocker/Arduino/libraries/MQTT/src/lwmqtt/helpers.c
Using cached library dependencies for file: /home/ccdocker/Arduino/libraries/MQTT/src/lwmqtt/packet.c
Using cached library dependencies for file: /home/ccdocker/Arduino/libraries/MQTT/src/lwmqtt/string.c
Generating function prototypes...
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/include -I/tmp/arduino_build_649514/core -c -w -Os -g -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++11 -ffunction-sections -fdata-sections -fno-exceptions -DNEW_OOM_ABORT -DBEARSSL_SSL_BASIC -w -x c++ -E -CC -DNONOSDK22x_190703=1 -DF_CPU=160000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_OOM -DDEBUG_ESP_MDNS -DARDUINO=10813 -DARDUINO_ESP8266_NODEMCU -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU\"" -DLED_BUILTIN=2 -DFLASHMODE_DIO -DESP8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/variants/nodemcu -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src -I/home/ccdocker/Arduino/libraries/MQTT/src /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp -o /tmp/arduino_build_649514/preproc/ctags_target_for_gcc_minus_e.cpp -DARDUINO_LIB_DISCOVERY_PHASE
/home/ccdocker/bin/arduino-1.8.13/tools-builder/ctags/5.8-arduino11/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives /tmp/arduino_build_649514/preproc/ctags_target_for_gcc_minus_e.cpp
Compiling sketch...
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/python3/python3 /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/signing.py --mode header --publickey /home/ccdocker/projects/2020/robot_iot_3a/public.key --out /tmp/arduino_build_649514/core/Updater_Signing.h
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/include -I/tmp/arduino_build_649514/core -c -Wall -Wextra -Os -g -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++11 -MMD -ffunction-sections -fdata-sections -fno-exceptions -DNEW_OOM_ABORT -DBEARSSL_SSL_BASIC -DNONOSDK22x_190703=1 -DF_CPU=160000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_OOM -DDEBUG_ESP_MDNS -DARDUINO=10813 -DARDUINO_ESP8266_NODEMCU -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU\"" -DLED_BUILTIN=2 -DFLASHMODE_DIO -DESP8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/variants/nodemcu -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src -I/home/ccdocker/Arduino/libraries/MQTT/src /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp -o /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp.o
Compiling libraries...
Compiling library "ESP8266WiFi"
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiAP.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/include -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/include -I/tmp/arduino_build_649514/core -c -Wall -Wextra -Os -g -mlongcalls -mtext-section-literals -fno-rtti -falign-functions=4 -std=gnu++11 -MMD -ffunction-sections -fdata-sections -fno-exceptions -DNEW_OOM_ABORT -DBEARSSL_SSL_BASIC -DNONOSDK22x_190703=1 -DF_CPU=160000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DDEBUG_ESP_PORT=Serial -DDEBUG_ESP_SSL -DDEBUG_ESP_TLS_MEM -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_ESP_OOM -DDEBUG_ESP_MDNS -DARDUINO=10813 -DARDUINO_ESP8266_NODEMCU -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_NODEMCU\"" -DLED_BUILTIN=2 -DFLASHMODE_DIO -DESP8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/cores/esp8266 -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/variants/nodemcu -I/home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src -I/home/ccdocker/Arduino/libraries/MQTT/src /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/BearSSLHelpers.cpp -o /tmp/arduino_build_649514/libraries/ESP8266WiFi/BearSSLHelpers.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/CertStoreBearSSL.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiGeneric.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiGratuitous.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiMulti.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiSTA-WPS.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiScan.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiClient.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiSTA.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiClientSecureBearSSL.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiServer.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiServerSecureBearSSL.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiServerSecureAxTLS.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiClientSecureAxTLS.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiUdp.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/BearSSLHelpers.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/CertStoreBearSSL.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiAP.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiGeneric.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiGratuitous.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiMulti.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiSTA-WPS.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiSTA.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFiScan.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiClient.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiClientSecureAxTLS.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiClientSecureBearSSL.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiServer.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiServerSecureAxTLS.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiServerSecureBearSSL.cpp.o
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-ar cru /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/ESP8266WiFi/WiFiUdp.cpp.o
Compiling library "MQTT"
Using previously compiled file: /tmp/arduino_build_649514/libraries/MQTT/MQTTClient.cpp.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/client.c.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/helpers.c.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/packet.c.o
Using previously compiled file: /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/string.c.o
Compiling core...
Using precompiled core: /tmp/arduino_cache_141023/core/core_26e1ec1ef89b3cf4115e9604b2a00718.a
Linking everything together...
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-gcc -CC -E -P -DVTABLES_IN_FLASH /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld/eagle.app.v6.common.ld.h -o /tmp/arduino_build_649514/local.eagle.app.v6.common.ld
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-gcc -fno-exceptions -DNEW_OOM_ABORT -Wl,-Map -Wl,/tmp/arduino_build_649514/robot_iot_3a.ino.map -g -Wall -Wextra -Os -nostdlib -Wl,--no-check-sections -u app_entry -u _printf_float -u _scanf_float -Wl,-static -L/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib -L/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK22x_190703 -L/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld -L/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/lib -Teagle.flash.4m.ld -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read -o /tmp/arduino_build_649514/robot_iot_3a.ino.elf -Wl,--start-group /tmp/arduino_build_649514/sketch/robot_iot_3a.ino.cpp.o /tmp/arduino_build_649514/libraries/ESP8266WiFi/ESP8266WiFi.a /tmp/arduino_build_649514/libraries/MQTT/MQTTClient.cpp.o /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/client.c.o /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/helpers.c.o /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/packet.c.o /tmp/arduino_build_649514/libraries/MQTT/lwmqtt/string.c.o /tmp/arduino_cache_141023/core/core_26e1ec1ef89b3cf4115e9604b2a00718.a -lhal -lphy -lpp -lnet80211 -llwip2-536-feat -lwpa -lcrypto -lmain -lwps -lbearssl -laxtls -lespnow -lsmartconfig -lairkiss -lwpa2 -lstdc++ -lm -lc -lgcc -Wl,--end-group -L/tmp/arduino_build_649514
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/python3/python3 /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/elf2bin.py --eboot /home/ccdocker/Arduino/hardware/esp8266com/esp8266/bootloaders/eboot/eboot.elf --app /tmp/arduino_build_649514/robot_iot_3a.ino.elf --flash_mode dio --flash_freq 40 --flash_size 4M --path /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin --out /tmp/arduino_build_649514/robot_iot_3a.ino.bin
Creating BIN file "/tmp/arduino_build_649514/robot_iot_3a.ino.bin" using "/home/ccdocker/Arduino/hardware/esp8266com/esp8266/bootloaders/eboot/eboot.elf" and "/tmp/arduino_build_649514/robot_iot_3a.ino.elf"
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/python3/python3 /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/signing.py --mode sign --privatekey /home/ccdocker/projects/2020/robot_iot_3a/private.key --bin /tmp/arduino_build_649514/robot_iot_3a.ino.bin --out /tmp/arduino_build_649514/robot_iot_3a.ino.bin.signed --legacy /tmp/arduino_build_649514/robot_iot_3a.ino.bin.legacy_sig
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/python3/python3 /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/sizes.py --elf /tmp/arduino_build_649514/robot_iot_3a.ino.elf --path /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin
Executable segment sizes:
IROM   : 366252          - code in flash         (default or ICACHE_FLASH_ATTR) 
IRAM   : 28600   / 32768 - code in IRAM          (ICACHE_RAM_ATTR, ISRs...) 
DATA   : 1340  )         - initialized variables (global, static) in RAM/HEAP 
RODATA : 2248  ) / 81920 - constants             (global, static) in RAM/HEAP 
BSS    : 25584 )         - zeroed variables      (global, static) in RAM/HEAP 
Using library ESP8266WiFi at version 1.0 in folder: /home/ccdocker/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi 
Using library MQTT at version 2.4.8 in folder: /home/ccdocker/Arduino/libraries/MQTT 
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-size -A /tmp/arduino_build_649514/robot_iot_3a.ino.elf
Sketch uses 398440 bytes (38%) of program storage space. Maximum is 1044464 bytes.
Global variables use 29172 bytes (35%) of dynamic memory, leaving 52748 bytes for local variables. Maximum is 81920 bytes.
/home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/python3/python3 /home/ccdocker/Arduino/hardware/esp8266com/esp8266/tools/upload.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 erase_flash --before default_reset --after hard_reset write_flash 0x0 /tmp/arduino_build_649514/robot_iot_3a.ino.bin 
esptool.py v2.8
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 60:01:94:2b:ce:38
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Erasing flash (this may take a while)...
Chip erase completed successfully in 6.1s
Compressed 402592 bytes to 293099...
Wrote 402592 bytes (293099 compressed) at 0x00000000 in 25.9 seconds (effective 124.2 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

[earlephilhower]

Thanks, but we need the Serial log to tell what's going on w/the connection.

[schnorea]

Failing output log with all debug enabled.

This log has an example of failure (the exception is decoded above).

SDK:2.2.2-dev(38a443e)/Core:2.7.4=20704000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 15
cnt 

connected with ARSnet, channel 6
dhcp client start...
wifi evt: 0
ip:192.168.2.122,mask:255.255.255.0,gw:192.168.2.1
wifi evt: 3

Attempting to connect to SSID: ARSnet...scandone
ok!
Setting time using SNTP...pm open,type:2 0
............done!
Current time: Thu Dec 31 04:13:13 2020
MQTT connecting [hostByName] request IP for: a3kf0d3b3a0ocb-ats.iot.us-east-1.amazonaws.com
[hostByName] Host: a3kf0d3b3a0ocb-ats.iot.us-east-1.amazonaws.com IP: 3.222.90.31
:ref 1
BSSL:_connectSSL: start connection
:wr 139 0
:wrc 139 139 0
:ack 139
:rn 90
:rd 5, 90, 0
:rdi 90, 5
:rd 85, 90, 5
:rdi 85, 85
:c0 85, 90
:rn 536
:rd 5, 536, 0
:rdi 536, 5
:rd 531, 536, 5
:rdi 531, 531
:c0 531, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
BSSL:_run_until: Timeout
BSSL:_wait_for_handshake: failed
BSSL:Couldn't connect. Error = 'Unknown error code.'
SSL Error Code: 0
failed, reason -> Connection Accepted < try again in 5 seconds
:rn 536
Max stack 4580.:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
Fatal exception 28(LoadProhibitedCause):
epc1=0x402238a6, epc2=0x00000000, epc3=0x00000000, excvaddr=0x0000000c, depc=0x00000000

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (28):
epc1=0x402238a6 epc2=0x00000000 epc3=0x00000000 excvaddr=0x0000000c depc=0x00000000

>>>stack>>>

ctx: bearssl
sp: 3fff0cd0 end: 3fff1140 offset: 0190
3fff0e60:  00000055 3fff3560 0000099b 402231f1  
3fff0e70:  deadbeef deadbeef deadbeef deadbeef  
3fff0e80:  deadbe4d deadbeef deadbeef deadbeef  
3fff0e90:  deadbeef deadbeef deadbeef deadbeef  
3fff0ea0:  deadbeef deadbeef deadbeef deadbeef  
3fff0eb0:  deadbeef deadbeef deadbeef deadbeef  
3fff0ec0:  deadbeef deadbeef deadbeef deadbeef  
3fff0ed0:  deadbeef deadbeef deadbeef deadbeef  
3fff0ee0:  deadbeef deadbeef deadbeef deadbeef  
3fff0ef0:  deadbeef deadbeef deadbeef deadbeef  
3fff0f00:  deadbeef deadbeef deadbeef deadbeef  
3fff0f10:  deadbeef deadbeef deadbeef deadbeef  
3fff0f20:  deadbeef deadbeef deadbeef deadbeef  
3fff0f30:  deadbeef deadbeef deadbeef deadbeef  
3fff0f40:  deadbeef deadbeef deadbeef deadbeef  
3fff0f50:  deadbeef deadbeef deadbeef deadbeef  
3fff0f60:  deadbeef deadbeef deadbeef deadbeef  
3fff0f70:  deadbeef deadbeef deadbeef deadbeef  
3fff0f80:  deadbeef deadbeef deadbeef deadbeef  
3fff0f90:  deadbeef deadbeef deadbeef deadbeef  
3fff0fa0:  deadbeef deadbeef deadbeef deadbeef  
3fff0fb0:  deadbeef deadbeef deadbeef deadbeef  
3fff0fc0:  deadbeef deadbeef deadbeef deadbeef  
3fff0fd0:  deadbeef deadbeef deadbeef deadbeef  
3fff0fe0:  deadbeef deadbeef deadbeef deadbeef  
3fff0ff0:  deadbeef deadbeef deadbeef deadbeef  
3fff1000:  deadbeef deadbeef deadbeef deadbeef  
3fff1010:  deadbeef deadbeef deadbeef deadbeef  
3fff1020:  deadbeef deadbeef deadbeef deadbeef  
3fff1030:  deadbeef deadbeef deadbeef deadbeef  
3fff1040:  deadbeef deadbeef deadbeef deadbeef  
3fff1050:  deadbeef deadbeef deadbeef deadbeef  
3fff1060:  deadbeef deadbeef deadbeef deadbeef  
3fff1070:  deadbeef deadbeef deadbeef deadbeef  
3fff1080:  deadbeef deadbeef deadbeef deadbeef  
3fff1090:  deadbeef 4025526f deadbeef deadbeef  
3fff10a0:  3fff3280 3fff3304 deadbeef 3fff3260  
3fff10b0:  deadbeef 3fff2a7c 3fff3280 deadbeef  
3fff10c0:  deadbeef 00000055 3fff2a60 deadbeef  
3fff10d0:  deadbeef deadbeef deadbeef 3fff347c  
3fff10e0:  3fff357c 00000000 3fff2a7c 4022192c  
3fff10f0:  3fff3260 00000218 deadbeef deadbeef  
3fff1100:  00000250 00000005 deadbeef 3ffeeb38  
3fff1110:  00003a98 0000089f 3fff2a7c 40221e11  
3fff1120:  0000008b deadbeef deadbeef deadbeef  
3fff1130:  deadbeef 00000000 3ffee9f8 40202660  

ctx: cont
sp: 3ffffec0 end: 3fffffc0 offset: 0000
3ffffec0:  00003a98 00000000 00000010 40206506  
3ffffed0:  00000ab7 00000000 73646e6f 40209b59  
3ffffee0:  00000000 00008bf4 00000008 00000000  
3ffffef0:  40105761 00aaa38c 000011e4 00000000  
3fffff00:  00000000 00000000 3ffee9f8 402066c5  
3fffff10:  3ffe88e4 3ffee940 3ffee9f8 40205782  
3fffff20:  40209161 00001388 3ffee940 4020acbd  
3fffff30:  00000000 3ffee940 3ffeeb00 402022f8  
3fffff40:  00000000 000022b3 3ffee940 4020df96  
3fffff50:  00000000 00000000 3ffee940 3ffeec40  
3fffff60:  3fffdad0 3ffee940 3ffee9f8 402024b5  
3fffff70:  3fff2400 0025002f 80efeffe 3fff2400  
3fffff80:  0025002f 80efeffe 6e535200 fe007465  
3fffff90:  80efeffe 3fff2200 001f001f 80efeffe  
3fffffa0:  feefeffe 00000000 3ffeec00 40208bfc  
3fffffb0:  feefeffe feefeffe 3ffe853c 401012a1  
<<<stack<<<

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3584, room 16 
tail 0
chksum 0xb0
csum 0xb0
v2843a5ac
~ld

[schnorea]

Successful output log

This is an example of when it works with full debug info.

embedded in the below log is this line:

Received [robo/cmds]: Robo cmd 0x81

which is the result of a MQTT message sent via AWS IoT Core.

SDK:2.2.2-dev(38a443e)/Core:2.7.4=20704000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be
wifi evt: 2
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 0 (4)
reconnect
wifi evt: 1
STA disconnect: 4

scandone
Attempting to connect to SSID: ARSnet...scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 15
cnt 

connected with ARSnet, channel 6
dhcp client start...
wifi evt: 0
ip:192.168.2.122,mask:255.255.255.0,gw:192.168.2.1
wifi evt: 3
ok!
Setting time using SNTP..................pm open,type:2 0
...........done!
Current time: Thu Dec 31 04:13:58 2020
MQTT connecting [hostByName] request IP for: a3kf0d3b3a0ocb-ats.iot.us-east-1.amazonaws.com
[hostByName] Host: a3kf0d3b3a0ocb-ats.iot.us-east-1.amazonaws.com IP: 52.20.209.4
:ref 1
BSSL:_connectSSL: start connection
:wr 139 0
:wrc 139 139 0
:ack 139
:rn 90
:rd 5, 90, 0
:rdi 90, 5
:rd 85, 90, 5
:rdi 85, 85
:c0 85, 90
:rn 536
:rd 5, 536, 0
:rdi 536, 5
:rd 531, 536, 5
:rdi 531, 531
:c0 531, 536
:rn 1072
:rd 1072, 1072, 0
:rdi 536, 536
:c 536, 536, 1072
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 1072
:rd 1072, 1072, 0
:rdi 536, 536
:c 536, 536, 1072
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 109
:rd 63, 109, 0
:rdi 109, 63
:rd 5, 109, 63
:rdi 46, 5
:rd 32, 109, 68
:rdi 41, 32
:rd 5, 109, 100
:rdi 9, 5
:rd 4, 109, 105
:rdi 4, 4
:c0 4, 109
:wr 517 0
:wrc 517 517 0
:wr 517 0
:wrc 517 517 0
:ack 517
:ack 517
:wr 378 0
:wrc 378 378 0
:wr 6 0
:wrc 6 6 0
:wr 85 0
:wrc 85 85 0
:ack 378
:ack 91
:rn 91
:rd 5, 91, 0
:rdi 6, 5
:rd 1, 91, 5
:rdi 1, 1
:c 1, 6, 91
:rd 5, 85, 0
:rdi 85, 5
:rd 80, 85, 5
:rdi 80, 80
:c0 80, 85
BSSL:Connected!
:wr 85 0
:wrc 85 85 0
:ack 85
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
connected!
:wr 85 0
:wrc 85 85 0
:ack 85
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
Max stack 4596.:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:rn 85
:rd 5, 85, 0
:rdi 85, 5
:rd 80, 85, 5
:rdi 80, 80
:c0 80, 85
Received [robo/cmds]: Robo cmd 0x81
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69
:wr 69 0
:wrc 69 69 0
:ack 69
:rn 69
:rd 5, 69, 0
:rdi 69, 5
:rd 64, 69, 5
:rdi 64, 64
:c0 64, 69

[earlephilhower]

Sorry, but that doesn't look like it has BearSSL debugging enabled. The certificates sent and received (they're public certs and no private cert/key dumped so should be safe to post) as well as more handshake info is generally dumped. Make sure you're using the 2nd to last setting (longest one) as that also catches malloc fails and full logs elsewhere.

For example:

Trying: api.github.com:443...[hostByName] request IP for: api.github.com
[hostByName] Host: api.github.com IP: 192.30.255.117
:ur 1
:dsrcv 1367
:del
:ref 1
BSSL:_connectSSL: start connection

_iobuf_in:       0x3fff0bcc
_iobuf_out:      0x3fffb85c
_iobuf_in_size:  16709
_iobuf_out_size: 597
:wr 137 0
:wrc 137 137 0
:ack 137
:rn 536
:rch 536, 536
:rch 1072, 536
:rch 1608, 536
:rd 5, 2144, 0
:rdi 536, 5
:rd 90, 2144, 5
:rdi 531, 90
:rd 5, 2144, 95
:rdi 441, 5
:rd 512, 2144, 100
:rdi 436, 436
:c 436, 536, 2144
:rdi 536, 76
BSSL:CERT: 30 82 06 d4 30 82 05 bc a0 03 02 01 02 02 10 0c 07 3b 67 6f 67 45 78 f9 99 81 48 52 84 46 51 30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05 00 30 70 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 15 30 13 06 03 55 04 0a 13 0c 44 69 67 69 43 65 72 74 20 49 6e 63 31 19 30 17 06 03 55 04 0b 13 10 77 77 77 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 31 2f 30 2d 06 03 55 04 03 13 26 44 69 67 69 43 65 72 74 20 53 48 41 32 20 48 69 67 68 20 41 73 73 75 72 61 6e 63 65 20 53 65 72 76 65 72 20 43 41 30 1e 17 0d 32 30 30 36 32 32 30 30 30 30 30 30 5a 17 0d 32 32 30 38 31 37 31 32 30 30 30 30 5a 30 68 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 13 30 11 06 03 55 04 08 13 0a 43 61 6c 69 66 6f 72 6e 69 61 31 16 30 14 06 03 55 04 07 13 0d 53 61 6e 20 46 72 61 6e 63 69 73 63 6f 31 15 30 13 
:rd 5, 1608, 76
:rdi 460, 5
:rd 512, 1608, 81
:rdi 455, 455
:c 455, 536, 1608
:rdi 536, 57
BSSL:CERT: 06 03 55 04 0a 13 0c 47 69 74 48 75 62 2c 20 49 6e 63 2e 31 15 30 13 06 03 55 04 03 0c 0c 2a 2e 67 69 74 68 75 62 2e 63 6f 6d 30 82 01 22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 a0 e6 d3 87 ac 6f 4e 3b:rch 1072, 536
 29 75 60 4c 4a 1e fa dd af 81 0a 37 c4 89 ad b5 8e 9d 1d 0c 55 da a4 b1 cb ab d3 14 bc d2 6d b8 d1 7c 36 1f 45 a1 06 25 32 63 7f 94 4c f4 d6 97 06 3f 24 f2 85 f5 83 8c 27 8a 7f 6a c8 46 e8 04 6f 4c 5f 4c 48 9a a6 80 c1 08 db 9c 6e 81 8b 54 59 f0 c6 6d 58 2a 3d 42 ea da 5d aa 6b 90 7b af 12 34 30 1b 22 5c af cd ee f2 3c 08 90 99 91 be 41 16 c6 e0 95 59 a9 d6 52 39 de e9 a3 02 e2 68 e3 f9 b5 56 ce ae 62 27 5e ff a3 94 1f 89 82 0f 5d ea 82 4d af de 0f 3b aa 04 4a 6f a4 85 43 80 11 35 f1 3b d6 66 80 68 97 6e 0a e9 79 
BSSL:CERT: 57 63 44 91 c1 e2 45 db dd 2c b9 2d 3d 16 76 af 0b a4 04 80 c0 10 35 26 f2 9f 38 43 a8 1d a5 19 79 f0 b0 60 98 40 f8 4c 54 53 5f 32 0e de 86 65 e7 4f 5c 21 7a c7 35 7f de a3 e0 8e b2 d3 02 42 64 40 14 21 20 96 14 09 54 1f 53 02 03 01 00 01 a3 82 03 70 30 82 03 6c 30 1f 06 03 55 1d 23 04 18 30 16 80 14 51 68 ff 90 af 02 07 75 3c cc d9 65 64 62 a2 12 b8 59 72 3b 30 1d 06 03 55 1d 0e 04 16 04 14 79 53 13 17 5:rch 1608, 424
3 da 1e a9 73 a9 9e 88 8d c2 53 d9 36 e2 e5 2a 30 23 06 03 55 1d 11 04 1c 30 1a 82 0c 2a 2e 67 69 74 68 75 62 2e 63 6f 6d 82 0a 67 69 74 68 75 62 2e 63 6f 6d 30 0e 06 03 55 1d 0f 01 01 ff 04 04 03 02 05 a0 30 1d 06 03 55 1d 25 04 16 30 14 06 08 2b 06 01 05 05 07 03 01 06 08 2b 06 01 05 05 07 03 02 30 75 06 03 55 1d 1f 04 6e 30 6c 30 34 a0 32 a0 30 86 2e 68 
:rd 5, 2032, 57
:rdi 479, 5
:rd 512, 2032, 62
:rdi 474, 474
:c 474, 536, 2032
:rdi 536, 38
BSSL:CERT: 74 74 70 3a 2f 2f 63 72 6c 33 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 73 68 61 32 2d 68 61 2d 73 65 72 76 65 72 2d 67 36 2e 63 72 6c 30 34 a0 32 a0 30 86 2e 68 74 74 70 3a 2f 2f 63 72 6c 34 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 73 68 61 32 2d 68 61 2d 73 65 72 76 65 72 2d 67 36 2e 63 72 6c 30 4c 06 03 55 1d 20 04 45 30 43 30 37 06 09 60 86 48 01 86 fd 6c 01 01 30 2a 30 28 06 08 2b 06 01 05 05 07 02 01 16 1c 68 74 74 70 73 3a 2f 2f 77 77 77 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 43 50 53 30 08 06 06 67 81 0c 01 02 02 30 81 83 06 08 2b 06 01 05 05 07 01 01 04 77 30 75 30 24 06 08 2b 06 01 05 05 07 30 01 86 18 68 74 74 70 3a 2f 2f 6f 63 73 70 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 30 4d 06 08 2b 06 01 05 05 07 30 02 86 41 68 74 74 70 3a 2f 2f 63 61 63 
BSSL:CERT: 65 72 74 73 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 44 69 67 69 43 65 72 74 53 48 41 32 48 69 67 68 41 73 73 75 72 61 6e 63 65 53 65 72 76 65 72 43 41 2e 63 72 74 30 0c 06 03 55 1d 13 01 01 ff 04 02 30 00 30 82 01 7b 06 0a 2b 06 01 04 01 d6 79 02 04 02 04 82 01 6b 04 82 01 67 01 65 00 75 00 29 79 be f0 9e 39 39 21 f0 56 73 9f 63 a5 77 e5 be 57 7d 9c 60 0a f8 f9 4d 5d 26 5c 25 5d c7 84 00 00 01 72 dd b6 2a de 00 00 04 03 00 46 30 44 02 20 66 d5 ea ba af 15 0e 8d 17 0d 2c 63 6a e1 f9 b7 22 7e d9 dc 01 10 b6 ea e6 b4 4d 2c 87 cf fb fb 02 20 55 2b 93 7a 7a 61 d4 8f 0f 1a 57 4e 39 03 56 b3 ad bf 41 3b 55 cf 9b 3f 6b 99 65 25 8b 5f d7 60 00 75 00 41 c8 ca b1 df 22 46 4a 10 c6 a1 3a 09 42 87 5e 4e 31 8b 1b 03 eb eb 4b c7 68 f0 90 62 96 06 f6 00 00 01 72 dd b6 2a 
:rd 5, 1496, 38

[earlephilhower]

Quickly ooking at the code, it's related to the BearSSL x509 context info where the x509 cert is null. W/o code, can't really say how that could happen. If you can carve out a MCVE w/o the 3rd party libs it would be helpful. W/o it, you're going to need to be self sufficient as it's using a private client cert, to a private AWS instance, using 3rd party code which means nobody but you can repro your runs. :(

[schnorea]

I'm also using the Basic SSL package (Low Memory)

[schnorea]

Yeah, a hear you:(

When you say "third party libs" which ones you talking about so I am clear about what I need to do?

I'll give it a try. Thanks for the encouragement.

[earlephilhower]

The JSON lib and MQTT libs aren't part of the core. Something simple and minimal that does a SSL connection attempt and gets to your failed state is what we'd need.

That said, I still think you're just out of memory and not catching it properly somewhere or in a library. It's not stack usage related, the present amount is fine for every case we know of, your reported usage is small, and when an overflow occurs it is reported as a SSL stack overflow via canaries or via massive memory corruption (not nulls).

[schnorea]

I woke up with a similar thought. What if that transaction is processing things to slowly? The evidence is this:

BSSL:_run_until: Timeout

And, if i had turned on time stamps you would see that this time out is always 15 seconds. What if sometimes that transaction takes less than 15 seconds and succeeds but most of the time it doesn't because inadequate hardware performance.

No, I like your explination better. Low memory issue draws program execution out in the sticks (exception) and it can't get back in time.

Okay the logs from above were without the JSON Lib. Next I'll try pulling the MQTT to get at the raw SSL connection.

Thanks again.

[schnorea]

After a few days and a broken leg I still haven't figured this out.

I tried to decypher how to test BearSSL without the MQTT library and after looking at openssl I can't figure that out. Is there an example that anyone can point me to?

The other complication is doing it without a network connection.

[earlephilhower]

Sorry to hear about the leg. I guess on the bright side, more time inside to hack?

FWIW, there was a very minor change to catch a malloc() failure while reading an EC private key. Most keys are RSA, so it may not be related at all, but see #7823. Root problem would still be OOM, and the old code would crash in the EC key reader immediately anyway.

For MQTT testing, you can set up a local mosquitto instance (that's how I debugged the client cert code while writing the BearSSL stuff) and come close to simulating AWS...while being able to see both sides of the conversation w/Wireshark and having both encryption keys.

[Jason2866]

@s-hadinger has modified/optimized bearssl (lower footprint) so it can be used with Tasmota and AWS IoT. It works stable :-) Maybe it helps you.

[s-hadinger]

Doc is here, I've been using for 18 months and it's super stable.

https://tasmota.github.io/docs/AWS-IoT/

Implementation details can be found here

https://tasmota.github.io/docs/TLS/#implementation-notes

[earlephilhower]

While I have reservations about Tasmota dynamically allocating the BSSL stack(heap fragmentation could cause failure easily in my experience when trying to get a 6kb block), the other stuff looks interesting. If you or the Tasmota team would be interested in adding that option to the IDE menus (like the "minimal codes" option already) I think it would be a good addition!

[s-hadinger]

I understand your reservation but ram is so precious that I need to spare it. Based on experience, tls handshakes are rare with Mqtt, while they would be common with https. This makes it sustainable on Tasmota and actually does not hurt fragmentation.

[Jason2866]

If you or the Tasmota team would be interested in adding that option to the IDE menus (like the "minimal codes" option already) I think it would be a good addition!

We dont use Arduino IDE, we use Platformio. I think i can speak for the whole Tasmota team we dont know how to add stuff to Arduino IDE. Sorry we cant contribute.

[schnorea]

Okay this works but...

It is not stable but...

The while loop in 'connectToMqtt' will generally run until it connects.

And then I can control my robot over the internet.

I will pull the latest and see if the fixes above make it more stable.

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>
#include <MQTT.h>
#include <time.h>
#include "robosend.h"

#include "secrets.h"

#define emptyString String()

WiFiClientSecure net;

BearSSL::X509List cert(cacert);
BearSSL::X509List client_crt(client_cert);
BearSSL::PrivateKey key(privkey);

MQTTClient client(32);

time_t now;
time_t nowish = 1510592825;

void NTPConnect(void)
{
  Serial.print(F("Setting time using SNTP"));
  configTime(TIME_ZONE * 3600, DST * 3600, "pool.ntp.org", "time.nist.gov");
  now = time(nullptr);
  while (now < nowish)
  {
    delay(500);
    Serial.print(F("."));
    now = time(nullptr);
  }
  Serial.println(F("done!"));
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print(F("Current time: "));
  Serial.print(asctime(&timeinfo));
}

void connectToWiFi(String init_str)
{
  if (init_str != emptyString)
    Serial.print(init_str);
  while (WiFi.status() != WL_CONNECTED)
  {
    Serial.print(F("."));
    delay(1000);
  }
  if (init_str != emptyString)
    Serial.println(F("ok!"));
}

void connectToMqtt() {
  while (!client.connect(THINGNAME)) {
    Serial.print(".");
    delay(1000);
  }
  Serial.println("\nconnected!");
  client.subscribe(MQTT_SUB_TOPIC);
}

void checkWiFiThenMQTT(void)
{
  connectToWiFi(F("Checking WiFi"));
  connectToMqtt();
}

#define LED 2  //On board LED

void messageReceived(String &topic, String &payload)
{
  digitalWrite(LED,0);
  Serial.println("Received [" + topic + "]: " + payload);

  char robo_cmd_str[sizeof(payload)];
  payload.toCharArray(robo_cmd_str, sizeof(robo_cmd_str));

  u_char robo_cmd = (u_char)strtol(robo_cmd_str, NULL, 16);

  robo_send(robo_cmd);
  digitalWrite(LED,1);
}

void changeState()
{
  digitalWrite(LED, !(digitalRead(LED)));  //Invert Current State of LED  
}

void setup()
{
  pinMode(LED,OUTPUT);
  digitalWrite(LED,1);

  Serial.begin(115200);
  delay(500);
  Serial.println("");
  Serial.println("RUN SETUP");
  Serial.print("D2 is");
  Serial.println(D2);

  WiFi.hostname(THINGNAME);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, pass);

  connectToWiFi(String(F("Attempting to connect to SSID: ")) + String(ssid));

  NTPConnect();

  net.setTrustAnchors(&cert);
  net.setClientRSACert(&client_crt, &key);

  client.begin(MQTT_HOST, MQTT_PORT, net);
  client.onMessage(messageReceived);

  print_info();

  Serial.println(F("Start MQTT Setup *******"));
  connectToMqtt();
  Serial.println(F("MQTT Setup DONE *******"));

  digitalWrite(LED,0);
  ESP.wdtFeed();
  delay(1000);
  ESP.wdtFeed();
  delay(1000);
  ESP.wdtFeed();
  delay(1000);
  ESP.wdtFeed();
  delay(1000);
  ESP.wdtFeed();
  digitalWrite(LED,1);

  Serial.println(F("Start Robo IR Setup *******"));
  robo_setup();
  Serial.println(F("Robo IR Setup DONE*******"));

}

int led_count = 0;

void loop()
{
  if (!client.connected())
  {
    Serial.print(F("Not Connected"));
    checkWiFiThenMQTT();
  }
  else
  {
    client.loop();
    delay(10);

    led_count++;
    if (led_count < 50) {
      digitalWrite(LED, 0);
    }
    else {
      digitalWrite(LED, 1);
    }
    if (led_count > 750) {
      led_count = 0;
    }
  }
}

Here are the other files.

#ifndef robosend_h
#define robosend_h

void robo_send(u_char robo_cmd);
void robo_callback();
void robo_setup();
void print_info();

#endif

#include <ESP8266WiFi.h>
#include "robosend.h"

#define LED_PIN D2

#define TIMER1_WRITE_VALUE 867

bool isLEDOn = false;

#define PIN_HIGH false
#define PIN_LOW true

#define START_WAIT 520
#define ONE_WAIT 260
#define ZERO_WAIT 65
#define BIT_WAIT 65

#define TX_IDLE 0x00
#define TX_START 0x80
#define TX_START_BIT 0x40
#define TX_BIT_HIGH 0x20
#define TX_BIT_LOW 0x10

volatile u_char tx_state = TX_IDLE;
volatile u_char tx_byte = 0;
volatile u_char tx_mask = 0x80;
volatile int tx_count = 0x00;

bool tx_pin_state = PIN_HIGH;

void tx_send(u_char byte_to_send)
{
  while (tx_state != TX_IDLE);
  tx_byte = byte_to_send;
  tx_mask = 0x80;
  tx_count = START_WAIT;
  tx_state = TX_START;
}

void ICACHE_RAM_ATTR tx_start_bit()
{
  tx_pin_state = PIN_HIGH;
  if (!tx_mask)
  {
    tx_state = TX_IDLE;
    return;
  }
  if (tx_byte & tx_mask) {
    tx_count = ONE_WAIT;
  }
  else {
    tx_count = ZERO_WAIT;
  }
  tx_state = TX_BIT_HIGH;
}

void ICACHE_RAM_ATTR tx_bit()
{
  tx_count -= 1;
  if (tx_state == TX_BIT_HIGH && tx_count < 0)
  {
    tx_pin_state = PIN_LOW;
    tx_count = BIT_WAIT;
    tx_state = TX_BIT_LOW;
  }
  if (tx_state == TX_BIT_LOW && tx_count < 0)
  {
    tx_pin_state = PIN_HIGH;
    tx_mask >>=1; 
    tx_state = TX_START_BIT;
  }
}

void ICACHE_RAM_ATTR tx_start() 
{
  tx_pin_state = PIN_LOW;
  tx_count -= 1;
  if (tx_count < 0)
  {
    tx_state = TX_START_BIT;
  }
}

void ICACHE_RAM_ATTR tx_state_machine()
{
  if (tx_state == TX_IDLE)
  {

  }

  if (tx_state == TX_START)
  {
    tx_start();
  }

  if (tx_state == TX_START_BIT)
  {
    tx_start_bit();
  }

  if (tx_state == TX_BIT_HIGH || tx_state == TX_BIT_LOW)
  {
    tx_bit();
  } 
}

void robo_send(u_char robo_cmd)
{
  u_char cmd_check = robo_cmd;
  cmd_check >>= 4;
  if (cmd_check == 0x08 || cmd_check == 0x09 || cmd_check == 0x0A || 
      cmd_check == 0x0B || cmd_check == 0x0C || cmd_check == 0x0D )
  {
    tx_send(robo_cmd);
  }
  else 
  {
    Serial.println("Fail, command has to be in the range of 0x80 - 0xD0");
  }
  Serial.println(robo_cmd);
}

void robo_callback()
{
//    if (isLEDOn && tx_pin_state)
//      digitalWrite(LED_PIN, 1);
//    else
//      digitalWrite(LED_PIN, 0);

    isLEDOn = !isLEDOn;
    tx_state_machine();
}

void ICACHE_RAM_ATTR onTimerISR(){
//    if (isLEDOn && tx_pin_state)
//      digitalWrite(LED_PIN, 1);
//    else
//      digitalWrite(LED_PIN, 0);

    digitalWrite(LED_PIN, isLEDOn && tx_pin_state);

    isLEDOn = !isLEDOn;
    tx_state_machine();
    timer1_write(TIMER1_WRITE_VALUE); 
}

void robo_setup()
{
  pinMode(LED_PIN,OUTPUT);
  timer1_attachInterrupt(onTimerISR);
  timer1_write(TIMER1_WRITE_VALUE); 
  timer1_enable(TIM_DIV1, TIM_EDGE, TIM_LOOP);
}

void print_info()
{
  Serial.print("ESP.getBootMode(); ");
  Serial.println(ESP.getBootMode());
  Serial.print("ESP.getSdkVersion(); ");
  Serial.println(ESP.getSdkVersion());
  Serial.print("ESP.getBootVersion(); ");
  Serial.println(ESP.getBootVersion());
  Serial.print("ESP.getChipId(); ");
  Serial.println(ESP.getChipId());
  Serial.print("ESP.getFlashChipSize(); ");
  Serial.println(ESP.getFlashChipSize());
  Serial.print("ESP.getFlashChipRealSize(); ");
  Serial.println(ESP.getFlashChipRealSize());
  Serial.print("ESP.getFlashChipSizeByChipId(); ");
  Serial.println(ESP.getFlashChipSizeByChipId());
  Serial.print("ESP.getFlashChipId(); ");
  Serial.println(ESP.getFlashChipId());
  Serial.print("ESP.getFreeHeap(); ");
  Serial.println(ESP.getFreeHeap());
}

[dalbert2]

In case it helps, I'm getting a similar crash in a RESTful client with much more memory available (typically > 28K heap free, >20K largest available block) which seems to point away from the MQTT client or low memory. However, I am using an EC certificate.

Target: ESP12E, Core:3.1.1 SDK:2.2.2-dev(38a443e) LWIP:2.1.3 Free Heap:28744 Largest:20816 Frag:24 Stack:1488

The crashes (which are intermittent), always seem to occur around the same place; see a few samples below: Exc 9 @ 40237195: /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/ssl/ssl_hs_client.c:1916 Exc 0 @ 40236d44: /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/ssl/ssl_hs_client.c:1551 Exc 0 @ 40236ce0: /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/ssl/ssl_hs_client.c:1531

It looks like the exceptions are happening during the SSL connection phase. The first attempt to contact the server is an HTTPS GET request for current time (the GET time... debug output). The crash appears to happen before the http.GET() returns. Note that most of the time this code works; the exceptions are intermittent.

More of the traces:

GET time...
BSSL:_connectSSL: start connection
                                             BSSL:_run_until: Timeout
                                                                     BSSL:_wait_for_handshake: failed
  BSSL:Couldn't connect. Error = 'Unknown error code.'
                                                      response=-1
Server error: connection failed
GET time...
           --------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (0):
epc1=0x80069c84 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

>>>stack>>>

ctx: bearssl
sp: 3fff3328 end: 3fff3598 offset: 0190
3fff34b8:  3fff5a54 3fff5d30 3fff5a30 40236d44
3fff34c8:  3fff3548 4023a70c 00000000 3fff1c84
3fff34d8:  40276a03 02000000 01030317 81156821
3fff34e8:  ee3999cf 17bf8630 dbfa5071 ff43cf62
3fff34f8:  3fff5fb9 3fff3548 00000003 3fff5294
3fff3508:  00000121 00000020 40276df9 deadbeef
...

GET time...BSSL:connect: Unable to connect TCP socket
                                                     response=-1
Server error: connection failed
GET time...BSSL:_connectSSL: start connection
                                             BSSL:_run_until: Timeout
                                                                     BSSL:_wait_for_handshake: failed
  BSSL:Couldn't connect. Error = 'Unknown error code.'
                                                      response=-1
Server error: connection failed
GET time...
           --------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (9):
epc1=0x40237195 epc2=0x00000000 epc3=0x00000000 excvaddr=0x2e73636d depc=0x00000000

>>>stack>>>

ctx: bearssl
sp: 3fff3328 end: 3fff3598 offset: 0190
3fff34b8:  3fffaad4 3fffadb0 3fffaab0 40236d44
3fff34c8:  3fff3548 4023a70c 00000000 deadbeef
3fff34d8:  40276a03 00000000 00030316 e9c7c010
3fff34e8:  84201b01 abb16c56 957aed8d 953b85bc
3fff34f8:  3fff50f1 3fff3548 00000003 3fffa3a4
3fff3508:  00000010 00000020 40276df9 deadbeef
...

GET time...BSSL:_connectSSL: start connection
                                                        BSSL:CERT: 30 82 02 0b 30 82 01 b0 02 14 6c 32 20 e2 36 f2 39 92 86 08 81 23 70 69 bd ae c5 85 87 10 30 0a 06 08 2a 86 48 ce 3d 04 03 02 30 81 84 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 11 30 0f 06 03 55 04 08 0c 08 4d 61 72 79 6c 61 6e 64 31 11 30 0f 06 03 55 04 07 0c 08 43 6f 6c 75 6d 62 69 61 31 15 30 13 06 03 55 04 0a 0c 0c 54 65 6e 65 74 69 63 73 20 4c 4c 43 31 18 30 16 06 03 55 04 03 0c 0f 63 61 2e 74 65 6e 65 74 69 63 73 2e 63 6f 6d 31 1e 30 1c 06 09 2a 86 48 86 f7 0d 01 09 01 16 0f 63 61 40 74 65 6e 65 74 69 63 73 2e 63 6f 6d 30 20 17 0d 32 32 31 32 30 35 32 33 33 32 30 38 5a 18 0f 32 30 35 33 30 31 31 36 32 33 33 32 30 38 5a 30 81 87 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 11 30 0f 06 03 55 04 08 0c 08 4d 61 72 79 6c 61 6e 64 31 11 30 0f 06 03 55 04 07 0c
                                           BSSL:_run_until: Timeout
                                                                   BSSL:_wait_for_handshake: failed
                                                                                                   BSSL:Couldn't connect. Error = 'Unknown error code.'
                                                    response=-1
Server error: connection failed
GET time...
           --------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (0):
epc1=0x3cdf8a3e epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

>>>stack>>>

ctx: bearssl
sp: 3fff3328 end: 3fff3598 offset: 0190
3fff34b8:  000009e6 3fffb488 3fffb188 40236ce0
3fff34c8:  3fff3548 4023a70c 00000000 3fff1c84
3fff34d8:  40276a21 02000000 01030317 f4a1cd0e
3fff34e8:  01c84fdd 20e7d204 9854b9d0 ade0fc63
3fff34f8:  3fff5119 3fff3548 00000003 3fffa7dc
3fff3508:  0000010e 00000020 40276e1b deadbeef