Using SSL certs to publish to API server (MEGH-3886)

[chaudhariatul]

Hi,

I'm trying to create a function which allows to publish to data to an API server which uses SSL, so I tried using WiFiClientSecure.h with rainmaker. This allows to set setCACert, setCertificate and setPrivateKey, which enables connecting to the API server. However the Confirming Node Association step fails while I keep getting data on my API server.

Any suggestions on how to use SSL and allow publishing data to rainmaker as well as another API server?

[shahpiyushv]

@chaudhariatul , if you are talking about using the RainMaker REST APIs from the device, then that would not work using the private key and certificate. REST APIs use the AWS Cognito User Authentication and are meant for clients like phone apps and dashboard. This requires username+password which are used to login and get the refresh/access tokens, to be eventually used for the REST APIs.

[chaudhariatul]

Thanks @shahpiyushv for reply. I'm try to connect and send data to different self hosted server in the cloud. We are using different SSL certificates here. We want to display temperature on rainmaker mobile app as well as send data to another API server.

[shahpiyushv]

@chaudhariatul , so if I understand correctly, this is totally unrelated to RainMaker server itself, right? In that case, if you are seeing failures in "Confirming Node Association", it could be an issue of the firmware crashing after it was provisioned. Can you check in your logs and verify?

[chaudhariatul]

@shahpiyushv I see this in the Serial console:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x43c
load:0x403c9700,len:0xbec
load:0x403cc700,len:0x2a3c
entry 0x403c98d8
Client connection begin.

E (2630) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7680
E (2630) esp-tls: Failed to open new connection
E (2631) TRANSPORT_BASE: Failed to open a new connection
E (2636) HTTP_CLIENT: Connection failed, sock < 0
E (2639) esp_claim: Failed to open connection to https://esp-claiming.rainmaker.espressif.com/claim/initiate
E (2649) esp_claim: Claim Init Request Failed.
E (2652) esp_claim: Claim Init Sequence Failed.
E (2657) esp_rmaker_core: esp_rmaker_self_claim_perform() returned -1. Aborting

I can detect ESP32 as bluetooth device and then set the WiFi crendentials. Board is able to connect to WiFi and starts sending data to my API server. On the rainmaker mobile app it keeps progressing through Configuring Node association and then fails at Confirming Node association.

Error message as show in the box above is right after I provide the WiFi credentials.

[shahpiyushv]

Can you try this once

1. Just reboot the device and see if you still see the esp_claim error messages
2. If you are using any default example, press and hold the boot button for more than 3 seconds and release, so that it goes back into provisioning mode.
3. Try provisioning again.

Do not erase/reflash in between

[chaudhariatul]

Hi @shahpiyushv I tried these steps and got same error.

I used the default example as a reference but here is my code if it helps to reproduce the issues

//Based on Rainmaker example code in Arduino.
#include "RMaker.h"
#include "WiFi.h"
#include "WiFiProv.h"
#include <wifi_provisioning/manager.h>
#include <PubSubClient.h>
#include <ArduinoJson.h>
#include "secrets.h"

#include "Adafruit_SHTC3.h"

#include <WiFiClientSecure.h>
WiFiClientSecure net = WiFiClientSecure();
PubSubClient client(API_ENDPOINT, 8883, net);

unsigned long lastMillis = 0;

static int buffer = 128000;

#define DEFAULT_I2C_PORT &Wire

Adafruit_SHTC3 shtc3 = Adafruit_SHTC3();

// Set Defalt Values
#define DEFAULT_Temperature 0
#define DEFAULT_Humidity 0

//GPIO for push button
#if CONFIG_IDF_TARGET_ESP32C3
static int gpio_0 = 9;
static int gpio_switch = 7;
#else
//GPIO for virtual device
static int gpio_0 = 0;
static int gpio_switch = 16;
#endif

#if CONFIG_IDF_TARGET_ESP32C3
//GPIO for push button
static int gpio_reset = 9;
//GPIO for virtual device
static int gpio_power = 7;
static int gpio_swing = 3;
static int gpio_mode_auto = 4;
static int gpio_mode_cool = 5;
static int gpio_mode_heat = 6;
static int gpio_speed = 10;

#else
//GPIO for push button
static int gpio_reset = 0;
//GPIO for virtual device
static int gpio_power = 16;
static int gpio_swing = 17;
static int gpio_mode_auto = 18;
static int gpio_mode_cool = 19;
static int gpio_mode_heat = 21;
static int gpio_speed = 22;
#endif

#define SECONDARY_I2C_PORT &Wire1

bool switch_state = true;

const char *service_name = "Thing01";
const char *pop = "1234";

bool power_state = true;

bool wifi_connected = 0;

static TemperatureSensor temperature("Temperature");
static TemperatureSensor humidity("Humidity");

void sysProvEvent(arduino_event_t *sys_event)
{
    switch (sys_event->event_id) {
        case ARDUINO_EVENT_PROV_START:
#if CONFIG_IDF_TARGET_ESP32S2
        Serial.printf("\nProvisioning Started with name \"%s\" and PoP \"%s\" on SoftAP\n", service_name, pop);
        printQR(service_name, pop, "softap");
#else
        Serial.printf("\nProvisioning Started with name \"%s\" and PoP \"%s\" on BLE\n", service_name, pop);
        printQR(service_name, pop, "ble");
#endif
        break;
    case ARDUINO_EVENT_WIFI_STA_CONNECTED:
      Serial.printf("\nConnected to Wi-Fi!\n");
      wifi_connected = 1;
      delay(500);
      break;
    case ARDUINO_EVENT_PROV_CRED_RECV: {
        Serial.println("\nReceived Wi-Fi credentials");
        Serial.print("\tSSID : ");
        Serial.println((const char *) sys_event->event_info.prov_cred_recv.ssid);
        Serial.print("\tPassword : ");
        Serial.println((char const *) sys_event->event_info.prov_cred_recv.password);
        break;
      }
    case ARDUINO_EVENT_PROV_INIT:
      wifi_prov_mgr_disable_auto_stop(10000);
      break;
    case ARDUINO_EVENT_PROV_CRED_SUCCESS:
      Serial.println("Stopping Provisioning!!!");
      wifi_prov_mgr_stop_provisioning();
      break;
    }
}

void stream2API(float t, float h)
{
  // Create JSON payload
  StaticJsonDocument<200> doc;
  doc["time"] = millis();
  doc["temperature"] = t;
  doc["humidity"] = h;
  char jsonBuffer[512];
  serializeJson(doc, jsonBuffer);

  // Send data every second (1000 milliseconds)
  if (millis() - lastMillis > 1000) {
    lastMillis = millis();
    if (!client.publish(API_TOPIC, jsonBuffer))
    {
      Serial.println("Failed to publish to API server");
    }
  }
}

/*
Connecting using Certificates issued for each device.
*/
void connect() {
  Serial.println("Client connection begin.");
  Serial.println("");
  Serial.println("connecting...");

  net.setCACert(CERT_CA);
  net.setCertificate(CERT_CRT);
  net.setPrivateKey(CERT_PRIVATE);

  // Connect to topic my name of the thing
  while (!client.connect(THING)) {
    Serial.print(".");
    // Retry in 0.5 seconds
    delay(500);
  }

  // Change mqtt client buffersize to 128KB
  client.setBufferSize(buffer);
  Serial.println("");
  Serial.println("connected!");
}

void setup()
{
    Serial.begin(115200);
    while (!Serial)
      delay(10);

    // Serial.println("SHTC3 test");
    // if (! shtc3.begin(DEFAULT_I2C_PORT)) {
    //   Serial.println("Couldn't find SHTC3");
    //   while (1) delay(1);
    // }
    // Serial.println("Found SHTC3 sensor");

    pinMode(gpio_0, INPUT);

    Node my_node;
    my_node = RMaker.initNode("RoomTemp01");

    my_node.addDevice(temperature);
    my_node.addDevice(humidity);

    RMaker.enableOTA(OTA_USING_PARAMS);

    RMaker.enableTZService();

    RMaker.enableSchedule();

    RMaker.start();

    WiFi.onEvent(sysProvEvent);

#if CONFIG_IDF_TARGET_ESP32S2
    WiFiProv.beginProvision(WIFI_PROV_SCHEME_SOFTAP, WIFI_PROV_SCHEME_HANDLER_NONE, WIFI_PROV_SECURITY_1, pop, service_name);
#else
    WiFiProv.beginProvision(WIFI_PROV_SCHEME_BLE, WIFI_PROV_SCHEME_HANDLER_FREE_BTDM, WIFI_PROV_SECURITY_1, pop, service_name);
#endif

}

void loop()
{

    // sensors_event_t hum, temp;
    // shtc3.getEvent(&hum, &temp);
    // Serial.print("Temperature: "); Serial.print(temp.temperature); Serial.println(" degrees C");
    // Serial.print("Humidity: "); Serial.print(hum.relative_humidity); Serial.println("% rH");
    // float rTemp = temp.temperature;
    // float rHum = hum.relative_humidity;

    float rTemp = random(7000, 7600) / 100.00; 
    float rHum = random(1000, 3000) / 100.00;

    if(digitalRead(gpio_0) == LOW) { 

        delay(100);
        int startTime = millis();
        while(digitalRead(gpio_0) == LOW) delay(50);
        int endTime = millis();

        if ((endTime - startTime) > 10000) {
          // If key pressed for more than 10secs, reset all
          Serial.printf("Reset to factory.\n");
          wifi_connected = 0;
          RMakerFactoryReset(2);
        } else if ((endTime - startTime) > 3000) {
          Serial.printf("Reset Wi-Fi.\n");
          wifi_connected = 0;
          // If key pressed for more than 3secs, but less than 10, reset Wi-Fi
          RMakerWiFiReset(2);
        }
    }

    if (wifi_connected) {
      client.loop(); 

      temperature.updateAndReportParam("Temperature", rTemp);
      humidity.updateAndReportParam("Temperature", rHum);

      // Reconnect if disconnected
      if (client.connected()) {
        stream2API(rTemp, rHum);
      } else {
        connect();
      }
    }

    delay(10000);
}

---

Serial console output

Reset Wi-Fi.
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40376d80
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x43c
load:0x403c9700,len:0xbec
load:0x403cc700,len:0x2a3c
entry 0x403c98d8

Provisioning Started with name "Thing01" and PoP "1234" on BLE
Scan this QR code from the ESP RainMaker phone app.

If QR code is not visible, copy paste the below URL in a browser.
https://rainmaker.espressif.com/qrcode.html?data={"ver":"v1","name":"Thing01","pop":"1234","transport":"ble"}
E (27393) BT_GATT: gatts_write_attr_perm_check - GATT_INSUF_ENCRYPTION

Received Wi-Fi credentials
    SSID : WIFISSID
    Password : WIFIPASSWORD

Connected to Wi-Fi!
Stopping Provisioning!!!
E (48959) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7680
E (48959) esp-tls: Failed to open new connection
E (48959) TRANSPORT_BASE: Failed to open a new connection
E (48964) MQTT_CLIENT: Error transport connect
E (48967) esp_mqtt_glue: MQTT_EVENT_ERROR
Client connection begin.

connecting...

connected!
E (65658) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7680
E (65658) esp-tls: Failed to open new connection
E (65658) TRANSPORT_BASE: Failed to open a new connection
E (65664) MQTT_CLIENT: Error transport connect
E (65666) esp_mqtt_glue: MQTT_EVENT_ERROR
E (81632) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7680
E (81633) esp-tls: Failed to open new connection
E (81633) TRANSPORT_BASE: Failed to open a new connection
E (81638) MQTT_CLIENT: Error transport connect
E (81641) esp_mqtt_glue: MQTT_EVENT_ERROR
E (97300) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7680
E (97300) esp-tls: Failed to open new connection
E (97300) TRANSPORT_BASE: Failed to open a new connection
E (97306) MQTT_CLIENT: Error transport connect
E (97308) esp_mqtt_glue: MQTT_EVENT_ERROR
E (114503) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7680
E (114503) esp-tls: Failed to open new connection
E (114504) TRANSPORT_BASE: Failed to open a new connection
E (114509) MQTT_CLIENT: Error transport connect
E (114512) esp_mqtt_glue: MQTT_EVENT_ERROR

[shahpiyushv]

@chaudhariatul , do the default examples work well for you? If yes, one option is to wait for RMAKER_MQTT_EVENT_CONNECTED under the RMAKER_COMMON_EVENT and only then start your connection attempts. You can see here for a sample of how to register event handlers.

[chaudhariatul]

@shahpiyushv Kindly share documentation link of all possible RMAKER events. If there are examples it will be very helpful. I wasn't able to find much here : https://rainmaker.espressif.com/docs/get-started.html

[shahpiyushv]

@chaudhariatul , various RainMaker components use the ESP IDF's event framework and so the events are spread out into respective header files, the list for which can be found in the API docs.

RMAKER_EVENT

RMAKER_COMMON_EVENT

RMAKER_OTA_EVENT

There is no specific Arduino API for this, so you can refer the esp-idf based switch example. Handlers can be registered as shown here and defined as shown here.

[chaudhariatul]

@shahpiyushv are there any arduino examples on how to check RMAKER_MQTT_EVENT_CONNECTED?

[chaudhariatul]

My guess is there are 2 issues with my code:

1. Using in the connect function is causing SSL errors to rainmaker cloud connection

   net.setCACert(CERT_CA);
   net.setCertificate(CERT_CRT);
   net.setPrivateKey(CERT_PRIVATE);

2. Rainmaker arduino library doesn't have RMAKER_COMMON_EVENT? I tried this example with arduino but made no progress. esp_event_handler_register(RMAKER_COMMON_EVENT, ESP_EVENT_ANY_ID, &event_handler, NULL)

[shahpiyushv]

@chaudhariatul , I think the reason for the issue you are seeing is because of insufficient memory and hence, the suggestion to let provisioning finish and mqtt connect.

With regards to the event handler, is your event_handler function not getting called at all? Can you check if esp_event_handler_register is returning ESP_OK? Can you share your event handler function?