About provisioning existing devices

[alkonosst]

Hello there, thank you for this good and comprehensive library!

First some context:

• Using ESP32 with PlatformIO ESP32 v6.4.0.
• Using this library v0.12.0.
• Using TinyGSM with SIM7600G-H.
• Using SSLClientESP32 to connect to Thingsboard via MQTTS.
• I've adapted the provision example to my firmware.
• In the future, I want a generic firmware (.bin) which I will upload to a lot of devices (e.g. >100). The devices with this firmware would provision themselves to Thingsboard, using unique MAC address as Device Name.

Provisioning device to obtain Access Token

This works well. The device provisions itself using its MAC as Device Name. But I have a issue...

Provisioning request and response (credentials replaced with ***):

[ 60754][I][Network.cpp:311] taskFunction(): Connecting to Thingsboard (provision)...
[ 60794][I][Network.cpp:319] taskFunction(): Connected! Sending provisioning request...
[ 60794][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (26 bytes)...
[TB] Sending data to server over topic (/provision/request) with data ({"deviceName":"25ef49c0","provisionDeviceKey":"***","provisionDeviceSecret":"***"})
[ 60841][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (139 bytes)...
[ 60902][I][Network.cpp:336] taskFunction(): Provision request sent
[TB] Received data from server over topic (/provision/response)
[ 62982][I][Network.cpp:624] processProvisionResponse(): Received device provision response
[ 62985][D][Network.cpp:625] processProvisionResponse(): {"credentialsValue":"***","credentialsType":"ACCESS_TOKEN","status":"SUCCESS"}
[ 62999][D][Network.cpp:641] processProvisionResponse(): Client ID:  - Username: *** - Password:
[ 63010][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (2 bytes)...
[ 63047][V][ssl_client.cpp:349] stop_ssl_socket(): Cleaning SSL connection.

Connection to Thingsboard with received credentials:

[ 64067][I][Network.cpp:349] taskFunction(): Connecting to Thingsboard...
[ 64075][D][SSLClientESP32.cpp:114] connect(): Connecting to mqtt.thingsboard.cloud:8883
[ 64083][V][ssl_client.cpp:156] start_ssl_client(): Free internal heap before TLS 258220
[ 64092][I][ssl_client.cpp:158] start_ssl_client(): Connecting to mqtt.thingsboard.cloud:8883
[64152] ### Unhandled: +CIPCLOSE: 0,0
[ 64422][V][ssl_client.cpp:172] start_ssl_client(): Seeding the random number generator
[ 64423][V][ssl_client.cpp:181] start_ssl_client(): Setting up the SSL/TLS structure...
[ 64428][V][ssl_client.cpp:216] start_ssl_client(): Attaching root CA cert bundle
[ 64437][V][ssl_client.cpp:280] start_ssl_client(): Setting hostname for TLS session...
[ 64445][V][ssl_client.cpp:293] start_ssl_client(): Setting up IO callbacks...
[ 64453][V][ssl_client.cpp:296] start_ssl_client(): Performing the SSL/TLS handshake...
[ 66745][D][esp_crt_bundle.c:108] esp_crt_verify_callback(): 138 certificates in bundle
[ 66763][I][esp_crt_bundle.c:142] esp_crt_verify_callback(): Certificate validated
[ 69573][V][ssl_client.cpp:317] start_ssl_client(): Verifying peer X.509 certificate...
[ 69573][V][ssl_client.cpp:325] start_ssl_client(): Certificate verified.
[ 69578][V][ssl_client.cpp:340] start_ssl_client(): Free internal heap after TLS 217080
[ 69587][I][SSLClientESP32.cpp:126] connect(): SSL connection established
[ 69594][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (38 bytes)...
[ 69881][I][Network.cpp:361] taskFunction(): Connected!

But if the device loses the credentials (saved in NVS) and start with a "fresh" uploaded firmware (NVS formatted), the cloud respond to the provisioning request with the message "Failed to provision device!":

[  1384][I][Network.cpp:311] taskFunction(): Connecting to Thingsboard (provision)...
[  1392][D][SSLClientESP32.cpp:114] connect(): Connecting to mqtt.thingsboard.cloud:8883
[  1401][V][ssl_client.cpp:156] start_ssl_client(): Free internal heap before TLS 259012
[  1410][I][ssl_client.cpp:158] start_ssl_client(): Connecting to mqtt.thingsboard.cloud:8883
[  1714][V][ssl_client.cpp:172] start_ssl_client(): Seeding the random number generator
[  1716][V][ssl_client.cpp:181] start_ssl_client(): Setting up the SSL/TLS structure...
[  1722][V][ssl_client.cpp:216] start_ssl_client(): Attaching root CA cert bundle
[  1729][V][ssl_client.cpp:280] start_ssl_client(): Setting hostname for TLS session...
[  1738][V][ssl_client.cpp:293] start_ssl_client(): Setting up IO callbacks...
[  1745][V][ssl_client.cpp:296] start_ssl_client(): Performing the SSL/TLS handshake...
[  4019][D][esp_crt_bundle.c:108] esp_crt_verify_callback(): 138 certificates in bundle
[  4036][I][esp_crt_bundle.c:142] esp_crt_verify_callback(): Certificate validated
[  6885][V][ssl_client.cpp:317] start_ssl_client(): Verifying peer X.509 certificate...
[  6885][V][ssl_client.cpp:325] start_ssl_client(): Certificate verified.
[  6890][V][ssl_client.cpp:340] start_ssl_client(): Free internal heap after TLS 217196
[  6898][I][SSLClientESP32.cpp:126] connect(): SSL connection established
[  6906][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (30 bytes)...
[  7199][I][Network.cpp:319] taskFunction(): Connected! Sending provisioning request...
[  7199][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (26 bytes)...
[TB] Sending data to server over topic (/provision/request) with data ({"deviceName":"25ef49c0","provisionDeviceKey":"***","provisionDeviceSecret":"***"})
[  7235][D][ssl_client.cpp:394] send_ssl_data(): Writing SSL (139 bytes)...
[  7276][I][Network.cpp:336] taskFunction(): Provision request sent
[TB] Received data from server over topic (/provision/response)
[  9375][I][Network.cpp:624] processProvisionResponse(): Received device provision response
[  9378][D][Network.cpp:625] processProvisionResponse(): {"errorMsg":"Failed to provision device!","status":"FAILURE"}
[  9389][E][Network.cpp:628] processProvisionResponse(): Provision response error: Failed to provision 
device!

This event forces me to delete the 25ef49c0 device in the cloud, because it already exists. Is there some way the device 25ef49c0 could obtain its already created Access Token?

Auto-provisioning with X.509 certificate

This also works well. I generate a Device Key with a unique Common Name using OpenSSL, which is used by the cloud to create a new device with a certain quantity of words in the Common Name (using regular expression).

But, of course, this forces me to create a unique key file and upload to every device separately (I can't imagine the time it takes to do this to >100 devices).

---

So, is there a way a device could connect to Thingsboard for the first time, and the cloud creates this device?

Hope I've clear enough! Thanks again for the library and your help :)

[MathewHDYT]

But if the device loses the credentials (saved in NVS) and start with a "fresh" uploaded firmware (NVS formatted), the cloud respond to the provisioning request with the message "Failed to provision device!"

The aforementioned behaviour is as expected if you plan to provision a device with the same id as an already existing device that will not work.

If you plan for the device to restore its own access token and connection somehow if it is lost from NVS that will not be possible. Because the access token is the only way to ensure the device is the one initially authenticated and not any arbitrary device posing as such.

It is possible to receive the Access Token but not using the the MQTT API, you would have to use the Rest API for that to fetch the credentials of a device with a given id. With the getDeviceCredentialsByDeviceIdUsingGET.

For your use case I would look into ensuring that the NVS is not wiped instead. You can upload the firmware (.bin) file without overwriting the file system or NVS portion. I'm not sure which toolchain you use, but the PlatformIO upload does not overwrite the file system or NVS portion per default.

Additionally if you update using Over the Air updates it will also not wipe the file system or NVS portion of the partitions file.

[alkonosst]

Thank you for your answer.

The aforementioned behaviour is as expected if you plan to provision a device with the same id as an already existing device that will not work. If you plan for the device to restore its own access token and connection somehow if it is lost from NVS that will not be possible. Because the access token is the only way to ensure the device is the one initially authenticated and not any arbitrary device posing as such.

I imagined that would be the behavior of the cloud. Thanks for ensuring that.

It is possible to receive the Access Token but not using the the MQTT API, you would have to use the Rest API for that to fetch the credentials of a device with a given id. With the getDeviceCredentialsByDeviceIdUsingGET.

I also looked for it to see if it could be an alternative, but yes, the API asks for that ID generated by the cloud, so it is not viable either.

For your use case I would look into ensuring that the NVS is not wiped instead. You can upload the firmware (.bin) file without overwriting the file system or NVS portion. I'm not sure which toolchain you use, but the PlatformIO upload does not overwrite the file system or NVS portion per default.

Additionally if you update using Over the Air updates it will also not wipe the file system or NVS portion of the partitions file.

You are right, in our firmware the NVS is not wiped on updates (using OTA with thingsboard). I forgot to mention this issue happens in a testing environment, where we use a CLI command (nvs -f) to clean all settings from NVS. It would be nice if I could have the same device on the cloud for telemetry purposes, and not having to delete this test device on the cloud.

I think I can overpass this issue with some modifications to keep Access Token not cleared after this wipe command.

Thank you, closing this issue.