Asanga-Viraj commented 2 years ago

@slaff I am working on connecting ESP into MQTT AWS IoT Core. I couldn't' connect it because it shows a debug message "please start sntp first !". I went deep and I saw "sntp_init()" included in "configTime" is not called any where. Then, I called below lines on gotIP function.

sntp_stop(); sntp_setservername(0, (char*)"pool.ntp.org"); sntp_setservername(1, (char*)"0.au.pool.ntp.org"); sntp_set_timezone(10); // GTM+10 sntp_init();

Now, message "please start sntp first !" is gone and got "SSL Validator: list empty, allow connection". Can you help me with this?

Thanks.

slaff commented 2 years ago

@Asanga-Viraj Isn't this sample working for you? In it you can see how to attach SSL certificates:

https://github.com/SmingHub/Sming/blob/develop/samples/Basic_AWS/app/application.cpp#L21-L24

In addition if you want to syncrhonise the local clock you can use the NTP sample as a starting point.

Asanga-Viraj commented 2 years ago

@slaff I couldn't manage to work Basic_AWS example with the current MQTT AWS IoT Core. To connect to AWS, we need to assign a device certificate, private key, and Root CA1 certificate. But there is no entry to assign the Root CA1 certificate. Refer this Arduino example. it is working with AWS.

I forgot to tell you that the message "please start sntp first !" only appears on ESP8266. Not in ESP32. I will test with NTP sample. Thank you for that.

slaff commented 2 years ago

To connect to AWS, we need to assign a device certificate, private key, and Root CA1 certificate. But there is no entry to assign the Root CA1 certificate.

Short Story

The Amazon Root CA1 certificate is not used in the sample because we skip SERVER CERTIFICATE verification. We should probably update it to include full SERVER CERTIFICATE verification. But it does not mean that the connection is not working or that the client application is not able to send or receive MQTT data.

Long Story

Ok, even with over-simplifications, this is probably going to be a long answer but once you read this you should know better

How TLS/SSL communication works
How to debug network communication in Sming

How TLS/SSL communication works

Here we are focusing only on the communication from our device, the client, to the remote AWS cloud, the server. When a client connects to a remote server using TLS it does many things. Some of the important ones are:

A) it establishes encrypted communication
B) it trusts the remote side.

Let's start with A). What happens here is the following:

A1) client first tries to make a TCP connection to the provided hostname and port.
A2) IF this succeeds the client communicates with the remote server using ASYMMETRICAL encryption. Basically the server sends to the client a SERVER CERTIFICATE which describes the server plus contains a public key. Further the client and the server negotiate a TOKEN to be used to continue the communication. This process is slow. The client encrypts its data using the public key from the server and the server decrypts the data using its private key. Once such a TOKEN is known to both sides the communication switches to SYMMETRICAL encryption. Now this is much faster.

Let's focus on the B). In order to trust the remote side the client could/should verify the SERVER CERTIFICATE. The fact that the communication is encrypted does not prevent the client from talking to a malicious server pretending to be a valid AWS server. The SERVER CERTIFICATE verification does. A full verification is expensive, especially for embedded devices like ESP8266. A full verification requires the expiration date of the SERVER CERTIFICATE to be checked. For this your device will need a VALID TIME. Getting the current time will require another network call to an NTP server which additionally increases the time and resource needed. If your device is not time critical and not running on a battery then fine. If you want to go this route then make sure to replace the code below:

mqtt.setSslInitHandler([](Ssl::Session& session) {
        session.options.verifyLater = true;
        session.keyCert.assign(privateKeyData, certificateData);
    });

with

mqtt.setSslInitHandler([](Ssl::Session& session) {
        session.options.verifyLater = false; // Do expensive certificate validation!
        session.keyCert.assign(privateKeyData, certificateData);
    });

Now in order to fully verify a certificate a lot of steps need to be done. Not only the expiration date, hostname in the certificate matching the one one used BUT also its signature. My bold guess is that the SERVER CERTIFICATE is signed using the Amazon Root CA1 certificate. This way the client can verify with high certainty if the SERVER CERTIFICATE is the one that is expected. And thus trust the remote server to be the one that it pretends to be.

A slightly less secure but much faster way is to verify the provided SERVER CERTIFICATE against its SHA1 or SHA256 finger-prints. Basically the content of the certificate is used to feed a SHA1/SHA256 hash function and compared against expected finger-print. Of course even small changes in the SERVER CERTIFICATE will cause our fingerprinting check to fail. Slightly less error-prone and a bit slower is to extract the public key from the SERVER CERTIFICATE and compare its SHA1/SHA256 hash against expected ones. Our Basic_SSL example explains how to set such fingerprint checks: https://github.com/SmingHub/Sming/blob/da1ff089d29dfa00e6183722656221ae67ca4e9d/samples/Basic_Ssl/app/application.cpp#L78-L100

But there is no entry to assign the Root CA1 certificate.

In our Basic_AWS example we DON'T do full SERVER CERTIFICATE verification this is why you don't see it used.

got "SSL Validator: list empty, allow connection".

Basic_AWS is not using any server certificate validators and this is the reason why you see this message. You can either set finger-print validators or even set a callback validator where the received SERVER CERTIFICATE is provided to the validator and in your application you can do all needed manual validations.

In the example the client provides its CLIENT CERTIFICATE and private key.

mqtt.setSslInitHandler([](Ssl::Session& session) {
        session.options.verifyLater = true; 
        session.keyCert.assign(privateKeyData, certificateData);
    });

The CLIENT CERTIFICATE is sent to AWS and used by the remote server to check if it was issued by them. This way our client is ALLOWED to communicate further with the AWS servers. The private key is used from our client to encrypt the communication. AWS generates the private key, then the CLIENT CERTIFICATE and allows you to download the private key. (We assume that) AWS itself does not keep your private key. The CLIENT CERTIFICATE helps the remote server to verify if our client is valid and also to communicate in a secure manner.

How to debug network communication in Sming

If you are using Windows or Linux the fastest way to debug the network communication is to compile the example using the Host architecture. For example you go to the root folder of the Basic_AWS sample and type:

make SMING_ARCH=Host DEBUG_VERBOSE_LEVEL=3 ENABLE_GDB=1

DEBUG_VERBOSE_LEVEL=3 will make sure that the developers will see all debug messages from the network stack and the MQTT client. ENABLE_GDB=1 will allow you to use the debugger and go step by step.

In addition to this since the sample application will be running on your local machine you should be able to sniff the network communication using Wireshark.

I couldn't' connect it because it

As I wrote before, connecting will require A1) and A2).

A successful A1) looks like this:

18848 TCP 56632400 DNS record found: a2e224ln0t7m96-ats.iot.eu-west-1.amazonaws.com = 52.18.195.218
18932 TCP connect result: 0 << ----- IMPORTANT
...
203233 TCP 56632400 connected: useSSL: 1, Error: 0

If error is 0 you are connected via TCP.

A successful A2) looks like this:

445312 SSL Validator: list empty, allow connection
445336 TCP 56632400 connected << ----- IMPORTANT
445342 TCP 56632400 onReadyToSendData: 0
445373 TCP 56632400 Written: 23, Available: 2780, isFinished: 0, PushCount: 1
445397 TcpClient stream finished
445407 [MQTT] Sending message type 8 << ----- IMPORTANT

There is communication with the remote server, data is sent and all looks good.

This is all you need to know. Now try to compile your application with DEBUG_VERBOSE_LEVEL=3 and paste the output here.

Asanga-Viraj commented 2 years ago

@slaff

Thank you for the full detail information. Extremely sorry for my lack of knowledge about TSL/SSL communication. Now I have the idea about how root certificate authority (RootCA1) file works. So, nothing wrong about setSslInitHandler(). I tried to work with Basic_AWS example on ESP32. Then I got this with DEBUG_VERBOSE_LEVEL=3

1117438 MQTT start connection 1118530 TCP 3ffb4354 +connection 1121764 TCP 3ffb4354 connect to "aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com:8883" 1129938 subscription 'esp32/sub' registered 1132926 TCP 3ffb4354 connection closing 1136229 MQTT closed previous connection 1139758 MQTT start connection 1142475 MQTT replacing connect message 1146067 TCP 3ffb4354 +connection 1148903 TCP 3ffb4354 connect to "aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com:8883" 1156530 subscription 'esp32/sub' registered publish message 1162005 [MQTT] Sending message type 1 1165369 MemoryDataStream::realloc 0 -> 24 1168978 Sending stream. Bytes to send: 24 1172779 subscription '$aws/things/ESP32_Test/shadow/get' registered 1190234 TCP 3ffb4354 DNS record found: aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com = 52.52.86.23 1197254 TCP connect result: 0 1198892 TCP 3ffb4354 DNS record found: aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com = 52.52.86.23 1207651 TCP connect result: -10 1389417 TCP 3ffb4354 connected: useSSL: 1, Error: 0 1707433 TCP 3ffb4354 sent: 127 3845648 TCP 3ffb4354 sent: 1503 3853587 SSL Validator: list empty, allow connection 3853777 TCP 3ffb4354 connected 3853888 TCP 3ffb4354 onReadyToSendData: 0 3854748 TCP 3ffb4354 Written: 24, Available: 5744, isFinished: 0, PushCount: 1 3860644 TcpClient stream finished 3863724 [MQTT] Sending message type 8 3867182 MemoryDataStream::realloc 0 -> 16 3870820 Sending stream. Bytes to send: 16 3875077 TCP 3ffb4354 Written: 16, Available: 5659, isFinished: 0, PushCount: 1 3881456 TcpClient stream finished 3884555 [MQTT] Sending message type 8 3887884 MemoryDataStream::realloc 0 -> 16 3891630 Sending stream. Bytes to send: 16 3895838 TCP 3ffb4354 Written: 16, Available: 5574, isFinished: 0, PushCount: 1 3902281 TcpClient stream finished 3905350 [MQTT] Sending message type 3 3908732 MemoryDataStream::realloc 0 -> 31 3912464 Sending stream. Bytes to send: 31 3916671 TCP 3ffb4354 Written: 31, Available: 5489, isFinished: 0, PushCount: 1 3923114 TcpClient stream finished 3926192 [MQTT] Sending message type 8 3929550 MemoryDataStream::realloc 0 -> 41 3933297 Sending stream. Bytes to send: 41 3937549 TCP 3ffb4354 Written: 41, Available: 5404, isFinished: 0, PushCount: 1 3943948 TcpClient stream finished 4159341 TCP 3ffb4354 sent: 441 4159537 TCP 3ffb4354 onReadyToSendData: 2 4361913 TCP 3ffb4354 sent: 69 4362141 TCP 3ffb4354 onReadyToSendData: 2 4362241 TCP 3ffb4354 receive: pbuf is NULL 4362299 TCP 3ffb4354 received: (null) 4365310 TCP 3ffb4354 connection closing 4442872 -TCP connection

At the end, TCP connection is disconnected. The I commented mqtt subscribes and publishes.

After, I tried this on ESP32 and got this.

1117588 MQTT start connection 1118532 TCP 3ffb4354 +connection 1121743 TCP 3ffb4354 connect to "aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com:8883" 1161594 TCP 3ffb4354 DNS record found: aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com = 52.52.86.23 1162624 TCP connect result: 0 1546500 TCP 3ffb4354 connected: useSSL: 1, Error: 0 1848115 TCP 3ffb4354 sent: 127 3790660 TCP 3ffb4354 sent: 1503 3796186 SSL Validator: list empty, allow connection 3796325 TCP 3ffb4354 connected 3796419 [MQTT] Sending message type 1 3796618 MemoryDataStream::realloc 0 -> 24 3799670 Sending stream. Bytes to send: 24 3803401 TCP 3ffb4354 onReadyToSendData: 0 3807910 TCP 3ffb4354 Written: 24, Available: 5744, isFinished: 0, PushCount: 1 3814065 TcpClient stream finished 4210096 TCP 3ffb4354 sent: 85 4210284 TCP 3ffb4354 onReadyToSendData: 2 4212401 TCP 3ffb4354 timeout updating: 70 -> 65535 4212575 TCP 3ffb4354 received: 4 bytes 4213505 TCP 3ffb4354 onReadyToSendData: 1 25832954 [MQTT] Sending message type 12 25833287 MemoryDataStream::realloc 0 -> 2 25833371 Sending stream. Bytes to send: 2 25837099 TCP 3ffb4354 Written: 2, Available: 5744, isFinished: 0, PushCount: 1 25840026 TcpClient stream finished 26115031 TCP 3ffb4354 sent: 69 26115213 TCP 3ffb4354 onReadyToSendData: 2 26116915 TCP 3ffb4354 received: 2 bytes 26117067 TCP 3ffb4354 onReadyToSendData: 1 47842931 [MQTT] Sending message type 12 47843228 MemoryDataStream::realloc 0 -> 2 47843308 Sending stream. Bytes to send: 2 47845274 TCP 3ffb4354 Written: 2, Available: 5744, isFinished: 0, PushCount: 1 47849934 TcpClient stream finished 48031624 TCP 3ffb4354 sent: 69 48031717 TCP 3ffb4354 onReadyToSendData: 2 48032492 TCP 3ffb4354 received: 2 bytes 48032640 TCP 3ffb4354 onReadyToSendData: 1

Connection is stablished and didn't disconnect. We can see ping data packet after each 10 seconds. Then, I put a timer to call publish message after 10 seconds. ESP published messages successfully. Debug output is below.

1615658 MQTT start connection 1616709 TCP 3ffb4354 +connection 1619931 TCP 3ffb4354 connect to "aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com:8883" 1658472 TCP 3ffb4354 DNS record found: aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com = 52.53.78.134 1659583 TCP connect result: 0 1877007 TCP 3ffb4354 connected: useSSL: 1, Error: 0 2078796 TCP 3ffb4354 sent: 127 4123302 TCP 3ffb4354 sent: 1503 4128007 SSL Validator: list empty, allow connection 4128150 TCP 3ffb4354 connected 4128244 [MQTT] Sending message type 1 4128439 MemoryDataStream::realloc 0 -> 24 4131498 Sending stream. Bytes to send: 24 4135215 TCP 3ffb4354 onReadyToSendData: 0 4139757 TCP 3ffb4354 Written: 24, Available: 5744, isFinished: 0, PushCount: 1 4145883 TcpClient stream finished 4542006 TCP 3ffb4354 sent: 85 4542194 TCP 3ffb4354 onReadyToSendData: 2 4544220 TCP 3ffb4354 timeout updating: 70 -> 65535 4544390 TCP 3ffb4354 received: 4 bytes 4545412 TCP 3ffb4354 onReadyToSendData: 1 publish message 11628585 [MQTT] Sending message type 3 11628749 MemoryDataStream::realloc 0 -> 31 11628826 Sending stream. Bytes to send: 31 11631124 TCP 3ffb4354 Written: 31, Available: 5744, isFinished: 0, PushCount: 1 11636786 TcpClient stream finished 11909173 TCP 3ffb4354 sent: 85 11909286 TCP 3ffb4354 onReadyToSendData: 2

It keeps publishing data into AWS without disconnecting. After 10 seconds I tried this mqtt.subscribe("esp32/sub") without publishing messages. It disconnected the AWS connection. Don't know the reason. The debug output is below.

1615669 MQTT start connection 1616786 TCP 3ffb4354 +connection 1619988 TCP 3ffb4354 connect to "aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com:8883" 1668724 TCP 3ffb4354 DNS record found: aXXXXXXXXa-ats.iot.us-west-1.amazonaws.com = 52.9.168.113 1669739 TCP connect result: 0 1860862 TCP 3ffb4354 connected: useSSL: 1, Error: 0 2068531 TCP 3ffb4354 sent: 127 4011077 TCP 3ffb4354 sent: 1503 4014527 SSL Validator: list empty, allow connection 4014670 TCP 3ffb4354 connected 4014759 [MQTT] Sending message type 1 4014954 MemoryDataStream::realloc 0 -> 24 4018015 Sending stream. Bytes to send: 24 4021733 TCP 3ffb4354 onReadyToSendData: 0 4026250 TCP 3ffb4354 Written: 24, Available: 5744, isFinished: 0, PushCount: 1 4032398 TcpClient stream finished 4317235 TCP 3ffb4354 sent: 85 4317415 TCP 3ffb4354 onReadyToSendData: 2 4318998 TCP 3ffb4354 timeout updating: 70 -> 65535 4319176 TCP 3ffb4354 received: 4 bytes 4320636 TCP 3ffb4354 onReadyToSendData: 1 subscribing for messages 11628498 subscription 'esp32/sub' registered 12312929 [MQTT] Sending message type 8 12313302 MemoryDataStream::realloc 0 -> 16 12313387 Sending stream. Bytes to send: 16 12316889 TCP 3ffb4354 Written: 16, Available: 5744, isFinished: 0, PushCount: 1 12320190 TcpClient stream finished 12510655 TCP 3ffb4354 sent: 85 12510779 TCP 3ffb4354 onReadyToSendData: 2 12716334 TCP 3ffb4354 sent: 69 12716570 TCP 3ffb4354 onReadyToSendData: 2 12716671 TCP 3ffb4354 receive: pbuf is NULL 12716730 TCP 3ffb4354 received: (null) 12720080 TCP 3ffb4354 connection closing 12822842 -TCP connection

Things I saw

while stablishing connection, it is better not to subscribe or publish. Needs some time.
mqtt EvenHandlers are not calling during two way SSL mode. Only mqtt.setCompleteDelegate() calls when disconnected.
mqtt.subscribe() disconnects TCP AWS connection.
Tested one way SSL mode via ThingsBoard. It is working perfectly.

These are the things I found. Thanks.

SmingHub / Sming

Outdated Basic_AWS example #2511

Short Story

Long Story

How TLS/SSL communication works

How to debug network communication in Sming