Cannot connect downstream device to IoT Edge (In Python it's working)

mleister97 commented 3 years ago

Configured a downstream device

I configured my ESP8266 to work with Azure IoT.
It is possible to directly connect to the IoT Hub with the SharedAccessKey in the Azure Panel (symmetric).
I am using the Azure example which can be found here: https://github.com/Azure/azure-iot-arduino/tree/master/examples/esp8266/iothub_ll_telemetry_sample
I appended ;GatewayHostName=<ip_address_of_raspberry> to the connection string (String can be found below).
I added the root certificate (azure-iot-test-only.root.ca.cert.pem) to my esp8266 and configured the library to work with this certificate
But: I cannot connect to my configured IoT Edge device.

Configured the edge device

I configured a raspberry pi according to the documentation. https://docs.microsoft.com/en-us/azure/iot-edge/how-to-create-transparent-gateway?view=iotedge-2018-06
I created demo certificates (also according to the documentation) and added them to the file /etc/iotedge/config.yaml
- Root CA certificate
- Device CA certificate
- Device CA private key
I can successfully verify the certificate on my windows machine with the following command

ESP8266 Logs:

Creating IoTHub Device handle...
Sending message 1 to IoTHub...
dowork TLSIO_STATE_OPENING_WAITING_SOCKET
dowork TLSIO_STATE_OPENING_WAITING_SSL
Error opening socket 0
Calling error callback
The device client has been disconnected
Error: failure opening connection to endpoint

How I added the certificate:

static const char edgeCert [] =
"-----BEGIN CERTIFICATE-----\r\n"
...
"-----END CERTIFICATE-----";
IoTHubDeviceClient_LL_SetOption(device_ll_handle, OPTION_TRUSTED_CERT, edgeCert);

Verification of the certificate:

openssl s_client -connect 192.168.4.1:8883 -CAfile "C:\Users\micha\Desktop\azure-iot-test-only.root.ca.cert.pem" -showcerts

Results in: Verify return code: 0 (ok)

Connection string on the ESP8266:

HostName=leister-iot-hub.azure-devices.net;DeviceId=wemosd1mini;SharedAccessKey=xxxxxxxxxxxxxxxxxx=;GatewayHostName=192.168.4.1"

mleister97 commented 3 years ago

Maybe I need to get rid of the IP address and really add a hostname? That would I need to create a custom DNS server just to be able to test this?

sandervandevelde commented 3 years ago

Is the ESP able to ping the Edge? Do these two devices see each other over the local network?

ericwolz commented 3 years ago

Yes, if you connect by IP address, then the TLS host name validation is going to fail. What is the CN value of the TLS cert on the edge gateway endpoint?

mleister97 commented 3 years ago

@sandervandevelde Both devices can see each other (are pingable from each other). But: Only using the IP of the gateway, not the hostname.

@ericwol-msft Thanks for your time. I think I can resolve the issue by changing the IP address of the gateway to a hostname. For this purpose I think I need to setup a local DNS including the IP address of the edge/gateway and bind it to the hostname defined in the configuration file (/etc/iotedge/config.yaml).

Is it really necessary to set up a local dns for testing purposes? Or is it possible to connect with the IP instead of the hostname? The problem is as already mentioned by @ericwol-msft most likely the tls hostname validation.

ericwolz commented 3 years ago

Yes, you can create a cert with the IP address as the CN subject name. I assume you are using mbed TLS on the device? Just verify that they will validate certs with IP address, but AFAIK it should.

mleister97 commented 3 years ago

I created a custom DNS and made sure that the hostname is pingable. So far so good. Also tried a Python example where everything worked fine - I can send messages to the IoT Hub via the Edge on my computer.

Still, I cannot send messages with my ESP8266.

I am using the same certificate and the connection string. Can anybody tell me why my example isn't working so far?

ESP8266 Code example

```c++ #include #include #include #include "sample_init.h" #include "iot_configs.h" #include "Esp.h" #include "AzureIoTProtocol_MQTT.h" #include "iothubtransportmqtt.h" static const char ssid[] = IOT_CONFIG_WIFI_SSID; static const char pass[] = IOT_CONFIG_WIFI_PASSWORD; static const char* connectionString = DEVICE_CONNECTION_STRING; // HostName=leister-iot-hub.azure-devices.net;DeviceId=wemosd1mini;SharedAccessKey=XXXX;GatewayHostName=iot-edge.leister IOTHUB_MESSAGE_HANDLE message_handle; IOTHUB_DEVICE_CLIENT_LL_HANDLE device_ll_handle; int receiveContext = 0; size_t messages_sent = 0; static size_t g_message_count_send_confirmations = 0; // same certificate as in python example (content of file azure-iot-test-only.root.ca.cert.pem) static const char edgeCert [] = "-----BEGIN CERTIFICATE-----\r\n" "MIIFjTCCA3WgAwIBAgIUJuEzz2uSFIDSuYsWAS6N6Hqfd0cwDQYJKoZIhvcNAQEL\r\n" // ... "-----END CERTIFICATE-----\r\n"; const char* telemetry_msg = "{\"temperature\":11.11,\"humidity\":12.12,\"scale\":\"13.13\"}"; static void send_confirm_callback(IOTHUB_CLIENT_CONFIRMATION_RESULT result, void* userContextCallback) { (void)userContextCallback; g_message_count_send_confirmations++; LogInfo("Confirmation callback received for message %lu with result %s\r\n", (unsigned long)g_message_count_send_confirmations, MU_ENUM_TO_STRING(IOTHUB_CLIENT_CONFIRMATION_RESULT, result)); } static void connection_status_callback(IOTHUB_CLIENT_CONNECTION_STATUS result, IOTHUB_CLIENT_CONNECTION_STATUS_REASON reason, void* user_context) { (void)reason; (void)user_context; if (result == IOTHUB_CLIENT_CONNECTION_AUTHENTICATED) { LogInfo("The device client is connected to iothub\r\n"); } else { LogInfo("The device client has been disconnected\r\n"); } } void setup() { IOTHUB_CLIENT_TRANSPORT_PROVIDER protocol = MQTT_Protocol; sample_init(ssid, pass); (void)IoTHub_Init(); device_ll_handle = IoTHubDeviceClient_LL_CreateFromConnectionString(connectionString, protocol); LogInfo("Creating IoTHub Device handle...\r\n"); if (device_ll_handle == NULL) { LogInfo("Error AZ002: Failure creating Iothub device. Hint: Check you connection string.\r\n"); } else { int diag_off = 0; // turn off diagnostic sampling bool urlEncodeOn = true; IoTHubDeviceClient_LL_SetOption(device_ll_handle, OPTION_DIAGNOSTIC_SAMPLING_PERCENTAGE, &diag_off); IoTHubDeviceClient_LL_SetOption(device_ll_handle, OPTION_TRUSTED_CERT, edgeCert); // set trusted certs for the client IoTHubDeviceClient_LL_SetOption(device_ll_handle, OPTION_AUTO_URL_ENCODE_DECODE, &urlEncodeOn); (void)IoTHubDeviceClient_LL_SetConnectionStatusCallback(device_ll_handle, connection_status_callback, NULL); } } void destroy() { IoTHubDeviceClient_LL_Destroy(device_ll_handle); IoTHub_Deinit(); } void loop() { message_handle = IoTHubMessage_CreateFromString(telemetry_msg); LogInfo("Sending message %d to IoTHub...\r\n", (int)(messages_sent + 1)); IoTHubDeviceClient_LL_SendEventAsync(device_ll_handle, message_handle, send_confirm_callback, NULL); IoTHubMessage_Destroy(message_handle); messages_sent++; IoTHubDeviceClient_LL_DoWork(device_ll_handle); ThreadAPI_Sleep(3000); } ```

Output:

Creating IoTHub Device handle...

Sending message 1 to IoTHub...

dowork TLSIO_STATE_OPENING_WAITING_SOCKET
Sending message 2 to IoTHub...

dowork TLSIO_STATE_OPENING_WAITING_SSL
Error opening socket 0
Calling error callback
The device client has been disconnected

Error: failure opening connection to endpoint
The device client has been disconnected

Additional notes:

I am able to ping the hostname on my ESP8266 (tested with an example ping sketch)
I am able to verify the certificate
@ericwol-msft Subject CN name is iot-edge.leister

mleister97 commented 3 years ago

@ericwol-msft @sandervandevelde Any ideas why the code isn't working on ESP8266? As a said, the python example using the same connection string and certificate is working fine.

ericwolz commented 3 years ago

No, error points to a cert issue. You probably need to capture the network packets and look at the TLS cert exchange to understand if the cert is configured correctly. Also probably you will to trace into your TLS stack to see the SSL connect is failing.

mleister97 commented 3 years ago

To be honest I have no idea how to do that. The output of the following command produces this result:

 openssl s_client -connect iot-edge.leister:8883 -CAfile "C:\Users\micha\Desktop\azure-iot-test-only.root.ca.cert.pem" -showcerts

Output of the certificate command

``` CONNECTED(000001A0) depth=4 CN = Azure_IoT_Hub_CA_Cert_Test_Only verify return:1 depth=3 CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only verify return:1 depth=2 CN = ca.ca verify return:1 depth=1 CN = iotedged workload ca verify return:1 depth=0 CN = iot-edge.leister verify return:1 --- Certificate chain 0 s:CN = iot-edge.leister i:CN = iotedged workload ca -----BEGIN CERTIFICATE----- MIIEbjCCAlagAwIBAgIEa4tFZzANBgkqhkiG9w0BAQsFADAfMR0wGwYDVQQDDBRp ... -----END CERTIFICATE----- 1 s:CN = iotedged workload ca i:CN = ca.ca -----BEGIN CERTIFICATE----- MIIFUzCCAzugAwIBAgIEa4tFZzANBgkqhkiG9w0BAQsFADAQMQ4wDAYDVQQDDAVj ... -----END CERTIFICATE----- 2 s:CN = ca.ca i:CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only -----BEGIN CERTIFICATE----- MIIFWTCCA0GgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwNDEyMDAGA1UEAwwpQXp1 ... -----END CERTIFICATE----- 3 s:CN = Azure_IoT_Hub_Intermediate_Cert_Test_Only i:CN = Azure_IoT_Hub_CA_Cert_Test_Only -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- --- Server certificate subject=CN = iot-edge.leister issuer=CN = iotedged workload ca --- No client certificate CA names sent Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1:DSA+SHA224:DSA+SHA1:DSA+SHA256:DSA+SHA384:DSA+SHA512 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1:DSA+SHA224:DSA+SHA1:DSA+SHA256:DSA+SHA384:DSA+SHA512 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 5992 bytes and written 423 bytes Verification: OK --- New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 1FE693D33341B30444891078605B57F1014D2596C5D13568BE7804DF5630BD8D Session-ID-ctx: Master-Key: 73D8D88648F78CC14CD33C4ED465234C8A58402001DB6AED43700F21C5BE80571AEAD0666A31CA6521DE1A7E9C51BFE8 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - d9 32 19 1d 02 4d 23 70-a2 21 05 7a 58 a5 d7 23 .2...M#p.!.zX..# ... Start Time: 1616522274 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: yes --- ```

mleister97 commented 3 years ago

I am receiving the follwing error on the edge module:

[WRN] - TLS handshake failed., System.IO.IOException: Channel is closed, 7f6b90c6

ESP8266 Logs:

dowork TLSIO_STATE_OPENING_WAITING_SOCKET
Waiting for TLS connection
dowork TLSIO_STATE_OPENING_WAITING_SSL
Error opening socket 0
Calling error callback
Error signalled by underlying IO
Waiting for TLS connection
Open HTTP connection failed (result = HTTPAPI_OPEN_REQUEST_FAILED (4))
unable to recover sending to a working state
unable to HTTPAPIEX_SAS_ExecuteRequest
Confirmation callback received for message 1 with result IOTHUB_CLIENT_CONFIRMATION_ERROR

mleister97 commented 3 years ago

Seems like my issue is closely related to: https://github.com/Azure/iotedge/issues/387

mleister97 commented 3 years ago

Finally it's working. Switched the hardware from ESP8266 to ESP32 and running (same code) without any issues!

ericwolz commented 3 years ago

Ok, thanks. I'm going to close out the issue.

Azure / azure-iot-sdk-c