search

Azure / iot-hub-device-update

Device Update for IoT Hub agent
MIT License
52 stars 41 forks source link

Device Update Agent attempting MQTT connection over TLSv1 #549

Closed sandipanaich closed 10 months ago

sandipanaich commented 11 months ago

Hi,

I am facing an issue with a device update not reaching the CONNECTED state and I was hoping if you could share some debugging steps. I have 2 identical Azure Ubuntu VMs, but they are with 2 different but identical IoT Hubs (say, Dev and QA). For one of them (Dev) the device-update-agent is fine, but for the other one (ie. QA) it is not. Please note that all the other aspects of a device registering with the IoT Hub and having a proper Device Update for IoT Hub are in place.

We took a tcpdump and checked the packets. We noticed that on Dev, the MQTT connection on port 8883 is initiated by the device-update-agent over TLSv1.2, but on QA it is attempting the connection over TLSv1. Since the IoT Hub is configured to only accept TLSv1.2 connection, it is rejecting the TLSv1.2 on QA.

Can you please suggest any steps that we can follow to enforce the MQTT connection attempt over TLSv1.2 on QA as well? Otherwise, it would be helpful if you could provide us with any other debugging steps that we can carry out.

Since it is a blocker for QA, may I request it to be prioritized?

Thanks, Sandipan

jw-msft commented 11 months ago

@sandipanaich Could you run and provide output for the following?

(I am assuming "iotHubProtocol" in /etc/adu/du-config.json has value of "mqtt". Change 8883 port to 443 below instead if it has "mqtt/ws" for using MQTT over Websockets)

  1. openssl s_client -connect <ip4 addr of iothub host>:8883 -showcerts -prexit -servername localhost -debug It would be interesting to see if, independent of ADU client, what TLS version it negotiates with server during TLS handshake--I would be surprised if it is lower than TLS v1.2 unless the openssl installed on the system is really old, or there is a misconfiguration in openssl.cnf.

  2. tcpdump for TLS handshake

    
sudo systemctl stop deviceupdate-agent
sudo tcpdump -v -A -x "tcp port 8883 and (tcp[((tcp[12] & 0xf0) >> 2)] = 0x16)" > /tmp/out.pcap.txt 2>&1
# Now, do <Ctrl + Z> and run "bg" to background the task

then, run agent standalone (not as a service)

sudo /usr/bin/AducIotAgent -l0 -e

let it run for 1-2 seconds and then:

<Ctrl + C> fg <Ctrl + C>

provide /tmp/out.pcap.txt


Also, you could set MinProtocol in openssl.cnf to ensure it doesn't negotiate down to SSL or TLS 1.0/1.1.

First, find the basedir of openssl.cnf -- e.g. on my ubuntu 20.04 system:
```sh
openssl version -d

jw@host:~$ sudo openssl version -d
[sudo] password for jw:
OPENSSLDIR: "/usr/lib/ssl"

jw@host:~$ ll /usr/lib/ssl
total 20
lrwxrwxrwx   1 root root    16 Jul  1  2022 private -> /etc/ssl/private
lrwxrwxrwx   1 root root    14 Jul  1  2022 certs -> /etc/ssl/certs
lrwxrwxrwx   1 root root    20 May 24 10:14 openssl.cnf -> /etc/ssl/openssl.cnf
drwxr-xr-x   2 root root  4096 May 31 06:08 misc
drwxr-xr-x   3 root root  4096 May 31 06:08 .
drwxr-xr-x 162 root root 12288 Oct  4 20:13 ..

Then you can add MinProtocol = TLSv1.2 to the bottom. Make sure any intermediate proxies and firewalls also allow TLSv1.2

jw-msft commented 10 months ago

Resolving as incident was resolved. Check that intermediate proxies/nodes are not attempting to negotiate down to below TLS 1.2.