search

Azure / iotedge

The IoT Edge OSS project
MIT License
1.45k stars 457 forks source link

IoT Edge child node unable to connect to IoT Edge transparent gateway #7288

Open Freddy2es opened 1 month ago

Freddy2es commented 1 month ago

Expected Behavior

Following the documentation provided in the portal, I have configured two IoT Edge nodes, with one of them (master, with deviceid=my-edge-device-1) acting as a transparent gateway for the second one (slave, with deviceid=my-edge-device-2). If everything worked fine, the slave should be able to reach the IoT Hub through the master.

Current Behavior

When the slave tries to open the connection to the master, an unexpected error is detected apparently by the slave and it closes the connection (and keeps retrying it from time to time).

Steps to Reproduce

  1. Install all the IoT Edge components separately in both nodes and make sure they can communicate properly with the IoT Hub (direct connection with standard connection strings). Enable the edgeHub container in the portal in both cases.
  2. Switch the master to transparent gateway mode and configure the slave to connect to the IoT Hub through the master with the instructions detailed in the next points.

master configuration

  1. Create the certificates, the routes and open the firewall as described here
  2. Add the following entries to the file /etc/aziot/config.toml (before the [provisioning] section): 
    
hostname = "iot-edge-gateway.local"
trust_bundle_cert = "file:///var/aziot/certs/azure-iot-test-only.root.ca.cert.pem"

[edge_ca] cert = "file:///var/aziot/certs/iot-edge-device-ca-root.cert.pem" pk = "file:///var/aziot/secrets/iot-edge-device-ca-root.key.pem"

3. Modify the file **/etc/hosts** to include the resolution of the master's name to its static IP address in the local network (set **iot-edge-gateway.local** in **/etc/hostname** too):

1.2.3.4 iot-edge-gateway.local

4. Restart and apply the new configuration with **iotedge config apply**.
5. At this point, the master can communicate with the IoT Hub and modules can be activated from the portal (it works fine as a standalone node).

**slave configuration**
1. Copy the 3 CA files, namely **azure-iot-test-only.intermediate.cert.pem**, **azure-iot-test-only.root.ca.cert.pem** and **iot-edge-device-ca-root.cert.pem**.
2. Move them to the local CA path and update the system's local CA folder and update the global CA folder with:

sudo cp azure-iot-test-only.intermediate.cert.pem /usr/local/share/ca-certificates/azure-iot-test-only.intermediate.cert.pem.crt sudo cp azure-iot-test-only.root.ca.cert.pem /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.pem.crt sudo cp iot-edge-device-ca-root.cert.pem /usr/local/share/ca-certificates/iot-edge-device-ca-root.cert.pem.crt

sudo update-ca-certificates


As a result, 3 keys are added and the following links have been created in **/etc/ssl/certs**:

lrwxrwxrwx 1 root root 40 May 10 09:29 05c5438d.0 -> azure-iot-test-only.root.ca.cert.pem.pem lrwxrwxrwx 1 root root 45 May 10 09:50 2a3cc244.0 -> azure-iot-test-only.intermediate.cert.pem.pem lrwxrwxrwx 1 root root 36 May 10 09:50 5a615dc3.0 -> iot-edge-device-ca-root.cert.pem.pem lrwxrwxrwx 1 root root 78 May 10 09:50 azure-iot-test-only.intermediate.cert.pem.pem -> /usr/local/share/ca-certificates/azure-iot-test-only.intermediate.cert.pem.crt lrwxrwxrwx 1 root root 73 May 10 09:29 azure-iot-test-only.root.ca.cert.pem.pem -> /usr/local/share/ca-certificates/azure-iot-test-only.root.ca.cert.pem.crt lrwxrwxrwx 1 root root 69 May 10 09:50 iot-edge-device-ca-root.cert.pem.pem -> /usr/local/share/ca-certificates/iot-edge-device-ca-root.cert.pem.crt

3. Test the connection with:

openssl s_client -connect iot-edge-gateway.local:8883 -CApath /etc/ssl/certs -showcerts

It works fine (a **Verify return code: 0 (ok)** response is received).
4. Update the file **/etc/hosts** (set **iot-edge-slave.local** in **/etc/hostname** too):

1.2.3.4 iot-edge-gateway.local 127.0.1.1 iot-edge-slave.local

5. Update the file **/etc/aziot/config.toml** to enable the slave mode:

parent_hostname = "iot-edge-gateway.local"

[provisioning] source = "manual"

choose one of these two connection strings

connection_string = "HostName=YYYY;DeviceId=my-edge-device-2;SharedAccessKey=YYYY;GatewayHostName=iot-edge-gateway.local"

connection_string = "HostName=iot-edge-gateway.local;DeviceId=my-edge-device-2;SharedAccessKey=YYYY"

6. Restart and apply the new configuration with **iotedge config apply**.

## Context (Environment)

### Output of `iotedge check` in master

<details>
<summary>Click here</summary>

Configuration checks (aziot-identity-service)

√ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK √ aziot-identity-service package is up-to-date - OK √ host time is close to reference time - OK √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK √ read all preloaded certificates from the Certificates Service - OK √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)

√ host can connect to and perform TLS handshake with iothub AMQP port - OK √ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK √ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks

√ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK √ configuration has correct URIs for daemon mgmt endpoint - OK √ aziot-edge package is up-to-date - OK √ container time is close to host time - OK √ DNS server - OK √ production readiness: logs policy - OK √ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK √ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK √ Agent image is valid and can be pulled from upstream - OK √ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks

√ container on the default network can connect to upstream AMQP port - OK √ container on the default network can connect to upstream HTTPS / WebSockets port - OK √ container on the default network can connect to upstream MQTT port - OK skipping because of not required in this configuration √ container on the IoT Edge module network can connect to upstream AMQP port - OK √ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK √ container on the IoT Edge module network can connect to upstream MQTT port - OK skipping because of not required in this configuration 35 check(s) succeeded. 2 check(s) were skipped due to errors from other checks.

</details>

### Output of `iotedge check` in slave

<details>
<summary>Click here</summary>

Configuration checks (aziot-identity-service)

√ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK √ host time is close to reference time - OK √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK √ read all preloaded certificates from the Certificates Service - OK √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)

√ host can connect to and perform TLS handshake with iothub AMQP port - OK √ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK √ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks

√ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK √ configuration has correct parent_hostname - OK × parent hostname is resolvable from inside container - Error Failed to resolve parent hostname iot-edge-gateway.local caused by: Failed to resolve parent hostname iot-edge-gateway.local caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. × configuration has correct URIs for daemon mgmt endpoint - Error Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. caused by: Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. × container time is close to host time - Error Could not query local time inside container caused by: Could not query local time inside container caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. √ DNS server - OK √ production readiness: logs policy - OK × production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error Could not check current state of edgeAgent container caused by: Could not check current state of edgeAgent container caused by: docker returned exit status: 1, stderr = Error: No such object: edgeAgent × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error Could not check current state of edgeHub container caused by: Could not check current state of edgeHub container caused by: docker returned exit status: 1, stderr = Error: No such object: edgeHub √ Agent image is valid and can be pulled from upstream - OK √ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks

× container on the default network can connect to upstream AMQP port - Error Container on the default network could not connect to iot-edge-gateway.local:5671 caused by: Container on the default network could not connect to iot-edge-gateway.local:5671 caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. × container on the default network can connect to upstream HTTPS / WebSockets port - Error Container on the default network could not connect to iot-edge-gateway.local:443 caused by: Container on the default network could not connect to iot-edge-gateway.local:443 caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. √ container on the default network can connect to upstream MQTT port - OK skipping because of not required in this configuration × container on the IoT Edge module network can connect to upstream AMQP port - Error Container on the azure-iot-edge network could not connect to iot-edge-gateway.local:5671 caused by: Container on the azure-iot-edge network could not connect to iot-edge-gateway.local:5671 caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. × container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error Container on the azure-iot-edge network could not connect to iot-edge-gateway.local:443 caused by: Container on the azure-iot-edge network could not connect to iot-edge-gateway.local:443 caused by: docker returned exit status: 125, stderr = Unable to find image 'iot-edge-gateway.local/azureiotedge-diagnostics:1.5.0' locally docker: Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: "". See 'docker run --help'. √ container on the IoT Edge module network can connect to upstream MQTT port - OK skipping because of not required in this configuration 26 check(s) succeeded. 9 check(s) raised errors. 2 check(s) were skipped due to errors from other checks.

</details>

### Device Information (both)
* Host OS [Debian 11 (Bullseye)]: 
* Architecture [arm32]: 
* Container OS [Linux containers]: 

### Runtime Versions (both)
* aziot-edged [1.5.0]: 
* Edge Agent [mcr.microsoft.com/azureiotedge-agent:1.5]: 
* Edge Hub [mcr.microsoft.com/azureiotedge-hub:1.5]: 
* Docker/Moby [moby 23.0.7+azure-1]: 

## Logs

<details>
<summary>edge-agent logs (master)</summary>

2024-05-11 16:52:37 Starting Edge Agent 2024-05-11 16:52:37 Creating UID 13622 as edgeagentuser 2024-05-11 16:52:37 Changing ownership of storage folder: /var/aziot/edgeAgent/module/edgeAgent to 13622 2024-05-11 16:52:37 Creating backup folder: /tmp/edgeAgent_backup 2024-05-11 16:52:37 Changing ownership of management socket: /var/run/iotedge/mgmt.sock 2024-05-11 16:52:37 Completed necessary setup. Starting Edge Agent. 2024-05-11 16:52:37.529 +00:00 Edge Agent Main()

<6> 2024-05-11 16:52:38.029 +00:00 [INF] - Initializing Edge Agent. <6> 2024-05-11 16:52:38.395 +00:00 [INF] - Version - 1.5.0.92636310 (7523e680ae9feacaa54df13de80fa82a2b2e7b74) <6> 2024-05-11 16:52:38.397 +00:00 [INF] - █████╗ ███████╗██╗ ██╗██████╗ ███████╗ ██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝ ███████║ ███╔╝ ██║ ██║██████╔╝█████╗ ██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝ ██║ ██║███████╗╚██████╔╝██║ ██║███████╗ ╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝ ██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗ ██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝ ██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗ ██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝ ██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗ ╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝ <6> 2024-05-11 16:52:38.408 +00:00 [INF] - ModuleUpdateMode: NonBlocking <6> 2024-05-11 16:52:38.552 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false} <6> 2024-05-11 16:52:38.886 +00:00 [INF] - Installing certificates [CN=root.ca:07/25/2032 15:15:22],[CN=Azure_IoT_Hub_CA_Cert_Test_Only:07/25/2032 15:13:53] to Root <6> 2024-05-11 16:52:39.284 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics <6> 2024-05-11 16:52:39.319 +00:00 [INF] - Updating performance metrics every 05m:00s <6> 2024-05-11 16:52:39.334 +00:00 [INF] - Started operation Get system resources <6> 2024-05-11 16:52:39.339 +00:00 [INF] - Collecting metadata metrics <6> 2024-05-11 16:52:39.552 +00:00 [INF] - Set metadata metrics: 1.5.0.92636310 (7523e680ae9feacaa54df13de80fa82a2b2e7b74), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"arm","Version":"1.5.0","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"23.0.7+azure-1","KernelVersion":"#1642 SMP Mon Apr 3 17:22:30 BST 2023","OperatingSystem":"raspbian","NumCpus":4,"TotalMemory":8282042368,"Virtualized":"no"}, True <6> 2024-05-11 16:52:39.605 +00:00 [INF] - Started operation Checkpoint Availability <6> 2024-05-11 16:52:39.616 +00:00 [INF] - Started operation refresh twin config <6> 2024-05-11 16:52:39.658 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only... <6> 2024-05-11 16:52:40.452 +00:00 [INF] - Created persistent store at /var/aziot/edgeAgent/module/edgeAgent <6> 2024-05-11 16:52:40.708 +00:00 [INF] - Started operation Metrics Scrape <6> 2024-05-11 16:52:40.708 +00:00 [INF] - Started operation Metrics Upload Scraping frequency: 01:00:00 Upload Frequency: 1.00:00:00 <6> 2024-05-11 16:52:41.764 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only. <6> 2024-05-11 16:52:41.770 +00:00 [INF] - Registering request handler UploadModuleLogs <6> 2024-05-11 16:52:41.771 +00:00 [INF] - Registering request handler GetModuleLogs <6> 2024-05-11 16:52:41.771 +00:00 [INF] - Registering request handler UploadSupportBundle <6> 2024-05-11 16:52:41.772 +00:00 [INF] - Registering request handler RestartModule <6> 2024-05-11 16:52:42.019 +00:00 [INF] - Initialized new module client with subscriptions enabled <6> 2024-05-11 16:52:42.129 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 37 and reported properties version 191. <6> 2024-05-11 16:52:45.185 +00:00 [INF] - Plan execution started for deployment 37 <6> 2024-05-11 16:52:45.191 +00:00 [INF] - Executing command: "[Null]" <6> 2024-05-11 16:52:45.200 +00:00 [INF] - Executing command: "[Null]" <6> 2024-05-11 16:52:45.201 +00:00 [INF] - Executing command: "[Null]" <6> 2024-05-11 16:52:45.201 +00:00 [INF] - Executing command: "[Null]" <6> 2024-05-11 16:52:45.231 +00:00 [INF] - Executing command: "Command Group: (\n [Command Group: (\n [Prepare module edgeHub]\n [Create module edgeHub]\n)]\n [Start module edgeHub]\n)" <6> 2024-05-11 16:52:45.236 +00:00 [INF] - Executing command: "Command Group: (\n [Prepare module edgeHub]\n [Create module edgeHub]\n)" <6> 2024-05-11 16:52:47.720 +00:00 [INF] - Executing command: "Start module edgeHub" <6> 2024-05-11 16:52:55.511 +00:00 [INF] - Executing command: "Command Group: (\n [Command Group: (\n [Prepare module nodered]\n [Create module nodered]\n)]\n [Start module nodered]\n)" <6> 2024-05-11 16:52:55.512 +00:00 [INF] - Executing command: "Command Group: (\n [Prepare module nodered]\n [Create module nodered]\n)" <6> 2024-05-11 16:53:00.122 +00:00 [INF] - Executing command: "Start module nodered" <6> 2024-05-11 16:53:02.461 +00:00 [INF] - Executing command: "Command Group: (\n [Command Group: (\n [Prepare module eclipse-mosquitto]\n [Create module eclipse-mosquitto]\n)]\n [Start module eclipse-mosquitto]\n)" <6> 2024-05-11 16:53:02.461 +00:00 [INF] - Executing command: "Command Group: (\n [Prepare module eclipse-mosquitto]\n [Create module eclipse-mosquitto]\n)" <6> 2024-05-11 16:53:06.607 +00:00 [INF] - Executing command: "Start module eclipse-mosquitto" <6> 2024-05-11 16:53:07.784 +00:00 [INF] - Plan execution ended for deployment 37 <6> 2024-05-11 16:53:08.238 +00:00 [INF] - Updated reported properties <6> 2024-05-11 16:53:13.359 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (37s left). <6> 2024-05-11 16:53:13.541 +00:00 [INF] - Updated reported properties <6> 2024-05-11 16:53:18.593 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (31s left). <6> 2024-05-11 16:53:23.632 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (26s left). <6> 2024-05-11 16:53:28.702 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (21s left). <6> 2024-05-11 16:53:33.778 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (16s left). <6> 2024-05-11 16:53:38.834 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (11s left). <6> 2024-05-11 16:53:44.547 +00:00 [INF] - Module 'nodered' scheduled to restart after 40s (05s left). <6> 2024-05-11 16:53:44.548 +00:00 [INF] - Plan execution started for deployment 37 <6> 2024-05-11 16:53:45.588 +00:00 [INF] - Plan execution ended for deployment 37 <6> 2024-05-11 16:53:50.706 +00:00 [INF] - Plan execution started for deployment 37 <6> 2024-05-11 16:53:50.706 +00:00 [INF] - Executing command: "Command Group: (\n [Stop module nodered]\n [Start module nodered]\n [Saving nodered to store]\n)" <6> 2024-05-11 16:53:50.707 +00:00 [INF] - Executing command: "Stop module nodered" <6> 2024-05-11 16:53:50.712 +00:00 [INF] - Executing command: "Start module nodered" <6> 2024-05-11 16:53:51.828 +00:00 [INF] - Executing command: "Saving nodered to store" <6> 2024-05-11 16:53:51.828 +00:00 [INF] - Plan execution ended for deployment 37 <6> 2024-05-11 16:53:52.065 +00:00 [INF] - Updated reported properties ... (other non-relevant output after this point) ```
edge-hub logs (master) ``` <6> 2024-05-11 17:13:58.660 +00:00 [INF] - Request starting "HTTP/1.1" "GET" "https"://"iot-edge-gateway.local""""/devices/my-edge-device-2/modules""?api-version=2017-11-08-preview" - null null <6> 2024-05-11 17:13:58.661 +00:00 [INF] - Executing endpoint '"Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController.ListModulesAsync (Microsoft.Azure.Devices.Edge.Hub.Http)"' <6> 2024-05-11 17:13:58.662 +00:00 [INF] - Route matched with "{action = \"ListModules\", controller = \"Registry\"}". Executing controller action with signature "System.Threading.Tasks.Task ListModulesAsync(System.String)" on controller "Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController" ("Microsoft.Azure.Devices.Edge.Hub.Http"). <6> 2024-05-11 17:13:58.663 +00:00 [INF] - Received request in ListModulesAsync: deviceId=my-edge-device-2, moduleId= <6> 2024-05-11 17:13:58.664 +00:00 [INF] - Client my-edge-device-2 in device scope authenticated locally. <6> 2024-05-11 17:13:58.665 +00:00 [INF] - Authenticated: deviceId=my-edge-device-2, moduleId= <6> 2024-05-11 17:13:58.989 +00:00 [INF] - CompleteRequest in ListModulesAsync: deviceId=my-edge-device-1, authChain=my-edge-device-2;my-edge-device-1, status=OK <6> 2024-05-11 17:13:58.990 +00:00 [INF] - Executed action "Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController.ListModulesAsync (Microsoft.Azure.Devices.Edge.Hub.Http)" in 327.1158ms <6> 2024-05-11 17:13:58.991 +00:00 [INF] - Executed endpoint '"Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController.ListModulesAsync (Microsoft.Azure.Devices.Edge.Hub.Http)"' <6> 2024-05-11 17:13:58.991 +00:00 [INF] - Request finished "HTTP/1.1" "GET" "https"://"iot-edge-gateway.local""""/devices/my-edge-device-2/modules""?api-version=2017-11-08-preview" - 200 1643 "application/json; charset=utf-8" 331.3034ms <6> 2024-05-11 17:13:59.023 +00:00 [INF] - Request starting "HTTP/1.1" "DELETE" "https"://"iot-edge-gateway.local""""/devices/my-edge-device-2/modules/DefenderIotMicroAgent""?api-version=2017-11-08-preview" - null null <6> 2024-05-11 17:13:59.024 +00:00 [INF] - Executing endpoint '"Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController.DeleteModuleAsync (Microsoft.Azure.Devices.Edge.Hub.Http)"' <6> 2024-05-11 17:13:59.025 +00:00 [INF] - Route matched with "{action = \"DeleteModule\", controller = \"Registry\"}". Executing controller action with signature "System.Threading.Tasks.Task DeleteModuleAsync(System.String, System.String)" on controller "Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController" ("Microsoft.Azure.Devices.Edge.Hub.Http"). <6> 2024-05-11 17:13:59.027 +00:00 [INF] - Received request in DeleteModuleAsync: deviceId=my-edge-device-2, moduleId=DefenderIotMicroAgent <6> 2024-05-11 17:13:59.027 +00:00 [INF] - Client my-edge-device-2 in device scope authenticated locally. <6> 2024-05-11 17:13:59.028 +00:00 [INF] - Authenticated: deviceId=my-edge-device-2, moduleId= <6> 2024-05-11 17:13:59.333 +00:00 [INF] - CompleteRequest in DeleteModuleAsync: deviceId=my-edge-device-1, authChain=my-edge-device-2;my-edge-device-1, status=PreconditionFailed <6> 2024-05-11 17:13:59.333 +00:00 [INF] - Executed action "Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController.DeleteModuleAsync (Microsoft.Azure.Devices.Edge.Hub.Http)" in 306.6126ms <6> 2024-05-11 17:13:59.333 +00:00 [INF] - Executed endpoint '"Microsoft.Azure.Devices.Edge.Hub.Http.Controllers.RegistryController.DeleteModuleAsync (Microsoft.Azure.Devices.Edge.Hub.Http)"' <6> 2024-05-11 17:13:59.333 +00:00 [INF] - Request finished "HTTP/1.1" "DELETE" "https"://"iot-edge-gateway.local""""/devices/my-edge-device-2/modules/DefenderIotMicroAgent""?api-version=2017-11-08-preview" - 412 259 "application/json; charset=utf-8" 310.3569ms ... (same pattern continues on every retry from the slave) ```
journalctl -xe (slave) ``` May 11 19:13:58 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:58Z [INFO] - Starting service... May 11 19:13:58 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:58Z [INFO] - Version - 1.5.0 May 11 19:13:58 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:58Z [INFO] - Provisioning starting. Reason: Startup May 11 19:13:58 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:58Z [INFO] - Updated device info for my-edge-device-2. May 11 19:13:58 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:58Z [INFO] - Provisioning complete. May 11 19:13:58 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:58Z [INFO] - Identity reconciliation started. Reason: Startup May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "374"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "374"} May 11 19:13:58 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:58Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [INFO] - Could not reconcile Identities with current device data. Reprovisioning. May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [INFO] - Updated device info for my-edge-device-2. May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "374"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "374"} May 11 19:13:59 iot-edge-slave.local aziot-keyd[24695]: 2024-05-11T17:13:59Z [INFO] - --> 200 {"content-type": "application/json"} May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: Hub client error May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [ERR!] - service encountered an error May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [ERR!] - caused by: Hub client error May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [ERR!] - caused by: invalid HTTP status code May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 2024-05-11T17:13:59Z [ERR!] - 0: May 11 19:13:59 iot-edge-slave.local aziot-identityd[24772]: 1: May 11 19:13:59 iot-edge-slave.local systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE ```
edge-agent logs (slave, just to show that once it's configured to communicate directly to the IoT Hub and there's a local copy of the container, it works fine) ``` 2024-05-12 11:44:27 Starting Edge Agent 2024-05-12 11:44:27 Creating UID 13622 as edgeagentuser 2024-05-12 11:44:27 Changing ownership of storage folder: /var/aziot/edgeAgent/module/edgeAgent to 13622 2024-05-12 11:44:27 Creating backup folder: /tmp/edgeAgent_backup 2024-05-12 11:44:27 Changing ownership of management socket: /var/run/iotedge/mgmt.sock 2024-05-12 11:44:27 Completed necessary setup. Starting Edge Agent. 2024-05-12 11:44:27.862 +00:00 Edge Agent Main() <6> 2024-05-12 11:44:28.342 +00:00 [INF] - Initializing Edge Agent. <6> 2024-05-12 11:44:28.708 +00:00 [INF] - Version - 1.5.0.92636310 (7523e680ae9feacaa54df13de80fa82a2b2e7b74) <6> 2024-05-12 11:44:28.709 +00:00 [INF] - █████╗ ███████╗██╗ ██╗██████╗ ███████╗ ██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝ ███████║ ███╔╝ ██║ ██║██████╔╝█████╗ ██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝ ██║ ██║███████╗╚██████╔╝██║ ██║███████╗ ╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝ ██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗ ██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝ ██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗ ██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝ ██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗ ╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝ <6> 2024-05-12 11:44:28.720 +00:00 [INF] - ModuleUpdateMode: NonBlocking <6> 2024-05-12 11:44:28.866 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false} <6> 2024-05-12 11:44:29.160 +00:00 [INF] - Installing certificates [CN=aziot-edge CA my-edge-device-2:08/06/2024 10:33:13] to Root <6> 2024-05-12 11:44:29.521 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics <6> 2024-05-12 11:44:29.563 +00:00 [INF] - Updating performance metrics every 05m:00s <6> 2024-05-12 11:44:29.580 +00:00 [INF] - Started operation Get system resources <6> 2024-05-12 11:44:29.584 +00:00 [INF] - Collecting metadata metrics <6> 2024-05-12 11:44:29.803 +00:00 [INF] - Set metadata metrics: 1.5.0.92636310 (7523e680ae9feacaa54df13de80fa82a2b2e7b74), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"arm","Version":"1.5.0","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"23.0.7+azure-1","KernelVersion":"#1642 SMP Mon Apr 3 17:22:30 BST 2023","OperatingSystem":"raspbian","NumCpus":4,"TotalMemory":8282042368,"Virtualized":"no"}, True <6> 2024-05-12 11:44:29.855 +00:00 [INF] - Started operation Checkpoint Availability <6> 2024-05-12 11:44:29.863 +00:00 [INF] - Started operation refresh twin config <6> 2024-05-12 11:44:29.905 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only... <6> 2024-05-12 11:44:30.670 +00:00 [INF] - Created persistent store at /var/aziot/edgeAgent/module/edgeAgent <6> 2024-05-12 11:44:30.907 +00:00 [INF] - Started operation Metrics Scrape <6> 2024-05-12 11:44:30.908 +00:00 [INF] - Started operation Metrics Upload Scraping frequency: 01:00:00 Upload Frequency: 1.00:00:00 <6> 2024-05-12 11:44:31.940 +00:00 [INF] - Registering request handler UploadModuleLogs <6> 2024-05-12 11:44:31.941 +00:00 [INF] - Registering request handler GetModuleLogs <6> 2024-05-12 11:44:31.941 +00:00 [INF] - Registering request handler UploadSupportBundle <6> 2024-05-12 11:44:31.942 +00:00 [INF] - Registering request handler RestartModule <6> 2024-05-12 11:44:32.060 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only. <6> 2024-05-12 11:44:32.306 +00:00 [INF] - Initialized new module client with subscriptions enabled <6> 2024-05-12 11:44:32.424 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 2 and reported properties version 6. <6> 2024-05-12 11:44:33.907 +00:00 [INF] - Plan execution started for deployment 2 <6> 2024-05-12 11:44:33.914 +00:00 [INF] - Executing command: "[Null]" <6> 2024-05-12 11:44:33.952 +00:00 [INF] - Executing command: "Command Group: (\n [Command Group: (\n [Prepare module edgeHub]\n [Create module edgeHub]\n)]\n [Start module edgeHub]\n)" <6> 2024-05-12 11:44:33.957 +00:00 [INF] - Executing command: "Command Group: (\n [Prepare module edgeHub]\n [Create module edgeHub]\n)" <6> 2024-05-12 11:44:36.435 +00:00 [INF] - Executing command: "Start module edgeHub" <6> 2024-05-12 11:44:37.558 +00:00 [INF] - Plan execution ended for deployment 2 <6> 2024-05-12 11:44:37.956 +00:00 [INF] - Updated reported properties <6> 2024-05-12 11:44:44.067 +00:00 [INF] - Updated reported properties <6> 2024-05-12 11:45:02.695 +00:00 [INF] - Termination requested, initiating shutdown. <6> 2024-05-12 11:45:02.695 +00:00 [INF] - Waiting for cleanup to finish <6> 2024-05-12 11:45:02.697 +00:00 [INF] - Main thread terminated <6> 2024-05-12 11:45:02.703 +00:00 [INF] - Initiating shutdown cleanup. <6> 2024-05-12 11:45:02.711 +00:00 [INF] - Stopping all modules... <6> 2024-05-12 11:45:02.769 +00:00 [INF] - Plan execution started for deployment -1 <6> 2024-05-12 11:45:02.773 +00:00 [INF] - Executing command: "Stop module edgeHub" <6> 2024-05-12 11:45:02.953 +00:00 [INF] - Updated reported properties <6> 2024-05-12 11:45:13.742 +00:00 [INF] - Plan execution ended for deployment -1 <6> 2024-05-12 11:45:13.743 +00:00 [INF] - Completed stopping all modules. <6> 2024-05-12 11:45:13.746 +00:00 [INF] - Completed shutdown cleanup. <6> 2024-05-12 11:45:13.750 +00:00 [INF] - Done with cleanup. Shutting down. <6> 2024-05-12 11:45:13.757 +00:00 [INF] - Disposing module connection object ```
edge-hub logs (slave, in parallel with the previous one of the edge-agent) ``` 2024-05-12 11:44:37 Starting Edge Hub 2024-05-12 11:44:37 Starting Edge Hub 2024-05-12 11:44:37 Creating UID 13623 as edgehubuser 2024-05-12 11:44:37 Changing ownership of storage folder: /var/aziot/edgeHub/module/edgeHub to 13623 2024-05-12 11:44:37 Creating backup folder: /tmp/edgeHub_backup 2024-05-12 11:44:38.166 +00:00 Edge Hub Main() <6> 2024-05-12 11:44:44.275 +00:00 [INF] - Installing certificates [CN=aziot-edge CA my-edge-device-2:08/06/2024 10:33:13] to Root <6> 2024-05-12 11:44:44.401 +00:00 [INF] - Installing certificates [CN=aziot-edge CA my-edge-device-2:08/06/2024 10:33:13] to Root <6> 2024-05-12 11:44:44.495 +00:00 [INF] - Enabling SSL protocols: Tls12, Tls13 <6> 2024-05-12 11:44:45.039 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false} <6> 2024-05-12 11:44:46.283 +00:00 [INF] - Created persistent store at /var/aziot/edgeHub/module/edgeHub <6> 2024-05-12 11:44:46.520 +00:00 [INF] - Initializing Edge Hub <6> 2024-05-12 11:44:46.521 +00:00 [INF] - █████╗ ███████╗██╗ ██╗██████╗ ███████╗ ██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝ ███████║ ███╔╝ ██║ ██║██████╔╝█████╗ ██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝ ██║ ██║███████╗╚██████╔╝██║ ██║███████╗ ╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝ ██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗ ██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝ ██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗ ██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝ ██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗ ╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝ <6> 2024-05-12 11:44:46.524 +00:00 [INF] - Version - 1.5.0.92636310 (7523e680ae9feacaa54df13de80fa82a2b2e7b74) <6> 2024-05-12 11:44:46.524 +00:00 [INF] - OptimizeForPerformance=True <6> 2024-05-12 11:44:46.524 +00:00 [INF] - MessageAckTimeoutSecs=30 <6> 2024-05-12 11:44:46.530 +00:00 [INF] - Loaded server certificate with expiration date of "2024-06-11T11:44:43.0000000+00:00" <6> 2024-05-12 11:44:46.558 +00:00 [INF] - Using Asp Net server for metrics <6> 2024-05-12 11:44:46.672 +00:00 [INF] - Created new message store <6> 2024-05-12 11:44:46.672 +00:00 [INF] - Started task to cleanup processed and stale messages <6> 2024-05-12 11:44:46.698 +00:00 [INF] - Add node: my-edge-device-2 <6> 2024-05-12 11:44:46.705 +00:00 [INF] - Add node: my-edge-device-2/$edgeAgent <6> 2024-05-12 11:44:46.705 +00:00 [INF] - Add node: my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:46.706 +00:00 [INF] - Add node: my-edge-device-2/DefenderIotMicroAgent <6> 2024-05-12 11:44:46.717 +00:00 [INF] - Created device scope identities cache <6> 2024-05-12 11:44:46.762 +00:00 [INF] - Starting refresh of device scope identities cache <6> 2024-05-12 11:44:46.855 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00 <6> 2024-05-12 11:44:46.915 +00:00 [INF] - Initialized storing twin manager <6> 2024-05-12 11:44:46.970 +00:00 [INF] - Updated node: my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:46.970 +00:00 [INF] - Initializing configuration <6> 2024-05-12 11:44:47.002 +00:00 [INF] - New device connection for device my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:47.022 +00:00 [INF] - Client my-edge-device-2/$edgeHub connected to edgeHub, processing existing subscriptions. <6> 2024-05-12 11:44:47.106 +00:00 [INF] - Attempting to connect to IoT Hub for client my-edge-device-2/$edgeHub via AMQP... <6> 2024-05-12 11:44:47.147 +00:00 [INF] - Processing pending subscriptions for my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:47.728 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false} <6> 2024-05-12 11:44:47.793 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds <6> 2024-05-12 11:44:47.877 +00:00 [INF] - No routes set in the edge hub <6> 2024-05-12 11:44:47.880 +00:00 [INF] - Updated message store TTL to 7200 seconds <6> 2024-05-12 11:44:47.881 +00:00 [INF] - Updated the edge hub store and forward configuration <6> 2024-05-12 11:44:47.943 +00:00 [INF] - Started operation Get EdgeHub config <6> 2024-05-12 11:44:47.944 +00:00 [INF] - Initialized edge hub configuration <6> 2024-05-12 11:44:47.955 +00:00 [INF] - Scheduling server certificate renewal for "2024-06-11T11:42:13.0016504Z". <6> 2024-05-12 11:44:47.959 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP) <6> 2024-05-12 11:44:47.973 +00:00 [INF] - Starting MQTT head <6> 2024-05-12 11:44:48.045 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head. <6> 2024-05-12 11:44:48.177 +00:00 [INF] - Starting AMQP head <6> 2024-05-12 11:44:48.211 +00:00 [INF] - Started MQTT head <6> 2024-05-12 11:44:48.562 +00:00 [INF] - Started AMQP head <6> 2024-05-12 11:44:48.566 +00:00 [INF] - Starting HTTP head <6> 2024-05-12 11:44:48.633 +00:00 [INF] - Not changed node: my-edge-device-2 <6> 2024-05-12 11:44:48.633 +00:00 [INF] - Not changed node: my-edge-device-2/$edgeAgent <6> 2024-05-12 11:44:48.634 +00:00 [INF] - Updated node: my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:48.724 +00:00 [INF] - Service identity for my-edge-device-2/$edgeHub in device scope was updated. <6> 2024-05-12 11:44:48.725 +00:00 [INF] - Not changed node: my-edge-device-2/DefenderIotMicroAgent <6> 2024-05-12 11:44:48.733 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes. <6> 2024-05-12 11:44:49.363 +00:00 [INF] - Started HTTP head <6> 2024-05-12 11:44:49.689 +00:00 [INF] - Exiting disconnected state <6> 2024-05-12 11:44:49.728 +00:00 [INF] - Received device connected callback <6> 2024-05-12 11:44:49.742 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients. <6> 2024-05-12 11:44:49.746 +00:00 [INF] - Processing subscriptions for client my-edge-device-2/$edgeHub on device connected to cloud. <6> 2024-05-12 11:44:49.747 +00:00 [INF] - Skipping my-edge-device-2/$edgeHub for subscription processing, as it is currently being processed. <6> 2024-05-12 11:44:49.750 +00:00 [INF] - Skipping cache refresh, waiting 118 seconds until refreshing again. <6> 2024-05-12 11:44:49.769 +00:00 [INF] - Entering connected state <6> 2024-05-12 11:44:49.778 +00:00 [INF] - Cloud connection for my-edge-device-2/$edgeHub is True <6> 2024-05-12 11:44:49.787 +00:00 [INF] - Connection status for my-edge-device-2/$edgeHub changed to ConnectionEstablished <6> 2024-05-12 11:44:49.791 +00:00 [INF] - Client my-edge-device-2/$edgeHub connected to cloud, processing existing subscriptions. <6> 2024-05-12 11:44:49.791 +00:00 [INF] - Skipping my-edge-device-2/$edgeHub for subscription processing, as it is currently being processed. <6> 2024-05-12 11:44:49.796 +00:00 [INF] - Created cloud proxy for client my-edge-device-2/$edgeHub via AMQP, with client operation timeout 20 seconds. <6> 2024-05-12 11:44:49.804 +00:00 [INF] - Initialized cloud proxy 589eba1f-d54f-4c88-88e1-779ee471518f for my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:49.807 +00:00 [INF] - Created cloud connection for client my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:50.126 +00:00 [INF] - Processing pending subscriptions for my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:50.165 +00:00 [INF] - Updated reported properties for my-edge-device-2/$edgeHub <6> 2024-05-12 11:44:50.486 +00:00 [INF] - Obtained edge hub config from module twin <6> 2024-05-12 11:44:52.155 +00:00 [INF] - Updated reported properties for my-edge-device-2/$edgeHub <6> 2024-05-12 11:45:02.688 +00:00 [INF] - Termination requested, initiating shutdown. <6> 2024-05-12 11:45:02.690 +00:00 [INF] - Stopping the protocol heads... <6> 2024-05-12 11:45:02.691 +00:00 [INF] - Closing protocol heads - (MQTT, AMQP, HTTP) <6> 2024-05-12 11:45:02.698 +00:00 [INF] - Stopping MQTT protocol head <6> 2024-05-12 11:45:02.740 +00:00 [INF] - Closing HTTP head <6> 2024-05-12 11:45:02.784 +00:00 [INF] - Closed HTTP head <6> 2024-05-12 11:45:02.785 +00:00 [INF] - Waiting for cleanup to finish ```
## Additional Information I get the same result either using the extended format in the connection string (adding the **GatewayHostName** attribute) or replacing the IoT Hub **HostName** to use the **hostname** of the master instead. All the logs have been generated using the second option (with **"HostName=iot-edge-gateway.local;DeviceId=my-edge-device-2;SharedAccessKey=YYYY"**). These are the contents of the most relevant config files in the slave after applying the config. **/etc/aziot/config.toml** ``` auto_reprovisioning_mode = "OnErrorOnly" prefer_module_identity_cache = false parent_hostname = "iot-edge-gateway.local" [provisioning] source = "manual" #connection_string = "HostName=XXX;DeviceId=my-edge-device-2;SharedAccessKey=YYY;GatewayHostName=iot-edge-gateway.local" connection_string = "HostName=iot-edge-gateway.local;DeviceId=my-edge-device-2;SharedAccessKey=YYY" [aziot_keys] [preloaded_keys] [cert_issuance] [preloaded_certs] [tpm] [agent] name = "edgeAgent" type = "docker" imagePullPolicy = "on-create" [agent.config] image = "mcr.microsoft.com/azureiotedge-agent:1.5" [agent.config.createOptions] [agent.env] [connect] workload_uri = "unix:///var/run/iotedge/workload.sock" management_uri = "unix:///var/run/iotedge/mgmt.sock" [listen] workload_uri = "fd://aziot-edged.workload.socket" management_uri = "fd://aziot-edged.mgmt.socket" [watchdog] max_retries = "infinite" [moby_runtime] uri = "unix:///var/run/docker.sock" network = "azure-iot-edge" ``` **/etc/aziot/edged/config.d/00-super.toml** ``` # This file is auto-generated by `iotedge config apply` # Do not edit it manually; any edits will be lost when the command is run again. hostname = "iot-edge-slave.local" trust_bundle_cert = "aziot-edged-trust-bundle" auto_reprovisioning_mode = "OnErrorOnly" homedir = "/var/lib/aziot/edged" allow_elevated_docker_permissions = true [edge_ca.auto_renew] rotate_key = true threshold = "80%" retry = "4%" [agent] name = "edgeAgent" type = "docker" imagePullPolicy = "on-create" [agent.config] image = "mcr.microsoft.com/azureiotedge-agent:1.5" [agent.config.createOptions] [agent.env] [connect] workload_uri = "unix:///var/run/iotedge/workload.sock" management_uri = "unix:///var/run/iotedge/mgmt.sock" [listen] workload_uri = "fd://aziot-edged.workload.socket" management_uri = "fd://aziot-edged.mgmt.socket" [watchdog] max_retries = "infinite" [moby_runtime] uri = "unix:///var/run/docker.sock" network = "azure-iot-edge" ``` **/etc/aziot/identityd/config.d/00-super.toml** ``` # This file is auto-generated by `iotedge config apply` # Do not edit it manually; any edits will be lost when the command is run again. hostname = "iot-edge-slave.local" homedir = "/var/lib/aziot/identityd" prefer_module_identity_cache = false [provisioning] local_gateway_hostname = "iot-edge-gateway.local" source = "manual" iothub_hostname = "iot-edge-gateway.local" device_id = "my-edge-device-2" [provisioning.authentication] method = "sas" device_id_pk = "device-id" [[principal]] uid = 994 name = "aziot-edge" ``` **/etc/aziot/keyd/config.d/00-super.toml** ``` # This file is auto-generated by `iotedge config apply` # Do not edit it manually; any edits will be lost when the command is run again. [aziot_keys] homedir_path = "/var/lib/aziot/keyd" [preloaded_keys] device-id = "file:///var/secrets/aziot/keyd/device-id" [[principal]] uid = 995 keys = ["aziot_identityd_master_id", "device-id"] [[principal]] uid = 997 keys = ["aziot-edged-ca", "aziot-edged-ca-temp"] [[principal]] uid = 994 keys = ["aziot-edged-ca", "iotedge_master_encryption_id", "aziot-edged-ca-temp"] ``` Since there's no local copy of the **edgeAgent** and **edgeHub** containers, I can't see any docker logs from them but I guess there should be enough information. If needed, I could reconfigure the slave to connect directly to the IoT Hub, make sure there's a local copy of the containers locally (I guess I removed them in one of my tests to start from scratch) and try again to get any additional information if they would actually provide it. **UPDATE** Added logs from **edgeAgent** and **edgeHub** in the slave after having switched it back to connect directly to the IoT Hub. With this, as indicated earlier, the containers are created again, but once I configure the node in slave mode, there are no more logs since the connection with the master can't be properly established. What I find more surprising is why **iotedge check --verbose** in the slave indicates that the master's domain name can't be resolved while it's actually trying to connect to it. I know there's no DNS server and I'm using **/etc/hosts** in both cases but the connection is there. **Minor UPDATE** Just in case this helps, I'm using a Raspberry Pi 4 Model B with 8 GB of RAM with this OS (Debian Bullseye 32-bit): [Raspberry OS / armhf](https://downloads.raspberrypi.com/raspios_oldstable_armhf/images/raspios_oldstable_armhf-2024-03-12/) Although this is supposed to be a 32-bit distro, it turns out both 32-bit and 64-bit kernels are included, and it somehow detects which one to use at startup, probably based on your hardware. It took me a while to figure out this and I had to add **arm_64bit=0** to **/boot/config.txt** and restart to enable the 32-bit kernet (the output of **uname -m** is **armv7l**). However, I did this **before** starting the tests, so everything I've reported was ejecuted on a 32-bit kernel (I wanted to make sure this wouldn't be a problem since the 32-bit version of Debian 11 is in Tier 1 while the 64-bit version is only in Tier 2).