Azure / meta-iotedge

Yocto layer for Azure IoT Edge
MIT License
50 stars 60 forks source link

Azure IoT Edge won't connect to portal with Kirkstone Update, whereas Dunfell version works just fine. #162

Closed StephenBeckwith-eaton closed 1 month ago

StephenBeckwith-eaton commented 3 months ago

This issue is an “embedded” application using Azure IoT Edge (build on top of Docker support). We have 2 different “build” setups: The working setup was built under the Yocto Dunfell build setup. The non-working setup was built under the Yocto Kirkstone setup (many updated items). It seems related to a connection issue from the container to the network which is blocking progress. Working setup log: DUNFELL:

<6> 2024-05-01 17:03:44.752 +00:00 [INF] - Set metadata metrics: 1.4.34.91223178 (53bd7ab7a450d6a6220174d81499352071af3d97), {“Enabled”:false,“DisableCloudSubscriptions”:false}, {“OperatingSystemType”:“Linux”,“Architecture”:“x86_64”,“Version”:“1.4.10”,“Provisioning”:{“Type”:“ProvisioningType”,“DynamicReprovisioning”:false,“AlwaysReprovisionOnStartup”:false},“ServerVersion”:“19.03.8-ce”,“KernelVersion”:“#1 SMP PREEMPT_DYNAMIC Mon Jun 19 17:58:28 UTC 2023”,“OperatingSystem”:“gateway-iotedge”,“NumCpus”:2,“TotalMemory”:3967021056,“Virtualized”:“no”}, True
<6> 2024-05-01 17:03:44.774 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2024-05-01 17:03:44.777 +00:00 [INF] - Started operation refresh twin config
<6> 2024-05-01 17:03:44.792 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only…
<6> 2024-05-01 17:03:45.122 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2024-05-01 17:03:45.164 +00:00 [INF] - Started operation Metrics Scrape
<6> 2024-05-01 17:03:45.164 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2024-05-01 17:03:45.658 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2024-05-01 17:03:45.658 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2024-05-01 17:03:45.659 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2024-05-01 17:03:45.659 +00:00 [INF] - Registering request handler RestartModule
<6> 2024-05-01 17:03:46.703 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_WebSocket_Only.
<6> 2024-05-01 17:03:46.992 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2024-05-01 17:03:47.106 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 6 and reported properties version 19.
<6> 2024-05-01 17:03:49.169 +00:00 [INF] - Plan execution started for deployment 6

Non-Working log: KIRKSTONE:

<6> 2024-04-25 18:44:15.167 +00:00 [INF] - Set metadata metrics: 1.4.34.91223178 (53bd7ab7a450d6a6220174d81499352071af3d97), {“Enabled”:false,“DisableCloudSubscriptions”:false}, {“OperatingSystemType”:“Linux”,“Architecture”:“x86_64”,“Version”:“1.4.27”,“Provisioning”:{“Type”:“ProvisioningType”,“DynamicReprovisioning”:false,“AlwaysReprovisionOnStartup”:false},“ServerVersion”:“25.0.2-ce”,“KernelVersion”:“#1 SMP PREEMPT_DYNAMIC Wed Nov 8 13:11:05 UTC 2023”,“OperatingSystem”:“gateway-iotedge”,“NumCpus”:4,“TotalMemory”:8188137472,“Virtualized”:“no”}, True
<6> 2024-04-25 18:44:15.187 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2024-04-25 18:44:15.191 +00:00 [INF] - Started operation refresh twin config
<6> 2024-04-25 18:44:15.204 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only…
<6> 2024-04-25 18:44:15.508 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2024-04-25 18:44:15.557 +00:00 [INF] - Started operation Metrics Scrape
<6> 2024-04-25 18:44:15.557 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2024-04-25 18:44:15.940 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2024-04-25 18:44:15.940 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2024-04-25 18:44:15.940 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2024-04-25 18:44:15.940 +00:00 [INF] - Registering request handler RestartModule
<4> 2024-04-25 18:44:21.001 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<6> 2024-04-25 18:44:21.003 +00:00 [INF] - Edge agent config backup file does not exist - /tmp/edgeAgent/backup.json
<4> 2024-04-25 18:44:31.011 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<4> 2024-04-25 18:44:41.012 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead
<6> 2024-04-25 18:44:50.267 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only…

→ and at this point, we are “stuck”. Repeats the “attempting to connect” every 2 minutes.

Dcoker documentation says" Container inherits Host resolv.conf file. This is INCORRECT. The resolv.conf inside the container != host version on BOTH systems. Yet DUNFELL works - why?

Changes to the daemon.json file to change/add configuration information has failed, nothing changes inside the container after a reload/restart after the changes. A suggestion related to some systemd Stub support is set to "no" per the suggestion, still not working. May of the "obvious" item that I've found on-line have been tried and have not resolved the issue.

Some version information is below. More detail can be provided, everything I’ve investigated for Docker, resolve, systemd is NOT working in getting the KIRKSTONE version to work.

Your Insights and suggestions are greatly appreciated.

Warmest regards, Stephen Beckwith

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns="http://www.w3.org/TR/REC-html40">

Kirkstone

Version | 25.0.2-ce -- | -- API Version | 1.44 Go Version | go1.20.7 Git Commit | 5f1b610fc3 Built | Fri Oct 20 21:31:25 2023 OS/Arch | linux/amd64 Context | default Experimental |   containerd |   Version | v1.6.19.m Git Commit | 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f.m runc |   Version | 1.1.12 Git Commit | v1.1.12-0-g51d5e946 docker-init |   Version | 0.19.0 Git Commit | b9f42a0-dirty systemd | systemd 251 (251) resolv.conf (host) | # Generated by NetworkManager   | nameserver 151.110.126.240 resolv.conf (container) | /app # cat /etc/resolv.conf   | nameserver 127.0.0.11   | options ndots:0 iotedge | /initlo/data/aziot # iotedge version   | iotedge 1.4.27 nmcli | /initlo/data/aziot # nmcli --version   | nmcli tool, version 1.36.2

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns="http://www.w3.org/TR/REC-html40">

Dunfell

Version | 19.03.8-ce -- | -- API Version | 1.40 Go Version | go1.16.5 Git Commit | afacb8b7f0 Built | Fri Jun 23 16:59:38 2023 OS/Arch | linux/amd64 Context |   Experimental | FALSE containerd |   Version | v1.2.14-3-g3b3e9d5f6.m Git Commit | 3b3e9d5f62a114153829f9fbe2781d27b0a2ddac.m runc |   Version | 1.0.0-rc8 Git Commit | 425e105d5a03fabd737a126ad93d62a9eeede87f-dirty docker-init |   Version | 0.18.0 Git Commit | fec3683-dirty systemd | 244 (244.5+) resolv.conf (host) | # Generated by NetworkManager   | nameserver 151.110.126.240 resolv.conf (container) | /app # cat /etc/resolv.conf   | nameserver 127.0.0.11   | options ndots:0 iotedge | /initlo/data/aziot # iotedge version   | iotedge 1.4.10 nmcli | /initlo/data/aziot # nmcli --version   | nmcli tool, version 1.22.16

StephenBeckwith-eaton commented 3 months ago

Dumps from "docker inspect".

KS-Inspect_List.txt

DF-Inspect_List.txt

ggjjj commented 1 month ago

we resolved this issue in an ICM as the customer had not enabled the network capability in the kirkstone version. Closing it