gordonwang0
commented
3 years ago Please provide the relevant logs from iotedged. The iotedged logs you posted don't match the timestamps in edgeAgent.
Open sabrioezer opened 3 years ago
gordonwang0
commented
3 years ago Please provide the relevant logs from iotedged. The iotedged logs you posted don't match the timestamps in edgeAgent.
derSchtefan
commented
3 years ago @sabrioezer Have your considered updating to iotedged 1.1.6 and specifying edgeAgent and edgeHub as 1.1.4?
github-actions[bot]
commented
3 years ago This issue is being marked as stale because it has been open for 30 days with no activity.
murali513
commented
2 years ago I am also facing this issue on RSB 3720, please let me know what will be the reason for this. deployed Azure IoT Edge 1.2 version on Device Trying to deployed modules (edgeAgent & edgeHub) version 1.1 from cloud IoT Hub
Could not check current state of edgeHub container
caused by: docker returned exit code: 1, stderr = Error: No such object: edgeHub√ Agent image is valid and can be pulled from upstream - OK √ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK
× container on the default network can connect to upstream AMQP port - Error Container on the default network could not connect to iothubmp.azure-devices.net:5671
Environment:
errorondeployingiotedge.txt
Expected Behavior
EdgeHub should be created and running. We had this problem since yesterday. There was a restart probably which caused a redeployement and we receive this error since then.
Current Behavior
2021-09-22 09:40:39.668 +00:00 [ERR] - Executing command for operation ["update Edge Agent"] failed.
2021-09-22 09:40:39.685 +00:00 [INF] - Executing command: "Create module edgeHub" 2021-09-22 09:42:19.696 +00:00 [ERR] - Executing command for operation ["create"] failed.
Steps to Reproduce
Context (Environment)
Output of
iotedge checkClick here
``` √ config.yaml is well-formed - OK √ config.yaml has well-formed connection string - OK √ container engine is installed and functional - OK √ config.yaml has correct hostname - OK √ config.yaml has correct URIs for daemon mgmt endpoint - OK ‼ latest security daemon - Warning Installed IoT Edge daemon has version 1.0.10.4 but 1.1.6 is the latest stable version available. Please see https://aka.ms/iotedge-update-runtime for update instructions. √ host time is close to real time - OK √ container time is close to host time - OK ‼ DNS server - Warning Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub. Please see https://aka.ms/iotedge-prod-checklist-dns for best practices. You can ignore this warning if you are setting DNS server per module in the Edge deployment. ‼ production readiness: certificates - Warning The Edge device is using self-signed automatically-generated development certificates. They will expire in 89 days (at 2021-12-21 07:08:29 UTC) causing module-to-module and downstream device communication to fail on an active deployment. After the certs have expired, restarting the IoT Edge daemon will trigger it to generate new development certs. Please consider using production certificates instead. See https://aka.ms/iotedge-prod-checklist-certs for best practices. √ production readiness: container engine - OK ‼ production readiness: logs policy - Warning Container engine is not configured to rotate module logs which may cause it run out of disk space. Please see https://aka.ms/iotedge-prod-checklist-logs for best practices. You can ignore this warning if you are setting log policy per module in the Edge deployment. ‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem. Data might be lost if the module is deleted or updated. Please see https://aka.ms/iotedge-storage-host for best practices. × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error Could not check current state of edgeHub container Connectivity checks ------------------- √ host can connect to and perform TLS handshake with DPS endpoint - OK √ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK √ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK √ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK √ container on the default network can connect to IoT Hub AMQP port - OK √ container on the default network can connect to IoT Hub HTTPS / WebSockets port - OK √ container on the default network can connect to IoT Hub MQTT port - OK √ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK √ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK √ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK 18 check(s) succeeded. 5 check(s) raised warnings. Re-run with --verbose for more details. 1 check(s) raised errors. Re-run with --verbose for more details. ```Device Information
Runtime Versions
iotedge version]: iotedge 1.0.10.4docker version]: 19.03.14+azureNote: when using Windows containers on Windows, run
docker -H npipe:////./pipe/iotedge_moby_engine versioninsteadLogs
aziot-edged logs
``` --sudo systemctl status iotedge-- iotedge.service - Azure IoT Edge daemon Loaded: loaded (/lib/systemd/system/iotedge.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-09-22 09:38:16 UTC; 29min ago Docs: man:iotedged(8) Main PID: 15444 (iotedged) Tasks: 70 (limit: 4915) CGroup: /system.slice/iotedge.service └─15444 /usr/bin/iotedged -c /etc/iotedge/config.yaml Sep 22 10:07:04 hymon-000018 iotedged[15444]: 2021-09-22T10:07:04Z [INFO] - [mgmt] - - - [2021-09-22 10:07:04.619139839 UTC] "PUT /identities/%24edgeHub?api-version=2020-07-07 HTTP/1.1" 200 OK 98 "-" "-" auth_id(-) Sep 22 10:07:04 hymon-000018 iotedged[15444]: 2021-09-22T10:07:04Z [INFO] - [work] - - - [2021-09-22 10:07:04.646923610 UTC] "POST /modules/%24edgeAgent/genid/637475083364206614/encrypt?api-version=2020-07-07 HTTP/1.1" 200 OK 893 "-" "-" Sep 22 10:07:09 hymon-000018 iotedged[15444]: 2021-09-22T10:07:09Z [INFO] - [mgmt] - - - [2021-09-22 10:07:09.651932486 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 528 "-" "-" auth_id(-) Sep 22 10:07:10 hymon-000018 iotedged[15444]: 2021-09-22T10:07:10Z [INFO] - [mgmt] - - - [2021-09-22 10:07:10.924742519 UTC] "GET /identities/?api-version=2020-07-07 HTTP/1.1" 200 OK 719 "-" "-" auth_id(-) Sep 22 10:07:11 hymon-000018 iotedged[15444]: 2021-09-22T10:07:11Z [INFO] - [mgmt] - - - [2021-09-22 10:07:11.999517449 UTC] "PUT /identities/%24edgeHub?api-version=2020-07-07 HTTP/1.1" 200 OK 98 "-" "-" auth_id(-) Sep 22 10:07:12 hymon-000018 iotedged[15444]: 2021-09-22T10:07:12Z [INFO] - [work] - - - [2021-09-22 10:07:12.047795673 UTC] "POST /modules/%24edgeAgent/genid/637475083364206614/encrypt?api-version=2020-07-07 HTTP/1.1" 200 OK 893 "-" "-" Sep 22 10:07:17 hymon-000018 iotedged[15444]: 2021-09-22T10:07:17Z [INFO] - [mgmt] - - - [2021-09-22 10:07:17.050529882 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 528 "-" "-" auth_id(-) Sep 22 10:07:18 hymon-000018 iotedged[15444]: 2021-09-22T10:07:18Z [INFO] - [mgmt] - - - [2021-09-22 10:07:18.325338721 UTC] "GET /identities/?api-version=2020-07-07 HTTP/1.1" 200 OK 719 "-" "-" auth_id(-) Sep 22 10:07:19 hymon-000018 iotedged[15444]: 2021-09-22T10:07:19Z [INFO] - [mgmt] - - - [2021-09-22 10:07:19.447729869 UTC] "PUT /identities/%24edgeHub?api-version=2020-07-07 HTTP/1.1" 200 OK 98 "-" "-" auth_id(-) Sep 22 10:07:19 hymon-000018 iotedged[15444]: 2021-09-22T10:07:19Z [INFO] - [work] - - - [2021-09-22 10:07:19.496108371 UTC] "POST /modules/%24edgeAgent/genid/637475083364206614/encrypt?api-version=2020-07-07 HTTP/1.1" 200 OK 893 "-" "-" ```edge-agent logs
``` 2021-09-22 09:38:43 Starting Edge Agent 2021-09-22 09:38:43 Changing ownership of management socket: /var/run/iotedge/mgmt.sock 2021-09-22 09:38:43.572 +00:00 Edge Agent Main() <6> 2021-09-22 09:38:43.707 +00:00 [INF] - Initializing Edge Agent. <6> 2021-09-22 09:38:43.807 +00:00 [INF] - Version - 1.0.10.4.37804714 (57772714c81c8b823a5ef05bf11bf343b923fb6a) <6> 2021-09-22 09:38:43.808 +00:00 [INF] - █████╗ ███████╗██╗ ██╗██████╗ ███████╗ ██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝ ███████║ ███╔╝ ██║ ██║██████╔╝█████╗ ██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝ ██║ ██║███████╗╚██████╔╝██║ ██║███████╗ ╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝ ██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗ ██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝ ██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗ ██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝ ██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗ ╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝ <6> 2021-09-22 09:38:43.836 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false} <6> 2021-09-22 09:38:43.923 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:12/21/2021 07:08:29] to Root <6> 2021-09-22 09:38:44.022 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics <6> 2021-09-22 09:38:44.116 +00:00 [INF] - Updating performance metrics every 05m:00s <6> 2021-09-22 09:38:44.119 +00:00 [INF] - Started operation Get system resources <6> 2021-09-22 09:38:44.121 +00:00 [INF] - Collecting metadata metrics <6> 2021-09-22 09:38:44.183 +00:00 [INF] - Set metadata metrics: 1.0.10.4.37804714 (57772714c81c8b823a5ef05bf11bf343b923fb6a), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"linux","Architecture":"x86_64","Version":"1.0.10.4","Provisioning":{"Type":"dps.tpm","DynamicReprovisioning":false},"ServerVersion":"19.03.14+azure","KernelVersion":"4.15.0-135-generic","OperatingSystem":"Ubuntu 18.04.5 LTS","NumCpus":32,"Virtualized":"no"}, True <6> 2021-09-22 09:38:44.197 +00:00 [INF] - Started operation Checkpoint Availability <6> 2021-09-22 09:38:44.199 +00:00 [INF] - Started operation refresh twin config <6> 2021-09-22 09:38:44.222 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only... <6> 2021-09-22 09:38:44.429 +00:00 [INF] - Created persistent store at /tmp/edgeAgent <6> 2021-09-22 09:38:44.466 +00:00 [INF] - Started operation Metrics Scrape <6> 2021-09-22 09:38:44.466 +00:00 [INF] - Started operation Metrics Upload Scraping frequency: 01:00:00 Upload Frequency: 1.00:00:00 <6> 2021-09-22 09:38:44.627 +00:00 [INF] - Registering request handler UploadModuleLogs <6> 2021-09-22 09:38:44.627 +00:00 [INF] - Registering request handler GetModuleLogs <6> 2021-09-22 09:38:44.627 +00:00 [INF] - Registering request handler UploadSupportBundle <6> 2021-09-22 09:38:44.627 +00:00 [INF] - Registering request handler RestartModule <4> 2021-09-22 09:38:49.675 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <6> 2021-09-22 09:38:49.679 +00:00 [INF] - Edge agent config backup file does not exist - /tmp/edgeAgent/backup.json <6> 2021-09-22 09:38:53.882 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only. <6> 2021-09-22 09:38:56.215 +00:00 [INF] - Initialized new module client with subscriptions enabled <6> 2021-09-22 09:38:57.283 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 36 and reported properties version 326. <6> 2021-09-22 09:38:59.623 +00:00 [INF] - Plan execution started for deployment 36 <6> 2021-09-22 09:38:59.641 +00:00 [INF] - Executing command: "Command Group: (\n [Prepare update module edgeAgent]\n [[Null]]\n [Update and start module edgeAgent]\n)" <3> 2021-09-22 09:40:39.668 +00:00 [ERR] - Executing command for operation ["update Edge Agent"] failed. System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.CancellationToken.ThrowIfCancellationRequested() at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.IO.BufferedStream.ReadFromUnderlyingStreamAsync(Memory`1 buffer, CancellationToken cancellationToken, Int32 bytesAlreadySatisfied, Task semaphoreLockTask) at Microsoft.Azure.Devices.Edge.Util.Uds.HttpBufferedStream.ReadLineAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpBufferedStream.cs:line 62 at Microsoft.Azure.Devices.Edge.Util.Uds.HttpRequestResponseSerializer.SetResponseStatusLine(HttpResponseMessage httpResponse, HttpBufferedStream bufferedStream, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpRequestResponseSerializer.cs:line 131 at Microsoft.Azure.Devices.Edge.Util.Uds.HttpRequestResponseSerializer.DeserializeResponse(HttpBufferedStream bufferedStream, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpRequestResponseSerializer.cs:line 66 at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 36 at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.GeneratedCode.EdgeletHttpClient.PrepareUpdateModuleAsync(String api_version, String name, ModuleSpec module, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/generatedCode/EdgeletHttpClient.cs:line 519 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.<>c__DisplayClass34_0.<edge-hub logs
```Additional Information
Please provide any additional information that may be helpful in understanding the issue.