Closed aleducq closed 2 years ago
Hi @aleducq. Are EdgeHub and EdgeAgent modules running as expected? Can I see iotedge check output?
@aleducq I just synced with a colleague who informed me this issue (along with #6551 too), are related issues and also support tickets opened last week.
Both github issues have the same root cause. Once you correct the missing routes field in the deployment manifest everything should begin operating as expected.
ok @and-rewsmith, maybe we will wait to solve the issue #6551 to be sure but for your information I have this kind of message in edgeAgent:
<6> 2022-08-02 08:17:42.965 +00:00 [INF] - Starting periodic operation refresh twin config...
<6> 2022-08-02 08:17:43.079 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 37 and reported properties version 86.
<6> 2022-08-02 08:17:43.080 +00:00 [INF] - Successfully completed periodic operation refresh twin config
<6> 2022-08-02 08:17:43.427 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2022-08-02 08:17:43.430 +00:00 [INF] - Scraping Metrics
<6> 2022-08-02 08:17:43.434 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2022-08-02 08:17:43.456 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2022-08-02 08:18:27.985 +00:00 [INF] - Error connecting to http://172.18.0.2:9600/metrics with result error code GatewayTimeout
<6> 2022-08-02 08:18:27.986 +00:00 [INF] - Error connecting to http://172.18.0.3:9600/metrics with result error code GatewayTimeout
<6> 2022-08-02 08:18:27.991 +00:00 [INF] - Storing Metrics
<6> 2022-08-02 08:18:28.009 +00:00 [INF] - Scraped and Stored Metrics
<6> 2022-08-02 08:18:28.009 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
Yes let's focus on #6551 and come back to this one.
@and-rewsmith, I still have the issue even if #6551 seems to be solved
Is iotedge check output all green? Any errors from EdgeHub / EdgeAgent logs?
If you don't care about data loss and want to try a further diagnostic step, you can force remove all containers on device and wait for them to be restarted. If the problem persists it will tell us something.
@and-rewsmith, the iotedge check command failed to execute (invalid uri character), I don't know why...
[root@sl24q682 dsmsaiotedgenoprod]# iotedge check
Error while fetching latest versions of edge components: could not create HTTP client
caused by: invalid uri character
For the logs, I've noticed some errors in EdgeAgent:
<6> 2022-08-05 13:21:50.035 +00:00 [INF] - Starting periodic operation Metrics Upload...
<6> 2022-08-05 13:21:50.035 +00:00 [INF] - Uploading Metrics
<4> 2022-08-05 13:21:50.036 +00:00 [WRN] - Error in periodic operation Metrics Upload
System.InvalidOperationException: Sequence contains no elements
at System.Linq.ThrowHelper.ThrowNoElementsException()
at Microsoft.Azure.Devices.Edge.Agent.Diagnostics.Storage.MetricsStorage.GetAllMetricsInternal(CancellationToken cancellationToken)+MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Diagnostics/storage/MetricsStorage.cs:line 51
at Microsoft.Azure.Devices.Edge.Agent.Diagnostics.Storage.MetricsStorage.GetAllMetricsInternal(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|620_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|620_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at Microsoft.Azure.Devices.Edge.Agent.Diagnostics.Storage.MetricsStorage.GetAllMetricsAsync() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Diagnostics/storage/MetricsStorage.cs:line 38
at Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsWorker.TryUploadAndClear(CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Diagnostics/MetricsWorker.cs:line 149
at Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsWorker.Upload(CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Diagnostics/MetricsWorker.cs:line 138
at Microsoft.Azure.Devices.Edge.Util.PeriodicTask.DoWork() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/PeriodicTask.cs:line 106
<6> 2022-08-05 13:32:57.741 +00:00 [INF] - Starting periodic operation Metrics Scrape...
<6> 2022-08-05 13:32:57.741 +00:00 [INF] - Scraping Metrics
<6> 2022-08-05 13:32:57.741 +00:00 [INF] - Scraping endpoint http://edgeHub:9600/metrics
<6> 2022-08-05 13:32:57.742 +00:00 [INF] - Scraping endpoint http://edgeAgent:9600/metrics
<6> 2022-08-05 13:33:42.260 +00:00 [INF] - Error connecting to http://172.18.0.6:9600/metrics with result error code GatewayTimeout
<6> 2022-08-05 13:33:42.260 +00:00 [INF] - Error connecting to http://172.18.0.2:9600/metrics with result error code GatewayTimeout
<6> 2022-08-05 13:33:42.260 +00:00 [INF] - Storing Metrics
<6> 2022-08-05 13:33:42.260 +00:00 [INF] - Scraped and Stored Metrics
<6> 2022-08-05 13:33:42.260 +00:00 [INF] - Successfully completed periodic operation Metrics Scrape
Questions:
Host OS: CentOS Linux release 7.9.2009 (Core) Architecture: Linux 3.10.0-1160.42.2.el7.x86_64 Container OS: Linux containers aziot-edged : iotedge 1.1.5 Edge Agent: azureiotedge-agent:1.1 Edge Hub: azureiotedge-hub:1.1 Docker/Moby: 3.0.6
No errors or warnings except the ones I shared in my previous comment.
I would need to take a closer look at what is going on. Specifically the system module logs for EdgeHub / EdgeAgent. If you send the support bundle to me as an attachment I can take a look (andsmi@microsoft.com).
Please collect 7 days worth: iotedge support-bundle --since "7d"
@aleducq It would be better if you can open support ticket so our support team can help triage the issue.
@aleducq Did you open a support ticket?
Hello @and-rewsmith, I'm back from vacation, I've just opened a ticket. I will let you know
Expected Behavior
Following the documentation https://docs.microsoft.com/en-us/azure/iot-edge/how-to-collect-and-transport-metrics?view=iotedge-2020-11&tabs=iothub I should be able to see my modules in workbook. Trying the Option 1 (with log analytics workspace)
Current Behavior
Steps to Reproduce
Provide a detailed set of steps to reproduce the bug.
Context (Environment)
Device Information
Runtime Versions
iotedge version
]: iotedge 1.1.5docker version
]: 3.0.6Additional Information
If I add modules http://twin:9701/metrics,http://publisher:9702/metrics in MetricsEndpointsCSV, there's no error scraping these endpoints but nothing is coming up to the LAW. From the module I've checked that the ports are open:
I also tried to add the module in deployment (not layered deployment), but the result is the same.