Azure / iotedge

The IoT Edge OSS project
MIT License
1.47k stars 462 forks source link

[MetricsCollector] Label/tag scrapped sources/modules if they don't provide tags. #6081

Closed alaendle closed 6 months ago

alaendle commented 2 years ago

For own modules it isn't a problem to evaluate the environment variables and expose tagged metrics like:

metric_name{edge_device="<device>",iothub="<iothub>",source="<module>"} 0.0

However 3rd party applications running on the edge aren't aware of their environment and may expose something like:

nginx_connections_accepted 54

If the metrics collector now transfers metrics from multiple devices or instances the metrics are no longer assignable in the cloud.

Since the metrics collector is aware of it's environment I suggest that it adds tags for edge_device, iothub and source if they are not already present.

I hope the problem is obvious. So I keep the description short; but please don't hesitate to contact me if I was unclear, could provide further explanations or could contribute in any way! Thanks in advance for your support.

alaendle commented 2 years ago

Small addendum: I thought I could workaround my problem by using the old "additionalTags" device-twin feature - until I realized that it was removed (https://github.com/Azure/iotedge/pull/4950). However, even if I would be able to specify additional tags this would be a stopgap solution. It would be error prone (need to be adjusted for every module, device!) and wouldn't scale well (only one scrapping target per collector). So my basic request is that container that run inside the azure iot edge but aren't aware of the edge context and provide prometheus metrics (think of something like rabbitmq, nginx, etc.) could be integrated. Since the collector is already aware of the edge_device and the iothub and a module identifier like sourceor module could be derived from the hostname of the scrapped endpoint, I really believe that the collector should add these tags to support such scenarios.

veyalla commented 2 years ago

Thanks @alaendle, please submit this suggestion at https://aka.ms/iotedge-suggestion. At this time, there is no plan to add this functionality. But we review community feedback regularly and will consider for future implementation if the suggestion receives community support via upvotes.

In the meantime, the source code for metrics collector lives here, and we are open to PRs if you are able and willing :)

alaendle commented 2 years ago

Submitted as suggestion: https://feedback.azure.com/d365community/idea/c5cdb9ba-398a-ec11-a81b-0022484bfd94 And already created a simple PR targeting a small imperfection as a test balloon - https://github.com/Azure/iotedge/pull/6113 - to see how contributions are handled. I will take a further look at the code and see how easy it is to add the required functionality - then I would open another PR (or not). (@veyalla)

alaendle commented 2 years ago

BTW, I haven't found any standard for the naming of the labels (https://github.com/Azure-Samples/iotedge-module-prom-custom-metrics/discussions/2) so I used the same as the edgeAgent (edge_device, iothub and module_name) in the PR.

alaendle commented 2 years ago

Small update: Unfortunately the current metrics collector release (v1.0.8) which should offer a solution to this problem contains a small bug - I hope it gets fixed with https://github.com/Azure/iotedge/pull/6422.