search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.41k stars 204 forks source link

Support Regex-based Label \ Metadata matching #926

Open Daniel-Vaz opened 5 months ago

Daniel-Vaz commented 5 months ago

Request

This FR is similar to the following issue - https://github.com/grafana/loki/issues/4987

The loki.process component should allow regex pattern matching on the structured_metadata and pack stages.

Use case

In a Alloy pipeline, when discovering k8s targets I can easily perform a labelmap to have access to all the discoverable __meta labels and keep the ones I need.

Now with these, in a loki.process component, I want to transform them into Loki Structured Metadata fields. This seems to be only possible by specifying a list of static names (Source).

With k8s discovered targets, they tend to have the __meta_kubernetes_*_annotation_* and __meta_kubernetes_*_label_* Labels. These values vary, and knowing all possible values, so that they get specified statically in the stage.structured_metadata block stage of the component doesn't seem feasible. Making me lose some k8s labels and annotations that I want to keep.

akevdmeer commented 5 months ago

You refer to service discovery as supporting flexible relabeling. In fact so does the Alloy component loki.relabel?

But selecting labels to move to structured metadata is currently left to the structured_metadata stage of loki.process that doesn't support regular expressions nor an analogue of labeldrop/labelkeep (which here would mean to move to structured metadata or not).

Could it be possible to move the structured metadata selection to loki.relabel by implementing an extra pair of actions? (Or make the structured_metadata stage of loki.process more expressive, of course.)

github-actions[bot] commented 4 months ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!