Enrich `loki.source` telemetry using `discovery` components

[ptodev]

Background

If you use a component such as loki.source.syslog, you may want to add extra labels to your logs using discovery components such as discovery.http. This is not possible at the moment.

Proposal

One way to do this in the future would be to have a new component which works as follows:

1. Both "discovery" and "log" labels need to have a common label in order to be joined. This can be configurable in the new component, e.g. under a join_label argument. By default this could be set to __address__.
2. The new component persists in memory the latest labels received from discovery.http.
3. When it receives logs from components such as loki.source.syslog, it will:
   1. Check if the logs contain the configured join_label.
   2. If they do, add the latest "discovery" labels to the logs as labels.
   3. If they don't, then passthrough the logs as they were.

A few notes:

• Should join_label be a string or a list? If it's a list, all labels in the list must be the same in the sets of "log" and "discovery" labels.
• The reason we need a new component as opposed to a new standard library function is that the component must be stateful. Standard library functions are stateless.
• Does the fact that the component is stateless has a negative effect on clustering?
• Can the component's memory consumption blow up if it persists too many labels in memory? Should we configure a limit?

[nicolai-hornung-bl]

Hi, I'm really happy this is proposed! I wanted to add that the same functionality would be applicable to metrics, not only logs. E.g. we have some vendor systems which can natively push prometheus / otel but where we can't specify custom label-sets at the source. I would also advocate for the list approach as it's much more flexible. Even better would be some set theory as e.g. you might want to join the label-sets by hostname [OR/AND] by IP.