vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.56k stars 1.54k forks source link

Support for `headers` in Loki Sink Configuration #21332

Open LinTechSo opened 5 days ago

LinTechSo commented 5 days ago

Hi

In the current Loki sink configuration in Vector (https://vector.dev/docs/reference/configuration/sinks/loki/), there is no support for specifying a tenant_id_key or custom header fields.

To make Vector more flexible for users running multi-tenant Loki clusters or who need to pass custom headers for authentication/other purposes, I propose adding these two features:

`tenant_id_key`: A configuration option to specify a key in the log event that will be used to dynamically assign the tenant ID key.
    This will enable multi-tenant support in a more flexible way by allowing different tenant IDs to be derived from the log event rather than being statically configured.
`header`:  add extra custom HTTP headers when sending logs to Loki.
    This feature would allow additional headers for specific use cases such as API tokens, custom authentication mechanisms, etc.

I saw a similar configuration for FluentBit: https://docs.fluentbit.io/manual/pipeline/outputs/loki

Use Cases

No response

Attempted Solutions

No response

Proposal

No response

References

No response

Version

0.40.0

jszwedko commented 4 days ago

Hey!

tenant_id is already dynamically configurable via templating: https://vector.dev/docs/reference/configuration/sinks/loki/#tenant_id

For example:

tenant_id: "{{ .some_field }} "

Will use some_field as the tenant id.

Adding in the ability to configure arbitrary HTTP headers seems useful though.