Adding extra labels to logs and metrics - what is the best way?

[hobbesuk]

Could someone help if what I'm trying to achieve is the right way to be going about this (and if it is help on the best way to do this). I would like to add some extra labels to the mertics and logs that are collected by the resources in this helm chart and sent to loki and mimir.

I have added some custom labels (with very low cardinality) to the resources in my cluster which I would like to then use in queries in loki and mimir.

I've been messing around with extraMetricRelabelingRules and extraRelabelingRules which look like from the docs what I want but the labels never appear in logs or metrics to then be queried on.

I picked up on some discussions that maybe this chart isn't designed for picking up custom labels, if thats the case is someone able to point me at a good resource on how I should be doing this sort of thing?

TIA!

[petewall]

Hello! There are so many ways to set labels. Are your labels using static values? I.e. you know the value at deploy time? Or, are you labels using values pulled from the workloads themselves?

Static labels are probably best set in the externalLabels section:

externalServices:
  prometheus:
    externalLabels:
      region: northwest

Or, if you're scraping metrics from a workload and want to get a value from that, let me know some more details and I can help out.

[hobbesuk]

these are dynamic labels (e.g. for example which ring the instance comes from), so not a huge number of values but are specific to the pod etc.

I want this so I can then write queries that can be filtered by the ring the instance comes from - maybe I've mis-understood the best way to write queries to do things like this, but when I was running grafana-alloy myself I could achieve this - but this helm chart has many more advantages to me running alloy myself I was hoping to combine the functionality!