Change integrations job names

Raboo commented 1 month ago

Hi,

Would it be possible to have an option to change the job_name of some or all of the integrations?

ex

integrations/kubernetes/kubelet
integrations/kubernetes/cadvisor
integrations/kubernetes/apiserver
integrations/kubernetes/kube-state-metrics

I would like them to be

kubelet
cadvisor
apiserver
kube-state-metrics

My reason is that i import a bunch of community dashboards into grafana and some of them rely on job label not being prefixed with integrations/kubernetes/

petewall commented 1 month ago

for each of these, you can set a relabel rule that'll change the job label to whatever you want:

metrics:
  kubelet:
    extraMetricRelabelingRules: |
      rule {
        target_label = "job"
        replacement = "kubelet"
      }

Raboo commented 1 month ago

Hi @petewall,

Thank, but that didn't work well. Helm chart broke a bit.

The k8s-monitoring-alloy configmap went from something like this:

apiVersion: v1
data:
  config.alloy: |-
    discovery.kubernetes "nodes" {
      role = "node"
    }

    discovery.kubernetes "services" {
      role = "service"
    }
..... cut ....

Into something like this:

apiVersion: v1
data:
  config.alloy: "discovery.kubernetes \"nodes\" {\n  role = \"node\"\n}\n\ndiscovery.kubernetes
    \"services\" {\n  role = \"service\"\n}\n\ndiscovery.kubernetes \"endpoints\"
    {\n  role = \"endpoints\"\n}\n\ndiscovery.kubernetes \"pods\" {\n  role = \"pod\"\n}\n\n//
    OTLP Receivers\notelcol.receiver.otlp \"receiver\" {\n  grpc {\n    endpoint =
..... cut ....

petewall commented 1 month ago

Is the extraMetricRelabelingRules changes the only changes you make? When kubernetes returns a configmap with \n instead of real newlines, it's sometimes due to extra space at the end of the configmap content: https://github.com/kubernetes/kubernetes/issues/36222

Raboo commented 1 month ago

@petewall Ok I noticed that what makes the break.

Doing extraMetricRelabelingRules: | causes the configmap do everyhing in one line and print \n instead of real newlines. Doing extraMetricRelabelingRules: |-, i.e. |- makes the output properly formated again.

Proper working example:

metrics:
  kube-state-metrics:
    extraMetricRelabelingRules: |-
      rule {
        target_label = "job"
        replacement = "kube-state-metrics"
      }
  apiserver:
    extraMetricRelabelingRules: |-
      rule {
        target_label = "job"
        replacement = "apiserver"
      }

However since I'm on vacation now, I can't test if the job label gets set correctly. But lets assume it's correct.

petewall commented 1 month ago

that's good info to have. I'll make sure that we use |- in all of our docs and maybe add something to troubleshooting.

petewall commented 1 month ago

I'm going to close this issue. When you've returned from vacation, try this out and re-open if it's not working for you!

petewall commented 1 month ago

that's good info to have. I'll make sure that we use |- in all of our docs and maybe add something to troubleshooting.

https://github.com/grafana/k8s-monitoring-helm/pull/674

grafana / k8s-monitoring-helm

Change integrations job names #636