grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.42k stars 210 forks source link

Enabling clustering for prometheus.operator.servicemonitors stops metrics delivery #1870

Open st-akorotkov opened 1 month ago

st-akorotkov commented 1 month ago

What's wrong?

I'm trying to run clustered alloy in our kubernetes clusters, replacing Grafana Agent. Then I'm enabling clustering in prometheus.operator.servicemonitors metrics are not delivered in Mimir anymore

Steps to reproduce

enable clustering in prometheus.operator.servicemonitors

System information

No response

Software version

1.4.1

Configuration

Deployed via helm chart with following values:

controller:
  type: statefulset
  replicas: 5
alloy:
  extraEnv:
    - name: KUBE_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
  configMap:
    content: |
      prometheus.remote_write "primary" {
        external_labels = {
          "k8s_cluster" = "<tenant>",
          "cluster"     = "<tenant>",
          "__replica__" = env("KUBE_POD_NAME"),
        }
        endpoint {
          url = "http://<mimir-ulr>/api/v1/push"
          headers = {
            "X-Scope-OrgId" = "<tenant>",
          }
          write_relabel_config {
            action = "drop"
            source_labels = ["__name__"]
            regex = "^st_Marketing_Emails_PlatformEmailAdapter.*|^st_Login_Oidc.*"
          }
        }
      }
      prometheus.operator.podmonitors "primary" {
        forward_to = [prometheus.remote_write.primary.receiver]
      }
      prometheus.operator.servicemonitors "primary" {
        forward_to = [prometheus.remote_write.primary.receiver]
      }

  clustering:
    enabled: true
    name: grafana-alloy-main
crds:
  create: false
serviceMonitor:
  enabled: true

Logs

level=error msg="error running crd manager" component_path=/ component_id=prometheus.operator.servicemonitors.primary err="creating and registering service discovery metrics: failed to register service discovery metrics: failed to create service discovery refresh metrics"

noe-charmet commented 5 days ago

Encountering the same issue here