Running alloy for one scrape target resulting in no metrics.

[st-akorotkov]

What's wrong?

I'm trying to run Grafana Alloy for 3-rd party service. Replacing existing Grafana Agent with it. Grafana Agent scrapes target fine but Alloy altough discovering the ServiceMonitor is not sending metrics to Mimir

Steps to reproduce

Replace Agent with Alloy.

System information

No response

Software version

Grafana Alloy 1.4.1 Grafana Agent 0.33.1

Configuration

MetricsInstance for Agent

kind: MetricsInstance
metadata:
  name: exteranl
  namespace: monitoring
  labels:
    agent: grafana-agent-external
spec:
  remoteWrite:
    - url: http://<mimir-url>/api/v1/push
      headers:
        X-Scope-OrgID: <tenant>
      writeRelabelConfigs:
        - sourceLabels:
            - __address__
          targetLabel: k8s_cluster
          replacement: <tenant>
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      grafana-agent: external
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      grafana-agent: external
  probeNamespaceSelector: {}
  probeSelector:
    matchLabels:
      grafana-agent: external

Alloy values for helm chart

controller:
  type: statefulset
  replicas: 1
alloy:
  extraEnv:
    - name: KUBE_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
  configMap:
    content: |
      prometheus.remote_write "primary" {
        external_labels = {
          "k8s_cluster" = "<tenant>",
          "cluster"     = "<tenant>",
          "__replica__" = env("KUBE_POD_NAME"),
        }
        endpoint {
          url = "http://<mimir-url>/api/v1/push"
          headers = {
            "X-Scope-OrgId" = "<tenant>",
          }
          queue_config {
            capacity = 200
            max_shards = 2
          }
        }
      }

      prometheus.operator.podmonitors "primary" {
        forward_to = [prometheus.remote_write.primary.receiver]
        selector {
          match_expression {
            key = "grafana-agent"
            operator = "In"
            values = ["external"]
          }
        }
      }

          prometheus.operator.servicemonitors "primary" {
            forward_to = [prometheus.remote_write.primary.receiver]
            selector {
              match_expression {
                key = "grafana-agent"
                operator = "In"
                values = ["external"]
              }
            }
          }
    crds:
      create: false
    serviceMonitor:
      enabled: true

Logs

No response

[ptodev]

Hi, thank you for submitting a bug report! Are there any errors or warnings in the logs?

[st-akorotkov]

No at all. All components are green in UI.