Replace Grafana-Agent Rules with Alloy

[Rotfuks]

Motivation

We want to replace Grafana Agents everywhere with GrafanaLabs new OpenTelemetry Agent Alloy. Our Grafana Agent which is just for importing prometheus-rules into Mimir is a good starting point to get first experience with Alloy.

Todo

• [x] Replace grafana agent rules with alloy
• [x] Make sure mimir still works as expected

Outcome

• We get rid of the grafana agent used for prometheus-rules import and have alloy and collect first experience with which will be our single agent for everything in the future.

[TheoBrigitte]

How to send import PrometheusRules into Mimir using Alloy

Assuming that you already have Mimir installed.

1. Install Alloy

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   kubectl create namespace alloy
   kubectl config set-context --current --namespace alloy
   helm install --namespace alloy alloy grafana/alloy

2. Configure Alloy to import rules into Mimir

   
   cat <<EOF > rules-alloy
   mimir.rules.kubernetes "local" {
   address = "http://mimir-gateway.mimir"
   tenant_id = "tenant-1"
   
   rule_namespace_selector {
       match_labels = {
           "kubernetes.io/metadata.name" = "alloy",
       }
   }
   
   rule_selector {
       match_labels = {
           "foo" = "bar",
       }
   }
   }
   EOF

cat < values.yaml alloy: configMap: create: true name: alloy-config key: alloy.config EOF

kubectl create configmap alloy-config "--from-file=alloy.config=./rules-alloy" helm upgrade -n alloy alloy grafana/alloy -f values.yaml

3. Create a PrometheusRule

```yaml
cat <<EOF > promrule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    foo: bar
  name: import-me
spec:
  groups:
  - name: dummy_group
    rules:
    - expr: dummy_expr
      record: dummy_record
EOF

kubectl apply -f promrule.yaml

4. Verify that rules are correctly imported into Mimir

   $ kubectl -namespace mimir port-forward svc/mimir-ruler 8080
   $ mimirtool rules list --address='http://127.0.0.1:8080' --id=tenant-1
   Namespace                                                  | Rule Group
   alloy/alloy/import-me/fe097cad-fe46-49eb-af70-0f6e38b7dda9 | dummy_group

[TheoBrigitte]

• Created a new Alloy app repository https://github.com/giantswarm/alloy-app
• Replace Grafana Agent with Alloy in prometheus-rules repository https://github.com/giantswarm/prometheus-rules/blob/main/helm/prometheus-rules/templates/alloy-rules.yaml
• Updated the Alerting rule to notify when Alloy is down
• Tested rules import in Mimir still works