grafana / cortex-tools

If you're using this tool with Grafana Mimir, please switch to "mimirtool" instead: https://github.com/grafana/mimir
Apache License 2.0
156 stars 68 forks source link

cortextool not creating or updating existing rules #122

Closed RyanW8 closed 3 years ago

RyanW8 commented 3 years ago

Seems to be that cortextool sync/cortextool load isn't working in my environment. We're running Cortex v1.5.0 and CortexTool v0.5.0.

I've created a new rulegroup in a new NameSpace using the below config:

namespace: collector-rules
groups:
    - name: collector-status
      rules:
        - record: ""
          alert: PrometheuServerIsDown
          expr: absent(up)
          for: 10m
          labels:
            severity: critical
          annotations:
            assignment_group: Site Reliability Engineering
            company: REDACTED
            description: Cortex has not received any metrics from the n4monitoring tenant for 10 minutes
            impact: "1"
            suggested_actions: Check if Prometheus is running in the REDACTED namespace
            summary: 'Cortex has not received metrics for REDACTED for 10minutes'
            urgency: "1"

The NS collector-rules does not exists. cortextool rules load throws an error:

ryan@WINDOWS-H8Q4C40:~/rw170/Documents/cortex-config$ cortextool rules load n4monitoring/rulegroups/collector.yml
ERRO[0000] unable to load rule group                     error="requested resource not found" group=collector-status namespace=collector-rules
cortextool: error: load operation unsuccessful, try --help

cortextool rules sync --rule-dirs=<dir> also throws an error:

ryan@WINDOWS-H8Q4C40:~/rw170/Documents/cortex-config$ cortextool rules sync --rule-dirs=n4monitoring/rulegroups
INFO[0000] creating group                                group=collector-status namespace=collector-rules
cortextool: error: sync operation unsuccessful, unable to complete executing changes.: requested resource not found, try --help

I've got a feeling it's potentially related to our ingress rules but I can't spot anything here is the ingress:

spec:
  rules:
  - host: REDACTED
    http:
      paths:
      - backend:
          serviceName: alertmanager
          servicePort: 80
        path: /multitenant_alertmanager/status
      - backend:
          serviceName: alertmanager
          servicePort: 80
        path: /alertmanager
      - backend:
          serviceName: alertmanager
          servicePort: 80
        path: /api/v1/alerts
      - backend:
          serviceName: ruler
          servicePort: 80
        path: /ruler/ring
      - backend:
          serviceName: ruler
          servicePort: 80
        path: /api/v1/rules
      - backend:
          serviceName: ruler
          servicePort: 80
        path: /api/prom/api/v1/alerts
      - backend:
          serviceName: ruler
          servicePort: 80
        path: /api/prom/rules
      - backend:
          serviceName: distributor
          servicePort: 80
        path: /distributor/all_user_stats
gotjosh commented 3 years ago

@RyanW8 IIRC this was related to cortextool not being able to handle trailing /, I've fixed that in the most recent release. I'm going to close this one for now but please do let me know if there's anything we can do.