Getting "the Alertmanager is not configured" when trying to use AlertManager

cpressland commented 2 years ago

Describe the bug

When deploying a fresh Grafana Mimir instance via Helm on a new Kubernetes Cluster we receive failed healthchecks when trying to connect Grafana to mimir-alertmanager. Trying to communicate with the APIs itself we get the following:

$ curl http://127.0.0.1:8080/alertmanager/api/v1/rules
the Alertmanager is not configured
$ curl -XPOST http://127.0.0.1:8080/alertmanager/api/v1/rules
route not supported by distributor

This feature appears to be broken out of the box, or, documentation appears to be lacking on how to configure this correctly.

To Reproduce

Steps to reproduce the behavior:

Deploy Mimir on a new Kubernetes Cluster with the following config via Helm:

  values:
    minio:
      enabled: false
    distributor:
      replicas: 3
    ingester:
      persistentVolume:
        size: 100Gi
    nginx:
      replicas: 3

    alertmanager:
      enabled: true
      readinessProbe:
        httpGet:
          path: /ready
          port: http-metrics
        initialDelaySeconds: 45

    mimir:
      config: |-
        multitenancy_enabled: false
        limits:
          max_global_series_per_user: 1000000
          max_global_series_per_metric: 1000000
          ingestion_rate: 100000
        activity_tracker:
          filepath: /data/metrics-activity.log
        alertmanager:
          data_dir: '/data'
          enable_api: true
          external_url: '/alertmanager'
        alertmanager_storage:
          backend: azure
          azure:
            account_name: <snipped>
            account_key: <snipped>
            container_name: mimir-alertmanager
        frontend_worker:
          frontend_address: {{ template "mimir.fullname" . }}-query-frontend-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverGrpcListenPort" . }}
        ruler:
          enable_api: true
          rule_path: '/data'
          alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}/alertmanager
        server:
          grpc_server_max_recv_msg_size: 104857600
          grpc_server_max_send_msg_size: 104857600
          grpc_server_max_concurrent_streams: 1000
        frontend:
          log_queries_longer_than: 10s
          align_queries_with_step: true
        compactor:
          data_dir: "/data"
        ingester:
          ring:
            final_sleep: 0s
            num_tokens: 512
        ingester_client:
          grpc_client_config:
            max_recv_msg_size: 104857600
            max_send_msg_size: 104857600
        runtime_config:
          file: /var/{{ include "mimir.name" . }}/runtime.yaml
        memberlist:
          abort_if_cluster_join_fails: false
          compression_enabled: false
          join_members:
          - {{ include "mimir.fullname" . }}-gossip-ring
        blocks_storage:
          backend: azure
          tsdb:
            dir: /data/tsdb
          bucket_store:
            sync_dir: /data/tsdb-sync
          azure:
            account_name: <snipped>
            account_key: <snipped>
            container_name: mimir-blocks
        ruler_storage:
          backend: azure
          azure:
            account_name: <snipped>
            account_key: <snipped>
            container_name: mimir-admin

Port-Forward to port 8080 on the mimir-alertmanager-0 pod
Attempt to GET or POST rules to the alertmanager API

Expected behavior

We expect alert-manager to become configured and usable after the first POST, and then be able to configure this further via Grafana.

Environment

Infrastructure: Kubernetes
Deployment tool: Helm via Flux 2

Additional Context

Full Disclosure, we are still trying to learn the Mimir product and may simply be getting the wrong end of the stick here. A Grafana Community for Mimir does not appear to exist, so we're not really sure where to get support and ask questions. I'd rather have avoided raising a GitHub issue, but, here we are.

Our requirements are simply to fire alerts to various third-party services such as OpsGenie, Microsoft Teams, etc. So it may be that we're trying to use a "prometheus-esque" approach by shoving alert-manager in where it may not be required. Any additional advice would be appreciated here.

pracucci commented 2 years ago

This issue is related to https://github.com/grafana/mimir/issues/2772. The Grafana issue should have been resolved by https://github.com/grafana/grafana/issues/49728 (will be included in the next Grafana version).

As a workaround, I suggest you to create a basic Alertmanager configuration .yaml file, store it on the local disk, and specify its path to the Mimir configuration parameter -alertmanager.configs.fallback, so that it will be used by default for any tenant for which no Alertmanager configuration has been uploaded yet.

same7ammar commented 2 years ago

Thanks @pracucci for sorting this issue out . I was facing the same issue and I fix it using minirtool or sending config direct via Alertmanager API endpoint.

Alertmanager config sample : alertmanager-config.yaml

route:
  receiver: "example_receiver"
  group_by: ["example_groupby"]
receivers:
  - name: "example_receiver"

 k port-forward svc/mimir-distributed-alertmanager 8080:8080

send a request to update Alertmanager config as below

curl -X POST http://localhost:8080/api/v1/alerts -H 'Content-Type: application/json' -H 'X-Scope-OrgID: monitoring ' --data-binary @alertmanager-config.yaml

if you're using multi-tenant , Please make sure to change tenant to correct tenant id . if you're not using multi-tenant use X-Scope-OrgID: anonymous .

using mimirtool to update alertmanager .


mimirtool alertmanager get  --address=http://localhost:8080  --id=anonymous

mimirtool alertmanager load alertmanager-config.yaml  --address=http://localhost:8080 --id=anonymous

gotjosh commented 1 year ago

@pracucci should we also close this one? As it related to https://github.com/grafana/mimir/issues/2772

grafana / mimir