Loki does not trigger or push alerts on alertmanager

[mradspieler]

Describe the issue The issue I am facing is that the Loki does not trigger or push alerts on alertmanager. I cannot see any alert on the AlertManager dashboard though I can run a LogQL query on Grafana which shows the condition was met for triggering an alert. Grafana alerts works fine and triggers an alert.

To Reproduce Steps to reproduce the behavior:

1. Started Loki 2.2.1 (loki-linux-amd64 -config.file /home/test/grafana-loki/loki-config.yml) loki-config.yml

server:
  http_listen_port: 3100
ruler:
  storage:
    type: local
    local:
      directory: /home/test/grafana-loki/rules2 # directory include 2 rule files
  rule_path: /tmp/scratch  # directory is always empty
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  # enable_alertmanager_v2: true

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
...

2. Started alertmanager (./alertmanager --config.file=alertmanager.yml &) alertmanager.yml

   
   resolve_timeout: 5m
   # The smarthost and SMTP sender used for mail notifications.
   smtp_smarthost: 'xxx'
   smtp_from: 'xxx'
   smtp_auth_username: 'xxx'
   smtp_auth_password: ''
   smtp_require_tls: false

route: receiver: 'admin-mails' group_wait: 30s group_interval: 5m repeat_interval: 3h

receivers:

• name: 'admin-mails' email_configs:
  • to: 'xxxx'

    3. Rule file 

    groups:

  • name: error-logs rules:
    • alert: ErrorInLogs expr: sum(rate({container_name="container_1"} |= "milter" [1m])) by (compose_project) > 0.01 for: 1m labels: severity: warning

      
      **Expected behavior**
      loki should send an alert.

Environment: Grafana - 7.5, Loki - 2.2, AlertManager - 0.21

Log file level=info ts=2021-04-15T13:18:37.152885779Z caller=main.go:130 msg="Starting Loki" version="(version=2.2.0, branch=HEAD, revision=a27c7991)" level=info ts=2021-04-15T13:18:37.153537484Z caller=server.go:229 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses" level=info ts=2021-04-15T13:18:37.15938588Z caller=mapper.go:38 msg="cleaning up mapped rules directory" path=/tmp/scratch level=info ts=2021-04-15T13:18:37.161473641Z caller=module_service.go:59 msg=initialising module=server level=info ts=2021-04-15T13:18:37.161516342Z caller=module_service.go:59 msg=initialising module=memberlist-kv level=info ts=2021-04-15T13:18:37.161549682Z caller=module_service.go:59 msg=initialising module=store level=info ts=2021-04-15T13:18:37.161669405Z caller=module_service.go:59 msg=initialising module=table-manager level=info ts=2021-04-15T13:18:37.161981756Z caller=module_service.go:59 msg=initialising module=ingester level=info ts=2021-04-15T13:18:37.162176072Z caller=lifecycler.go:521 msg="not loading tokens from file, tokens file path is empty" level=info ts=2021-04-15T13:18:37.162180641Z caller=table_manager.go:324 msg="synching tables" expected_tables=4 level=info ts=2021-04-15T13:18:37.162305812Z caller=lifecycler.go:550 msg="instance not found in ring, adding with no tokens" ring=ingester level=info ts=2021-04-15T13:18:37.162577756Z caller=lifecycler.go:397 msg="auto-joining cluster after timeout" ring=ingester level=info ts=2021-04-15T13:18:37.162602477Z caller=module_service.go:59 msg=initialising module=ring level=info ts=2021-04-15T13:18:37.162927906Z caller=module_service.go:59 msg=initialising module=ingester-querier level=info ts=2021-04-15T13:18:37.162962701Z caller=module_service.go:59 msg=initialising module=distributor level=info ts=2021-04-15T13:18:37.16302309Z caller=module_service.go:59 msg=initialising module=ruler level=info ts=2021-04-15T13:18:37.163072666Z caller=ruler.go:403 msg="ruler up and running" level=info ts=2021-04-15T13:18:37.163100541Z caller=loki.go:248 msg="Loki started"

similar problem as here: https://stackoverflow.com/questions/66780585/grafana-loki-does-not-trigger-or-push-alert-on-alertmanager

[mradspieler]

If Loki is running in single tenant mode, the required ID is fake (yes we know this might seem alarming but it’s totally fine, no it can’t be changed).

ruler:
  storage:
    type: local
    local:
      directory: /home/test/grafana-loki/rules2
    rule_path: /tmp/scratch/fake

mkdir /home/test/grafana-loki/rules2/fake mkdir /tmp/scratch/fake

copy your rule files into /home/test/grafana-loki/rules2/fake

So you have to add a fake sub-directory to the rule directory in single tenant mode and everthing worked perfectly.