gmauleon / alertmanager-zabbix-provisioner

Zabbix provisioner to automatically create host/items/triggers from Prometheus rules
13 stars 12 forks source link

[Bug] Items are not created #6

Closed paskal closed 5 years ago

paskal commented 5 years ago

I have Zabbix 3.4.12 and Prometheus 2.7.1 and can't get items and triggers created in Zabbix. Configuration:

$ kubectl exec -it prometheus-zabbix-webhook-78fc67bffc-6xr8p -c alertmanager-zabbix-provisioner -- /bin/sh -c 'cat /etc/provisioner/config.yaml'
# URL to the Status/Rules page
rulesUrl: http://prometheus-operator-prometheus.monitoring.svc.cluster.local:9090/rules

# Polling interval in seconds
rulesPollingInterval: 3600

# Full URL to your Zabbix API
zabbixApiUrl: https://zabbix.some.host/zabbix/api_jsonrpc.php

# Zabbix items key prefix, keys will be zabbixKeyPrefix.alertname
zabbixKeyPrefix: prometheus

# List of host configuration
zabbixHosts:
  # Name of the host in zabbix
  - name: gmauleon-test01
    selector:
      zabbix: gmauleon-test01
    hostGroups:
      - some_group
    itemDefaultApplication: prometheus

logs:

$ kubectl logs prometheus-zabbix-webhook-78fc67bffc-6xr8p -c alertmanager-zabbix-provisioner
time="2019-02-26T13:36:06Z" level=info msg="loading configuration at '/etc/provisioner/config.yaml'"
time="2019-02-26T13:36:06Z" level=info msg="configuration loaded"
time="2019-02-26T13:36:06Z" level=debug msg="&{http://prometheus-operator-prometheus.monitoring.svc.cluster.local:9090/rules 3600 https://zabbix.some.host/zabbix/api_jsonrpc.php  bsw-prometheus-zabbix-bot aco8De0die7goohu prometheus [{gmauleon-test01 map[zabbix_host:gmauleon-test01] [some_group]   prometheus   }]}"
time="2019-02-26T13:36:07Z" level=debug msg="Host from Prometheus: &{State:0 Host:{HostId: Host:gmauleon-test01 Available:1 Error: Name:gmauleon-test01 Status:0 InventoryMode:0 Inventory:map[deployment_status: tag:] GroupIds:[] Interfaces:[{DNS: IP:127.0.0.1 Main:1 Port:10050 Type:1 UseIP:1}]} HostGroups:map[some_group:{}] Applications:map[] Items:map[] Triggers:map[]}"
time="2019-02-26T13:36:07Z" level=info msg="GetHostGroupsByState = State: Equal, Name: some_group"
time="2019-02-26T13:36:07Z" level=info msg="GetHostByState = State: New, Name: gmauleon-test01"
time="2019-02-26T13:36:07Z" level=debug msg="Creating Hosts: [{HostId: Host:gmauleon-test01 Available:1 Error: Name:gmauleon-test01 Status:0 InventoryMode:0 Inventory:map[deployment_status: tag:] GroupIds:[{GroupId:102} {GroupId:422}] Interfaces:[{DNS: IP:127.0.0.1 Main:1 Port:10050 Type:1 UseIP:1}]}]\n"
time="2019-02-26T13:36:07Z" level=info msg="Updating host: gmauleon-test01"

Configuration for alert in Prometheus:

- alert: KubeStatefulSetReplicasMismatch
  annotations:
    zabbix: gmauleon-test01
    message: StatefulSet {{`{{ $labels.namespace }}`}}/{{`{{ $labels.statefulset }}`}} has not matched the expected number of replicas for longer than 15 minutes.
    runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch
  expr: |-
    kube_statefulset_status_replicas_ready{job="kube-state-metrics"}
      !=
    kube_statefulset_status_replicas{job="kube-state-metrics"}
  for: 15m
  labels:
    severity: critical

I do see this alert at prometheus:9090/rules page:

image

What I expect to see Host gmauleon-test01 in Zabbix at least one item and trigger for KubeStatefulSetReplicasMismatch alert.

What I see Host gmauleon-test01 in Zabbix with no items or triggers.

Sidenotes Debug logging could help to find out what is the problem, but it's hardcoded into application - no way to configure it via yaml.

paskal commented 5 years ago

Issue turned out to be present withing Prometheus rules parsing, after fix I've got item, trigger and application created in Zabbix.