zabbix-agent-extensions

Overview

A set of zabbix UserParameter scripts and production ready monitoring templates for linux systems.

• this project ist originated on: https://github.com/scoopex/zabbix-agent-extensions
• the ci is implmented with github actions
• docker images of a zabbix agent can be obtained at https://hub.docker.com/repository/docker/scoopex666/zabbix-agent-with-agent-extensions

Monitoring details

This package provides the following capabilities:

• linux standard monitoring using the standard items provided by the zabbix-agent
  • monitor memory behavior
  • monitor important servcies: smtp, ssh, cron
  • swap usage
  • 5min system load
  • monitor dmesg for bad behavior of the system
  • monitor the maximum and minimum of processes
  • automatic discovery
    • filesystems: inode and space measures
      (the amount of discovered filesystems can be limited by a configuration file on the monitored host)
    • network interfaces: packets and transferrates per second
    • storage devices: operations per second (the amount of discovered devices can be limited by a configuration file on the monitored host)
    • network interfaces: packets and transferrates per second
  • number of processes
• monitor ICMP ping
• monitor the MTA mailqueue
• monitor NFS operations/retransmits
• apache monitoring: (enable /server-status for localhost clients, https://httpd.apache.org/docs/2.4/mod/mod_status.html)
  • Check Apache Server-Status (Readers, Writers - alert if to many slots are in use)
  • Loadbalancer check
  • Monitor Mod JK backend status
• elasticsearch node and cluster monitoring (needs elasticsearch and es_stats_zabbix Python modules)
• redis Monitoring (needs redis Python module)
• NGINX Monitoring (enable /basic_status only for localhost clients, https://nginx.org/en/docs/http/ngx_http_stub_status_module.html)
• generic discovery (put json snippets to /var/run/zabbix-generic-discovery/ITEMNAME.json i.e. with puppet and get a combined discovery value)
• monitor puppet state (execution statistics, disabled state, ...)
• linux software raid state
• check quality of ntp sync behavior
• check for required reboot
• monitor open file descriptors
• Configurable autodiscovery of block devices
• Monitor performance havior of discovered block devices
• Configurable autodisovery of real disk devices
• Check smart state of disks and gather statistics
• Discover network devices and monitor performance and error behavior
• Zabbix agent version
• Monitor springboot servers
  • beta, monitoring is still rudimentary
  • contributions are very welcome

Almost all measures are integrated in graphs which are displayed on the various host screens.

The project inlcudes various zabbix templates for relase 5.2.

Note: There are also 2.2 and 3.0 templates, but they are outdated because we do not support them anymore. Probably these outdated templates might be a good starting point for your own environment.

A quick is provided by the following files:

• custom-os-linux
• custom-os-linux-hardware
• custom-os-puppet
• custom-service-apache
• custom-service-cups
• custom-service-elasticsearch-cluster
• custom-service-elasticsearch-node
• custom-service-nginx
• custom-service-nginx-logstats
• custom-service-redis
• custom-service-varnish
• custom-service-springboot
• custom-os-kubernetes-node

For development, debugging and contribution, please have a look for the development document.

Installation and usage

Install the agent extensions on linux systems

• Install the zabbix-agent according to the description at zabbix download page
• Download a package suitable for your distribution using the releases section
  (or build your own package as described in development document)
• Configure the most important configuration values for your zabbix agent
  • "Server=..." and/or "ServerActive=..."
  • "Hostname=" - your hostname
• Install the distribution package of the zabbix-agent-extensions using the package management utils or a configuration management systems

  rpm -Uvh zabbix-agent-extensions-<release number>.noarch.rpm
  dpkg -i zabbix-agent-extensions-<release number>.noarch.rpm

  This adds a include statement for /usr/share/zabbix-agent-extensions/include.d and restart the zabbix agent automatically.

Install the agent extensions on kubernetes nodes

This procedure deploy zabbix agents on all worker nodes of your kubernetes cluster and provides autodiscovery for all nodes

• Make zabbix available to the deployed agent, i.e. by deploying a zabbix proxy in k8s
• Import template custom-os-kubernetes-node.xml
• Define a automatic discovery:
  • Configuration -> Actions -> "Autoregistration actions"
  • Add conditions:
    • "Host metadata contains kubernetes-node"
    • "Proxy equals infra-app-dev"
  • Add operations:
    • "Add host"
    • "Add to host groups: [DEVELOPMENT]"
    • "Link to templates: Custom - OS - Kubernetes Node"
• Configure deployment
  • Download kubernetes yaml file
  • Adapt the file to your own needs
  • Configure ZBX_ACTIVESERVER
  • Configure ZBX_PASSIVESERVERS
  • Configure the version of the image check dockerhub
  • Apply deployment

    kubectl apply -f zabbix-agent-daemonset-kubernetes.yaml
    kubectl describe -n infra-zabbix-agent daemonsets.apps zabbix-agent
    kubectl logs -n infra-zabbix-agent zabbix-agent-8n4ss -f

How to configure the zabbix server/templates

• Open Zabbix web frontend
• Create a host
  • Configuration -> Hosts -> Create host
  • Add the name ds "Host name" efined in the first step of this section
  • Define a agent interface with the correct fully qualified hostname and the ip address
• Open "Configuration" => "Templates"
  • Press button "Import"
  • Activate Linux template
    • Load "zabbix_templates/5.2/Custom - OS - Linux.xml" (also works for higher versions)
    • Open template "Custom - OS - Linux" and modify the default values defined in the macros (if neccessary)
    • Devices
      • {$DISK_HIGH_READ_IOPS_LIMIT} : alert limit for read iops/sec
      • {$DISK_HIGH_WRITE_IOPS_LIMIT} : alert limit for read iops/sec
      • {$DISK_IOPS_LIMIT_MEASURES} : how many measures need to be above the configured limits before alerting
    • Filesystems
      • {$DISK_USAGE_ABOVE_1TB_MINFREE_GBYTES} : how many gigagbytes should be free for filesystem above 1TB
      • {$DISK_USAGE_PERCENT_ALARM} : send alarm for filesystems below 1 TB on prcentage
      • {$DISK_USAGE_PERCENT_WARN} : send warning for filesystems below 1 TB on prcentage
    • {$MAXIMUM_NUMBER_RETRANSMISSIONS} : how many nfs retransmissions are ok on every measurement cycle
    • NTP
      • {$MAX_NTP_OFFSET_MS} : the maximum offset limit in milliseconds
      • {$MIN_NTP_SERVER_COUNT} : how many good ntp sources should be avaiilable, it is istrongly recommended to change the default to 2
    • {$MONITOR_LOAD_WARNING_MULT} : a multiplicator with the number of cpus for load monitoring
    • {$MONITOR_TIMEOUT} : amount of time to complain if hosts does not provide values anymore
    • Assign template "Custom - OS - Linux" to the desired hosts and modify the default values to host specific settings
  • Activate Apache template
    • Load "zabbix_templates/*/Custom - Service - Apache.xml"
    • Open template "Custom - Service - Apache" and modify the default values defined in the macros
    • Assign template "Custom - Service - Apache" to the desired hosts and modify the default values to host specific settings
  • Activate and configure other templates
• Check the function
  • Have a look at Monitoring -> Hosts -> Latest data
    • Check: "Show items without data"
    • Check: "Show details"
    • Press "Apply"
  • Check dashboards at Monitoring -> Hosts -> Dashboards

How to configure discovery for zabbix agent

• Configure disk device discovery

  • Create files: (if the file does not exist, the default is used)
    • /etc/zabbix/item_zabbix_device_discovery.json
      
    • /etc/zabbix/item_zabbix_discovery_filesystems.json
      
  • Add content to include/exclude devices
    (what it does: include all devices and hardware models, after that filter out all devices and models which match to one of the python regexes)

    /usr/bin/zabbix_discovery_devices --help
    /usr/bin/zabbix_discovery_filesystems --help

  • Test

    zabbix_agentd -t "vfs.dev.discovery"
    /usr/bin/zabbix_discovery_devices --config /etc/zabbix/item_zabbix_discovery_devices.json --debug
    
    zabbix_agentd -t "vfs.fs.discovery"
    /usr/bin/zabbix_discovery_filesystems --config /etc/zabbix/item_zabbix_discovery_filesystems.json --debug

• Configure generic discovery
  • Decide to use a appname consiting of the following characters: a-zA-Z0-9. (Alphanumeric and dot characters)
  • Add file snippes to /var/run/zabbix-generic-discovery/ which look like -.json

    {
    "{#FOOOO}":"fooservice",
    "{#BAR}":"footype"
    },

  • Test the disovery by

    zabbix_agentd -t "generic.discovery[appname]"

Licence and Authors

Additional authors are very welcome - just submit your patches as pull requests.

• Marc Schoechlin ms@256bit.org
• Marc Schoechlin marc.schoechlin@vico-research.com (not active anymore)
• Marc Schoechlin marc.schoechlin@breuninger.de (not active anymore)

This software is licensed by GPLv2 - review file "LICENSE"