kube-icinga

kube-icinga automatically deploys icinga objects out of your kubernetes resources. It has built-in autodiscovery and will work out of the box. However you can change various default configurations and deploy custom icinga objects or disable/enable kubernetes objects to monitor.

Features

• Autodiscovery
• Icinga servicegroup support
• Create services for kubernetes nodes, services, ingresses and persistent volumes
• Completely customizable per resource or per resource type

How does it work?

Multiple watchers are bootstraped and listen for any kubernetes changes. Those changes will reflect immediately on your icinga environment.

• Kubernetes namespaces will result in icinga service groups
• Nodes will result in host objects
• Ingresses will result in icinga services (and or host objects)
• Services (ClusterIP, NodePort, LoadBalanacer) will result in icinga services (and/or host objects)
• Persistent volumes will result in icinga services (and/or host objects)

Table of Contents

• Features
• How does it work?
• Resource types
  • Namespaces
  • Nodes
  • Ingresses
  • Services
  • Volumes
• Requirements
• Setup icinga2 api user
• Deployment
• Resource visibility
• Advanced topics
  • Using icinga2 apply rules
  • Globally overwrite icinga object definitions
  • Overwrite icinga object definitions directly in kubernetes resources
• Configuration

Resource types

Namespaces

Namespaces will result in icinga servicegroups. Note that the servicegroup gets only created if one of the following resources exist in a given namespace.

Nodes

Kubernetes nodes get mappend 1:1 to icinga host objects. There is no other magic. However you may change the icinga host object by configuring the global config kubernetes.nodes or change annotations values in single node objects.

Ingresses

Each kubernetes ingress hostname/path will result in a single icinga service. By default all those services get attached to a single icinga host named kubernetes-ingresses and each service also gets attached to a servicegroup which is taken from the kubernetes namespace. You may change the hostname by configuring kubernetes.ingresses.hostName (or set a custom hostname in an annotation) or even set kubernetes.ingresses.hostName to null. If hostName is null, host objects get dynamically created named ingress-${namespace}-${name}. This will result in more host objects but the services still get created and get attached to their servicegroup (namespace). If you choose to create dynamic host objects you may want to disable service provisioning, you may set kubernetes.ingresses.applyServices to false and create services manually with icinga apply rules (see advanced). Besides that you can still configure advanced options for those host objects and/or service objects (See configuration). For instance you may attach ingress services directly to all kubernetes worker nodes by setting kubernetes.ingresses.attachToNodes to true. (This will result in many more services and checks depending on the size of your cluster!)

By default ingresses get checked with the http check_command. Usually you do not want to change this but you can by using annotations or changing the global ingress settings.

Services

Each service port will result in a single icinga service (like ingress paths). By default all those services get attached to a single host named kubernetes-${serviceType}-services. Supported service types are either ClusterIP, NodePort or LoadBalancer. Like ingresses you may change the hostname or configure host provisioning by setting hostName to false (kubernetes.services.${serviceType}.hostName). Dynamic service host objects get named service-${namespace}-${name}. You may also disable service provisioning by setting kubernetes.services.${serviceType}.applyServices to false if you prefer icinga apply rules. ClusterIP provisioning is disabled by default since ClusterIP services are only visible internally in kubernetes. You need either an icinga deployment within the kubernetes cluster or optionally an icinga satellite if you decide to enable ClusterIP provisioning. NodePort services get attached by default directly to the kubernetes icinga node objects.

By default services get checked with the service protocol which usually result in check_tcp. However you may set custom settings using annotations (or global service settings).

Volumes

Each persistent volume will result in a single icinga service. By default all those services get attached to a single host named kubernetes-volumes. Like for ingresses and services you may change the same advanced settings for volumes. Since a persistent volume is not directly attached to namespace the namespace for the service group gets taken from the persistent volume claim.

Note Volume services are created by default with the dummy check_command. Since there is no way for kube-icinga to specify the correct command automatically. You need to either overwrite the check_command in annotations, set the command in the serviceDefinition for volumes or disable service provisioning, enable host provisioning (set hostName to null) and create icinga apply rule based services manually.

You may wonder where your pods are in this setup. Well it does not make any sense to monitor single pods. Pods are mortal and (always) moving. The important thing is to monitor your services as an entity.

Requirements

• A running kubernetes cluster
• Icinga2 server with enabled API module

Note: Icinga2 does not have to be running on kubernetes! You may just use your icinga2 server outside your kubernetes cluster.

Setup icinga2 api user

kube-icina requires an icinga api user which first must be created.

/etc/icinga2/conf.d/api-users.conf:

object ApiUser "kube-icinga" {
  password = "kube-icinga"
  permissions = ["*"]
}

Deployment

The recommended (and logical) way to deploy kube-icinga is deplyoing kube-icinga on kubernetes itself using the gyselroth/kube-icinga docker image.

The kubernetes deployment can simply be added like this:

kubectl -f https://raw.githubusercontent.com/gyselroth/kube-icinga/master/kube-icinga.yaml

(Change the secret password and ICINGA_ADDRESS value accordingly)

Note: kube-icinga will be created as single pod deployment in the kubernetes kube-system namespace. You may changes this behaviour.

Resource visibility

The resource yaml also contains a new cluster role kube-icinga. kube-icinga will create icinga objects for all visible namespaces which are by default all namespaces since it is a kubernetes cluster role. You may specify resources visibile to kube-icinga with custom RBAC rules.

Advanced topics

Using icinga2 apply rules

You certainly can use icinga2 apply rules. You may disable auto service deployments via applyServices for ingresses, services and volumes and define your own services via apply rules. Of course you can also setup a mixed deployment. Automatically let kube-icinga deploy services and apply additional services via apply rules.

Note: Since icinga apply rules are not triggered if an object gets updated kube-icinga will delete and recreate those objects.

All icinga host objects are created with all kubernetes data available packed in the variable vars.kubernetes. Therefore you may apply rules with this data.

Globally overwrite icinga object definitions

It is possible to set custom options for the icinga objects during creation. You might set custom values via kubernetes.ingresses.hostTemplates or kubernetes.ingresses.serviceTemplate`. The same can also be done for services. For example it might be crucial to set a custom zone for ClusterIP services since they are only reachable within the cluster and shall be monitored by an icinga satelite node. Any valid setting for a specific icinga object can be set.

Note This will set the custom options for all objects from the same type (services.ClusterIP in the follwing example), this may not what you want. Using kubernetes annotations you may configure custom incinga attributes directly in the kubernetes resource definition.

{
  "kubernetes": {
    "services": {
      "ClusterIP":
        "discover": true,
        "serviceDefiniton": {
          "zone": "my_custom_icinga_zone"
          "vars.a_custom_icinga_variable": ["foo", "bar"]
        },
      }
    }
  }
}

Overwrite icinga object definitions directly in kubernetes resources

kube-icinga is able to parse kubernetes annotations and merge those with its default settings.

Note: With annotations you may set specific settings for each kubernetes resources while configure options for kube-icinga set settings on a resource type basis.

You may use the following annotations:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: generic-storage-24afe01f-2300-11e9-94e3-0050568fe3c2,
  selfLink":"/api/v1/persistentvolumes/generic-storage-24afe01f-2300-11e9-94e3-0050568fe3c2
  uid":"2a66425e-2300-11e9-94e3-0050568fe3c2
  resourceVersion: 65603642
  creationTimestamp: 2019-01-28T13:25:22Z
  annotations:
    kube-icinga/check_command: "check_storage_lun"
    kube-icinga/hostname: "storagesystem.example.lan"
    hpe.com/docker-volume-name": "generic-storage-24afe01f-2300-11e9-94e3-0050568fe3c2"
    pv.kubernetes.io/provisioned-by: "example.com/storage"
    volume.beta.kubernetes.io/storage-class: "generic-storage"

Configuration

kube-icinga itself can be configured via environment variables however you may also deploy and change the config.json file instead of environment variables.

List of configurable values: