Icingabeat v.6.3.3 does not send the hostname for the check, instead send own hostname

5nafu commented 6 years ago

After installing icingabeat (6.3.3) on a complete new machine, with minimal changes to the icingabeat.yml and outputting to a completely new elasticsearch index (5.5), accessing an icinga2 (2.9.1), icingabeat will log checkresults with the name of the host running icingabeat as host.name: screenshot_2018-09-12 kibana

My icingabeat.yml (with comments, hostnames and credentials redacted):

icingabeat:
  host: "XXXX"
  port: 5665
  user: "XXXX"
  password: "****"
  ssl.verify: false
  eventstream.types:
    - CheckResult
    - StateChange
    - Notification
    - AcknowledgementSet
    - AcknowledgementCleared
    - CommentAdded
    - CommentRemoved
    - DowntimeAdded
    - DowntimeRemoved
    - DowntimeStarted
    - DowntimeTriggered

  eventstream.filter: ""
  eventstream.retry_interval: 10s
  statuspoller.interval: 30s
setup.dashboards.enabled: false
setup.kibana:
output.elasticsearch:
  hosts: ["XXXX:9200"]
  protocol: "https"
  username: "XXXX"
  password: "*****"

Representative checkresult (in json):

{
  "_index": "icingabeat-6.3.3-2018.09.12",
  "_type": "doc",
  "_id": "AWXO07xNOFh-eQsJ_2M4",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-12T17:27:45.615Z",
    "check_result": {
      "active": true,
      "type": "CheckResult",
      "execution_end": "2018-09-12T17:25:24.348Z",
      "check_source": "icinga2-sat...",
      "state": 0,
      "vars_before": {
        "reachable": true,
        "state": 0,
        "state_type": 1,
        "attempt": 1
      },
      "exit_status": 0,
      "output": "OK",
      "schedule_end": "2018-09-12T17:25:24.348Z",
      "ttl": 0,
      "schedule_start": "2018-09-12T17:25:23.970Z",
      "execution_start": "2018-09-12T17:25:23.970Z",
      "vars_after": {
        "reachable": true,
        "state": 0,
        "state_type": 1,
        "attempt": 1
      },
      "command": [
        "/usr/lib/nagios/custom_plugins/check_multi",
       "..."
      ]
    },
    "beat": {
      "version": "6.3.3",
      "name": "icingabeat-stg-01",
      "hostname": "icingabeat-stg-01"
    },
    "host": {
      "name": "icingabeat-stg-01"
    },
    "service": "$SERVICENAME",
    "timestamp": "2018-09-12T17:25:24.352Z",
    "type": "icingabeat.event.checkresult"
  },
  "fields": {
    "check_result.execution_end": [
      1536773124348
    ],
    "check_result.schedule_end": [
      1536773124348
    ],
    "@timestamp": [
      1536773265615
    ],
    "check_result.execution_start": [
      1536773123970
    ],
    "check_result.schedule_start": [
      1536773123970
    ],
    "timestamp": [
      1536773124352
    ]
  },
  "sort": [
    1536773265615
  ]
}

Expected Behavior

Somewhere in the document should the hostname for the checkresult (notification, ...) be analog to the service.

Current Behavior

All hostnames will be the icingabeat hostname

Steps to Reproduce (for bugs)

install icingabeat as documented
configure icingabeat with access to icinga & elasticsearch
let icingabeat create the elasticsearch template icingabeat setup
restart icingabeat
create index patterns in kibana (and ignore error while importing kibana dashboards)
check resulting documents in Kibana

Context

We can not use icingabeat 6.3.3 as there is no way to know to which of our hosts the messages are related to.

Your Environment

Beat version (icingabeat -version): icingabeat version 6.3.3 (amd64), libbeat 6.3.3
Icinga 2 version (icinga2 --version): r2.9.1-1
Elasticsearch version (curl -XGET 'localhost:9200'): 5.5.0
Logstash version, if used (bin/logstash -V): 5.6.7 (not configured to be used)
Kibana version, if used (curl -XGET http://localhost:5601/status -I): 5.5.0
Operating System and version: Debian GNU/Linux 9.5 (stretch)

ekeih commented 6 years ago

I can confirm this issue. We had to downgrade to 6.1.1 as a workaround.

yoshi314 commented 5 years ago

on my end downgrade makes elastic reject the fields from 6.1.1

5nafu commented 5 years ago

Aparently this seems to be connected to the upgrade to libbeat 6.3:

As a solution for https://github.com/elastic/beats/issues/7050, we're adding a host.name field to all events. This is duplicate information from beat.name, but is used to avoid the mapping conflict and to slowly introduce the "host as an object" approach.

https://github.com/elastic/beats/pull/7051

jcarterch commented 5 years ago

Since host.name is now populated by libbeat, it seems like we'd need to rename the host field, or move it out of the document root.

Moving it out could be as simple as

--- a/beater/eventstream.go
+++ b/beater/eventstream.go
@@ -51,7 +51,11 @@ func BuildEventstreamEvent(e []byte) beat.Event {
        event.Fields = common.MapStr{}

        for key, value := range icingaEvent {
-               event.Fields.Put(key, value)
+               if key == "host" {
+                       event.Fields.Put("event.host", value)
+               } else {
+                       event.Fields.Put(key, value)
+               }
        }

which will populate event.host in your beat output with the host value from the original event.

Since host shows up in a number of different event stream types, I think it makes sense to use a generic event key (and not nest it in check_result) so you can search by host across all event types.

Any thoughts? Renaming to avoid collision might make more sense, with the rest of the fields (e.g., service) living in the document root.

bobapple commented 5 years ago

Same issue as in #26

Icinga / icingabeat