GelfWriter custom fields

marcofl commented 6 years ago

Unfortunately the GelfWriter Feature is not really documented (config options, etc.) but I was able to find: https://github.com/Icinga/icinga2/blob/master/lib/perfdata/gelfwriter.ti.

This tells me there is no way to add custom fields to the GELF format log messages.

Expected Behavior

It should be possible to add a subset of the custom host and services variables as fields in the log messages. Also adding downtime_depth and ack state by default would help for SLA reporting, etc. (because services in downtime for example are okay to be broken).

Current Behavior

Only a fixed set of fields are part of the log messages. See fields in https://github.com/Icinga/icinga2/blob/master/lib/perfdata/gelfwriter.cpp

Possible Solution

Implement to new options host_custom_vars and service_custom_vars

object GelfWriter "gelf" {
  host = "logging1.example.com"
  port = 12202
  source = "master1.monitoring.example.com"
  enable_send_perfdata = false
  host_custom_vars = "customer,location,sla"
  service_custom_vars = "sla"
}

Then customer for example should be added to host AND service check results / state changes log messages, this should be fairly easy, because a service always belongs to a host, therefore having the host.vars.customer in the service message as well should be possible.

Context

We would like to add custom host and service variables as fields of the GELF message. For example all our hosts have a variable customer which we would like to see in Graylog so we can create streams per customer, etc.

Also it would be interesting to see things like acknowledgment (yes / no) and downtime_depth in the log message.

Your Environment

Version used (icinga2 --version): r2.8.1-1
Operating System and version: not relevant for feature request
Enabled features (icinga2 feature list): not relevant for feature request
Icinga Web 2 version and modules (System - About): not relevant for feature request
Config validation (icinga2 daemon -C): not relevant for feature request

dnsmichi commented 6 years ago

Docs

Unfortunately the GelfWriter Feature is not really documented (config options, etc.) but I was able to find: https://github.com/Icinga/icinga2/blob/master/lib/perfdata/gelfwriter.ti.

https://www.icinga.com/docs/icinga2/latest/doc/09-object-types/#gelfwriter

Proposal

  host_custom_vars = "customer,location,sla"
  service_custom_vars = "sla"

just puts strings, which isn't telling the user what exactly does happen here. Please look at the InfluxDBWriter implementation for tags and fields on how it could be done. The name *_custom_vars also is misleading, one should be able to just use any object attribute if needed.

Furthermore, what output is to be expected ... field name, value as string, json-encoded, etc? How would you deal with custom attributes which are arrays or dictionaries?

Meta data

Acknowledgements, downtime_depth, etc. is something I would call "meta information". Not always needed, slowing down writes, and sometimes preferred. Best would be a similar option like enable_send_metadata as seen with other features.

marcofl commented 6 years ago

Thanks for the answer. I'm totally okay with having a switch like enable_send_metadata to include downtime and ack information, etc.

With list of variable names you are right, better do in a similar way as with influxdb:

    custom_fields = {
      customer = "$host.vars.customer$"
      sla = "$host.vars.sla$"
    }

About datatype I'm not sure what you could do here. I guess there could be a way to auto detect the type and then just use JSON for array / dicts otherwise it's just a string? JSON can then be extracted from the field in Graylog.

marcofl commented 6 years ago

We plan to sponsor this, checking with our customer first. We'll contact you directly then. Please keep this issue open in the mean time.

Al2Klimov commented 3 years ago

Hello again @marcofl!

Any news on this?

Best, AK

Icinga / icinga2