cmd/icinga-notifications: Fix SYSLOG_IDENTIFIER for zap fields in journald

[oxzi]

One line diff with a bit of history ahead.

When logging into "systemd-journald", the journaldCore type will be used. Eventually, a log event will be written, resulting in *journaldCore.Write() to be called. There, the name, now called identifier, will be set as the "SYSLOG_IDENTIFIER" field.

This is now passed down to journald, where the "SYSLOG_IDENTIFIER" is handled as the "[s]yslog compatibility fields containing […] the identifier string (i.e. "tag")"^0. After some digging, I found a specification of this tag in RFC 3164, stating that a "TAG is a string of ABNF alphanumeric characters that MUST NOT exceed 32 characters"^1.

As it turns out, "icinga-notifications" does not match this specification due to the presence of "-". When removing it, suddenly journalctl shows the fields prefixed with "ICINGANOTIFICATIONS_".

However, as this looks like a typo and someone is going to "fix" it in the future, I decided to truncate the tag (a.k.a. identifier a.k.a. name) to be only "notifications". Now, an error might look like "NOTIFICATIONS_ERROR".

[julianbrost]

Some puzzle piece is missing here, at least I don't see why SYSLOG_IDENTIFIER should be the culprit here. This is a full log message written by the v0.1.0 to journal:

{
  "_COMM" : "icinga-notifica",
  "_BOOT_ID" : "f74ea6b780ab457aa78b2588f84dccef",
  "_SYSTEMD_SLICE" : "system.slice",
  "_GID" : "113",
  "_RUNTIME_SCOPE" : "system",
  "__REALTIME_TIMESTAMP" : "1721829899105587",
  "_EXE" : "/usr/sbin/icinga-notifications",
  "_SYSTEMD_CGROUP" : "/system.slice/icinga-notifications.service",
  "_CMDLINE" : "/usr/sbin/icinga-notifications",
  "MESSAGE" : "incident: Successfully sent a notification via channel plugin",
  "_SYSTEMD_UNIT" : "icinga-notifications.service",
  "_PID" : "22933",
  "_CAP_EFFECTIVE" : "0",
  "__CURSOR" : "s=aed145c9a2e34842977fe930812a9ac8;i=f1c;b=f74ea6b780ab457aa78b2588f84dccef;m=d91b3842;t=61dfec37ff533;x=d2f77f8ce330bc66",
  "_SOURCE_REALTIME_TIMESTAMP" : "1721829899105532",
  "PRIORITY" : "6",
  "SYSLOG_IDENTIFIER" : "icinga-notifications",
  "_SYSTEMD_INVOCATION_ID" : "142cb77a33714e70ba0f66130c329b9b",
  "_TRANSPORT" : "journal",
  "_HOSTNAME" : "jb-d12",
  "_MACHINE_ID" : "c8e230a4abf54885a3be78a5a0317b59",
  "__MONOTONIC_TIMESTAMP" : "3642439746",
  "_SELINUX_CONTEXT" : "unconfined\n",
  "_UID" : "108"
}

It has "SYSLOG_IDENTIFIER" : "icinga-notifications" set, so it doesn't look like - there is a problem for journald.

[oxzi]

Short update: it's not the SYSLOG_IDENTIFIER, but using logger's the name, which is then used to set the SYSLOG_IDENTIFIER as prefix for each journald field.

Unfortunately, the field key is not really specified. For starters, the Native Journal Protocol documentation only defines those keys as "environment-like". The systemd.journal-fields man page shows a multitude of potential keys, but has no specification either. After some digging, I found it in the source, …/libsystemd/sd-journal/journal-file.c.

In a nutshell:

• Key length MUST be within (0, 64] characters.
• Key MUST NOT start with _, unless it is an protected variable.
• Key MUST NOT start with a digit.
• Key MUST only contain [A-Z0-9_].

[oxzi]

I have made the relevant changes over at the icinga-go-library in the just updated PR Icinga/icinga-go-library#48. More context is available in this comment, https://github.com/Icinga/icinga-go-library/pull/48#issuecomment-2250020876.

Please note, for the current fix, changes are only necessary in the icinga-go-library and there is nothing to do in this repository, unless bumping the dependency.

[oxzi]

Created #264 to keep track of this.