search

logtopus / tentacle

The daemon which provides data to the logtopus server
Apache License 2.0
2 stars 1 forks source link

Support systemd journal via systemd-journal-gatewayd #17

Open fredlahde opened 2 years ago

fredlahde commented 2 years ago

I was researching ways to read the systemd journal programmatically and it turns out, systemd-journal-gatewayd exists.

This is a systemd service itself, which can be installed on ubuntu via the systemd-journal-remote package.

Once installed, it listens on 0.0.0.0:19531 per default and, among others, exposes the /entries endpoint. Using the Accept: application/json header, we get the following JSON per log line:

{
  "_GID": "63112",
  "_MACHINE_ID": "1797d13f9e574405af3aedf42c8a7e12",
  "_SYSTEMD_CGROUP": "/system.slice/systemd-journal-gatewayd.service",
  "_EXE": "/usr/lib/systemd/systemd-journal-gatewayd",
  "_SYSTEMD_INVOCATION_ID": "65b63afd57414ea0aeaea8b06e1ed671",
  "__CURSOR": "s=1e558073a2a4450a852fc018977dad89;i=1638;b=9f444b408d434c6388fa6fe33efc0978;m=189893deb;t=5e2abd62cc1fc;x=58edfd5a21acb1fa",
  "SYSLOG_IDENTIFIER": "systemd-journal-gatewayd",
  "__MONOTONIC_TIMESTAMP": "6602440171",
  "_CAP_EFFECTIVE": "0",
  "TID": "12189",
  "_UID": "63112",
  "_SOURCE_REALTIME_TIMESTAMP": "1656602544095605",
  "PRIORITY": "6",
  "SYSLOG_FACILITY": "3",
  "_HOSTNAME": "node-0",
  "_BOOT_ID": "9f444b408d434c6388fa6fe33efc0978",
  "_SYSTEMD_SLICE": "system.slice",
  "_PID": "12189",
  "_SYSTEMD_UNIT": "systemd-journal-gatewayd.service",
  "MESSAGE": "microhttpd: MHD_OPTION_EXTERNAL_LOGGER is not the first option specified for the daemon. Some messages may be printed by the standard MHD logger.",
  "_TRANSPORT": "journal",
  "_SELINUX_CONTEXT": "unconfined\n",
  "_COMM": "systemd-journal",
  "_CMDLINE": "/lib/systemd/systemd-journal-gatewayd",
  "__REALTIME_TIMESTAMP": "1656602544095740"
}

We already have type: journal and unit: foobar config directives, with which we can easily filter down this JSON for a specific systemd Unit, and with __REALTIME_TIMESTAMP we can also support the from_ms query-parameter

More info about systemd-journal-gatewayd can be found here: https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html

fredlahde commented 2 years ago

PS: The /entries endpoint also supports a Range Header, aswell as a follow mode. Both of which could be used to deal with units which log a lot of stuff.. Always fetching everything is maybe not desirable from a performance standpoint.

Maybe we can remember the __CURSOR of the last entry we got and only request new entries