Systemd journal inputer for Hekad

[Micheletto]

Feature Request: Add a systemd journal inputer for Hekad. See http://www.freedesktop.org/wiki/Software/systemd/ for information about systemd.

[christianvozar]

You should not need a special input for systemd journal. I realize your request was 6 months ago but Heka now has the ProcessInput which should acomodate your needs. http://hekad.readthedocs.org/en/latest/configuration.html#processinput

Utilize the ProcessInput with the command "$ journalctl -b" and this should get you what you need to inject systemd journals into Heka.

[chancez]

There's also systemd-journal-gatewayd which lets you obtain data from a local, or remote systemd host. You could use the HttpInput to poll the gatewayd service, and use the application/json or another format, and set the proper decoder.

[rafrombrc]

Closing since the use case can be met w/o trouble using existing inputs.

[rafrombrc]

Okay, further investigation reveals that systemd is a bigger PoS than I even imagined, and it actually will require a custom journalctl input just so we can ensure that we invoke w/ the --show-cursor option, and that we capture and persist the cursor line when we're finished streaming. Although not sure what we're supposed to do if the shutdown isn't clean.

[nathwill]

+1 for this request; especially as journald takes over some functions of traditional syslog, a journald input would be really useful.

[chancez]

@rafrombrc Alternatively you can also use systemd-journal-gatewayd. In any method, you'll need a way to track your position in the logstream, so maybe the cursor/offset tracking logic can be generalized somewhere?

[chrissnell]

If anyone is interested in a "native" (well...with cgo) Go implementation, I'm happy to help out. I wrote a journald reader for another project with the github.com/coreos/go-system library and have been running it in production for a few months now. I don't think it would be hard to adapt to Heka. The one headache with of go-systemd is that newer versions of Ubuntu (and potentially others) have the journald library as part of libsystemd. Older versions use libsystemd-journal. So, the binaries become OS distro-specific. Not sure if that's a problem for this project.

[sathieu]

I'm not sure to understand the distro-specific part. Has the ".so" filename changed?

I currently have to build different .deb for Debian wheezy and jessie, so it won't change for me.

[nathwill]

also of note here: https://www.freedesktop.org/software/systemd/man/systemd-journal-upload.html. Especially with the --save-state option, it seems like it ought to be a good tool to use in combination with HttpListenInput. I'll screw around with it and see if I can come up with a working pattern.

[chrissnell]

@sathieu Yes, different .so files depending on which distro. I'm not totally sure that this was a distro packaging choice--it may be systemd version-dependent. At any rate, it's not impossible to work around and go-systemd works well.

[nathwill]

lame, systemd-journal-upload's a no-go for now it seems. it performs the entire upload as one continuous streaming POST, so an HttpListenInput in heka never chunks the messages :/. sooo cloose!

so far, the ways i know of to get journal data at present include:

• /run/systemd/journal/syslog as UdpInput + ForwardToSyslog=yes in journald.conf
  • pros: individually packaged messages
  • cons: mising a lot of context that journal provides, lossy as they're UDP and there's no history if you're not listening when a message is sent and no cursor tracking to resume where you left off.
• journalctl + ProcessInput
  • pros: multiple output formats, can get everything , filter by journal cli options (e.g. current boot, or specific unit, and lots of other journalctl control opts)
  • cons: no cursor tracking, so messages can be duped if your ProcessInput restarts

here's an example heka config i've used for this:

[hekad]
base_dir = '/tmp/heka'

[Journalctl]
type = "ProcessInput"
ticker_interval = 0
decoder = "JsonDecoder"
splitter = "TokenSplitter"

[TokenSplitter]

[Journalctl.command.0]
bin = "/usr/bin/journalctl"
args = ["--boot", "--follow", "--output=json", "--quiet"]

[JsonDecoder]
type = "SandboxDecoder"
filename = "lua_decoders/json.lua"

[RstEncoder]

[LogOutput]
message_matcher = "TRUE"
encoder = "RstEncoder"

which results in heka messages like:

:Timestamp: 2016-03-02 22:01:26 +0000 UTC
:Type: json
:Hostname: wyrd.home.nathwill.net
:Pid: 28595
:Uuid: bc93a564-dcbf-4e71-af0a-b79ed6f47df6
:Logger: Journalctl
:Payload: 
:EnvVersion: 
:Severity: 7
:Fields:
    | name:"CODE_FILE" type:string value:"src/core/job.c"
    | name:"_CMDLINE" type:string value:"/usr/lib/systemd/systemd --switched-root --system --deserialize 22"
    | name:"CODE_FUNCTION" type:string value:"job_log_status_message"
    | name:"SYSLOG_IDENTIFIER" type:string value:"systemd"
    | name:"SYSLOG_FACILITY" type:string value:"3"
    | name:"MESSAGE_ID" type:string value:"39f53479d3a045ac8e11786248231fbf"
    | name:"PRIORITY" type:string value:"6"
    | name:"CODE_LINE" type:string value:"774"
    | name:"__REALTIME_TIMESTAMP" type:string value:"1456956000657571"
    | name:"UNIT" type:string value:"sysstat-collect.service"
    | name:"_SOURCE_REALTIME_TIMESTAMP" type:string value:"1456956000657378"
    | name:"__MONOTONIC_TIMESTAMP" type:string value:"133772980231"
    | name:"_PID" type:string value:"1"
    | name:"_SELINUX_CONTEXT" type:string value:"system_u:system_r:init_t:s0"
    | name:"_GID" type:string value:"0"
    | name:"_COMM" type:string value:"systemd"
    | name:"_EXE" type:string value:"/usr/lib/systemd/systemd"
    | name:"_TRANSPORT" type:string value:"journal"
    | name:"_MACHINE_ID" type:string value:"14f2d832739a4500ba270174d9a63529"
    | name:"__CURSOR" type:string value:"s=cb5f1f6cad8a4760af865e3c4d8d6a0a;i=17ec8;b=60aba9208674459cabfe0726af1b7ef7;m=1f257db007;t=52d1800df60a3;x=1c6230d0d93e3092"
    | name:"_UID" type:string value:"0"
    | name:"_HOSTNAME" type:string value:"wyrd.home.nathwill.net"
    | name:"_SYSTEMD_SLICE" type:string value:"-.slice"
    | name:"_SYSTEMD_CGROUP" type:string value:"/"
    | name:"_CAP_EFFECTIVE" type:string value:"3fffffffff"
    | name:"RESULT" type:string value:"done"
    | name:"_BOOT_ID" type:string value:"60aba9208674459cabfe0726af1b7ef7"
    | name:"MESSAGE" type:string value:"Started system activity accounting tool."

which isn't bad, but still not as nice as a native input that interfaced directly with the journal.

[sekka1]

Thanks @nathwill that method of sending logs to kafka is working for me.