Add support for structured logging to systemd-journal

[faxm0dem]

All our nodes now run journald and it would be great if you could add support for it to your classpath. We successfully tested https://github.com/gnieh/logback-journal and it provides the following benefits over the syslog appender we used previously:

• structured logging
• give control over what's logged and what's not (stack traces, name of logger, etc.)
• add context fields (MDC)

Here's an example output of journalctl -t dcache -o json:

{
  "CODE_FILE": "PoolV4.java",
  "THREAD_NAME": "pool-test-test-2",
  "MESSAGE": "Repository finished",
  "CODE_LINE": "537",
  "CODE_FUNC": "org.dcache.pool.classic.PoolV4.lambda$afterStart$3",
  "LOGGER_NAME": "org.dcache.pool.classic.PoolV4",
  "SYSLOG_IDENTIFIER": "dcache",
  "DCACHE_CELLS_DOMAIN": "test-pool-test-test-Domain",
  "_PID": "56481",
  "DCACHE_CELLS_CELL": "pool-escape-test",
  "_SOURCE_REALTIME_TIMESTAMP": "1596023920496981"
}

This particular example is being achieved using the following logback appender config:

  <appender name="journal" class="org.gnieh.logback.SystemdJournalAppender">
    <logSourceLocation>true</logSourceLocation>
    <logStackTrace>true</logStackTrace>
    <logLoggerName>true</logLoggerName>
    <logMdc>true</logMdc>
    <mdcKeyPrefix>dcache_</mdcKeyPrefix>
    <syslogIdentifier>dcache</syslogIdentifier>
  </appender>

[kofemann]

Hi,

This is a nice idea. With dcache 6.2 we enable systemd and , as the result, all log files will go into journal. You can check the documentation here:

https://github.com/dCache/dcache/blob/master/docs/TheBook/src/main/markdown/install.md#using-systemd-service

We have to check how to combine logback-journal with other systemd related work we are doing.

[faxm0dem]

Hi,

Have you had time to check this ?

Cheers

[kofemann]

Hi,

sorry for the late reply. As we use systemd by default, this is a nice addition. However, this will break the current tools. I would like to get a confirmation dcache admins that thay are happy with it. can you please start a discussion at user forum.?

Thanks!

[faxm0dem]

How does this break the current tools ? It's just a supplementary jar file to add to the classpath

[maswan]

So, one comment on this from a site admin point of view. Our primary usecase would be for the billing logs, and there we already have several choices: text lines in files and sql database. A third option for structured logging into journald would be a useful addition. I assume this structured logging would also take care of proper escaping etc when exported as json events?

The other part is that I understand kofemann's reluctance to change the default logging format, since many sites have several different parsers for it for various purposes. But adding the software component so that sites could enable it in their custom logback configs would be a reasonable first step?

[faxm0dem]

I'm not suggesting to change the default logging format, just enabling the new by shipping the jar that makes it possible

[faxm0dem]

But adding the software component so that sites could enable it in their custom logback configs would be a reasonable first step?

Yes, definitely that's basically what I was asking for :-) just add the .jar and so sites can configure their logback to log using journald