search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.4k stars 1.06k forks source link

Can't easily ship logs from journald #2228

Closed yanndegat closed 7 years ago

yanndegat commented 8 years ago

Problem description

I can't ship logs from journald without getting logs through the syslog protocol and lose information.

Journald comes with log shipping facilities : journal-remote, journal-upload and journal-gateway If graylog could start a "journal-gateway" input, we could then easily ship logs from any systemd system

mariussturm commented 8 years ago

Did you try this: https://marketplace.graylog.org/addons/921903ac-4a7f-4655-8558-0ebd9ca28a1d ?

yanndegat commented 8 years ago

well,

here's what i've done.

that way, i can ship logs without adding any 3rd party log shipper on my hosts.

but as systemd is now on every distro, and so does journald. journald comes with the journalctl upload tool.

if graylog could implement a journald listener, we could ship logs very easily

mikkolehtisalo commented 8 years ago

Summary of journald tools:

Journald was designed for pull model instead of push, so a journald input would have to connect out to gateway services, and take more responsibility of the state than usually. Closest to pushing logs continuously would perhaps be something like

/usr/lib/systemd/systemd-journal-upload -u http://graylog:inputport --save-state=/var/tmp/systemd-journal-upload-state.tmp

and adding it to crontab, running every minute. That would probably be a bit less error prone than piping solutions, because it will handle connections issues etc.

The input handling incoming data could be probably RAW/plaintext input that uses extractors to parse the incoming data (messages are delimited by single empty line):

__CURSOR=s=cf18e1a153964962be63af390c07dcaa;i=27f76f;b=74bab27e0a914fb0831b502673ec893c;m=135b2d76db;t=52d2bee485a54;x=2378b000b73d15aa
__REALTIME_TIMESTAMP=1457041588116052
__MONOTONIC_TIMESTAMP=83134084827
_BOOT_ID=74bab27e0a914fb0831b502673ec893c
_TRANSPORT=stdout
_UID=1000
_GID=1000
_CAP_EFFECTIVE=0
_AUDIT_SESSION=1
_AUDIT_LOGINUID=1000
_SYSTEMD_CGROUP=/user.slice/user-1000.slice/session-1.scope
_SYSTEMD_SESSION=1
_SYSTEMD_OWNER_UID=1000
_SYSTEMD_UNIT=session-1.scope
_SYSTEMD_SLICE=user-1000.slice
_BOOT_ID=74bab27e0a914fb0831b502673ec893c
_MACHINE_ID=098a2dec0f91473cb5f3563b5a6ea9a5
_HOSTNAME=xxxx.localdomain
PRIORITY=4
SYSLOG_IDENTIFIER=/usr/libexec/gdm-x-session
_PID=1949
_COMM=gdm-x-session
_EXE=/usr/libexec/gdm-x-session
_CMDLINE=/usr/libexec/gdm-x-session --run-script gnome-session
_SELINUX_CONTEXT=system_u:system_r:xdm_t:s0-s0:c0.c1023
MESSAGE=(II) intel(0): switch to mode 1920x1080@59.9 on HDMI2 using pipe 0, position (0, 0), rotation normal, reflection none
jalogisch commented 8 years ago

a journal beat is currently under heavy development.

That is not what this request ask for, but that might help people who like to ship the logs to graylog.

For your reference - https://github.com/mheese/journalbeat

jalogisch commented 7 years ago

this is currently nothing that we can solve with Graylog.

You will need to use some kind of transport, as long as journald does not allow to send direct over network.

jcberthon commented 6 years ago

Hi @jalogisch

As far as I know journald can send direct logs over the network, see manpage for systemd-journal-gatewayd (8).

If Graylog could support this type of input that would be great, so there is no need to add an extra in-between thing which will just do the bridging. Such a feature will decrease complexity and increase robustness for delivering journald logs.

Would it be possible that you reconsider opening this issue and the possibility to implement this feature?

jcberthon commented 6 years ago

Sorry, in my previous email I wrongly pointed out to systemd-journal-gatewayd which is the "active" (so pull method in systemd parlance, I got confused by the term). For the push method, there is systemd-journal-upload.

And as far as I can see, it is not limited to do snapshot, the description of this command state:

(...) all journal entries accessible to the user the program is running as will be uploaded, and then the program will wait and send new entries as they become available.

Perhaps since @mikkolehtisalo reported, systemd is now no longer limited to snapshot of the logs with the systemd-journal-upload service.

mikkolehtisalo commented 6 years ago

@jcberthon Yes it seems systemd-journal-upload has developed! This is actually very good news.

jcberthon commented 6 years ago

On the systemd GitHub group there is a project to support Syslog Protocol output: https://github.com/systemd/systemd-netlogd

However, it will never be merged in systemd itself (see https://github.com/systemd/systemd/pull/1890#issuecomment-173264607). And it does not seem to be shipped as a separated utility, on Ubuntu at least.

joschi commented 6 years ago

@jcberthon I'd recommend using the previously mentioned Journalbeat to ship logs from journald to Graylog. It seems to be well maintained, it's working today, and Graylog has a Beats input.

Alternatively, someone from the community might create an input working directly with systemd-journal-upload.

mikkolehtisalo commented 6 years ago

@jcberthon If you'd rather use Logstash, this should do the trick: https://github.com/logstash-plugins/logstash-input-journald

jcberthon commented 6 years ago

Thanks @joschi.

I will try that as anyway none of the above solutions really address my concern for reliability (so keeping the amount of elements low, in all mentioned cases, one needs to add another tool between journald and Graylog).

jcberthon commented 6 years ago

thanks @mikkolehtisalo for the LogStash suggestion.

My main goal is to have a log aggregator for home usage. I want to monitor SSH login to my server and also some specific error logs such as OOM (I'm using Docker for a few services, and sometimes they get killed and restarted when they reach their resource limit, I want to know that so I can perhaps increase the resource limit) or such as FS errors (e.g. I have 3 Raspberry Pis in a small Docker cluster, they all use SD card (so no SMART), therefore only way to see problems with the card is when the FS reports them).

So I want a simple tool to monitor these few things and alert me. Logstash seems like it would require lots of configuration to make it run, and I thought that maybe Graylog would be easier. But maybe both products are much too powerful (and therefore complex) for my very simple use case.

deoren commented 6 years ago

@jcberthon: So I want a simple tool to monitor these few things and alert me. Logstash seems like it would require lots of configuration to make it run, and I thought that maybe Graylog would be easier. But maybe both products are much too powerful (and therefore complex) for my very simple use case.

FWIW, rsyslog is a good option for monitoring/alerting for setups such as the one that you are describing. For 5-10 servers I would have no problem forgoing other products and exclusively using rsyslog for all monitoring/alerting.

rsyslog supports email alerts via ommail and web hooks via its http_request function (though I've not yet to use that support myself).

I would consider larger, more powerful tools like Graylog for monitoring and alerting in larger environments where you may be working with a team that has mixed skill sets and whose members may find maintaining flat-files challenging.

jcberthon commented 6 years ago

Hi @deoren

The problem with rsyslog and emails is that:

But you're right and we have to find a trade-off between the effort to deploy such powerful tools and what they really bring for added value for a small environment. So I will consider your proposal and discuss it internally if we could find a way with it.

Thank you for your suggestion.

drzraf commented 4 years ago

https://github.com/parse-nl/SystemdJournal2Gelf ?

louwrentius commented 4 years ago

https://github.com/parse-nl/SystemdJournal2Gelf ?

This GO-based tool seems to work as intended. It requires a GELF UDP listener on the Graylog Server and this GO-based systemd service on the clients. GELF TCP is not supported.