Support multiline event correlation in Filebeat auditd module

[adriansr]

Filebeat's auditd module consumes logs from Linux auditd daemon. Unlike Auditbeat auditd, which consumes data directly from the kernel, Filebeat's auditd module cannot correlate multiple log lines into a single event, resulting in some audit events being split into multiple documents, which makes it harder to craft queries over this data.

For example, a single execve event is composed of the following lines:

type=SYSCALL msg=audit(1653312742.539:861): arch=c000003e syscall=59 success=yes exit=0 a0=26bb3f0 a1=26a5dd0 a2=26a92c0 a3=7ffe7734a5a0 items=2 ppid=2062 pid=2086 auid=5468821 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="auditcmd"^]ARCH=x86_64 SYSCALL=execve AUID="admin" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=EXECVE msg=audit(1653312742.539:861): argc=2 a0="cat" a1="/etc/passwd"
type=CWD msg=audit(1653312742.539:861): cwd="/root"
type=PATH msg=audit(1653312742.539:861): item=0 name="/bin/cat" inode=18346 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0^]OUID="root" OGID="root"
type=PATH msg=audit(1653312742.539:861): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33635826 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0^]OUID="root" OGID="root"
type=PROCTITLE msg=audit(1653312742.539:861): proctitle=636174002F6574632F706173737764

Ideally, it should result in a single document enriched with the different PATHs and process information.

We should investigate a method to aggregate the different log lines and see if it's possible to update the ingest pipeline to parse the new multiline events.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Keep in mind that auditd records can be interleaved ~and out of order~. The auditd's auparse has a good description of the behavior at https://github.com/linux-audit/audit-userspace/blob/c19119fe4bf25d3e755196cfef908f4c160dd7a7/auparse/internal.h#L36-L84.

You could use Filebeat multiline as it exists today (e.g. trigger the start with type=SYSCALL), but there are edge cases that multiline cannot handle properly which would result in incomplete groupings or grouping where the sequence numbers are not all the same. The latter could be mitigated in post-processing by rejecting messages that don't match the sequence, but that means you are dropping some messages.

[adriansr]

@andrewkroh the linked docs seem to contradict this claim "can be [...] out of order."

The auditd system does guarantee that the records that make up an event will appear in order

My idea was that if the following conditions are met:

• Records for a given event appear in order.
• It can be determined that an event is complete when the last record for that event is observed. (no need for timeouts/expiration).

... then events can be coalesced with a Beats script processor that observes the source log line by line.

Anyway, I have analyzed the module's test logs and all of them are correctly ordered and I don't observe any interleaving. I understand they are already ordered by auditd/auparse lib? Am I missing something?

[andrewkroh]

It can be determined that an event is complete when the last record for that event is observed. (no need for timeouts/expiration).

If you we can reliably detect the last record in a sequence then that could be used trigger when to flush the event. One of the end of sequence markers mentioned is AUDIT_EOE, and the audit.log that Filebeat consumes doesn't actually contain this message (source ref). Without that EOE message, I think this will be a challenge to accomplish without timeouts.

And as you know, Beat processors are only driven by incoming events so flushing on a timeout could only be triggered by a new event. And since beat processors can only output a single event it would be a problem deciding on outputting the timed-out event or the newly received event...

I like the idea, but I think the edge cases cannot be overcome with the Beat processor model.

[andrewkroh]

I think we should consider implementing the Filebeat parser interface, with the reassembler code from go-libaudit. Basically seeing if https://github.com/elastic/beats/issues/6484 can be added as a Filebeat parser. The benefit being that we can converge the parsing logic of the Filebeat audit module, Fleet auditd log integration, and Auditbeat.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ipnerds]

This feature would be much appreciated.

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)