Missing msgtype and events?

[juju4]

with laurel 0.1.1 on Ubuntu 18.04 and base laurel configuration

# jq -r 'keys | nth(1)' /var/log/laurel/audit.log | sort | uniq -c | sort -nr | head -20
   2268 SECCOMP
     66 ID
     33 USER_START
     33 USER_END
     33 USER_ACCT
     33 LOGIN
      3 SERVICE_STOP
      3 SERVICE_START

while

# cut -d ' ' -f 7 /var/log/audispd/audispd.log | sort | uniq -c | sort -nr | head -20
  56855 type=SYSCALL
  56855 type=PROCTITLE
  43944 type=PATH
  30833 type=OBJ_PID
  13681 type=EXECVE
   7771 type=SECCOMP
    156 type=USER_END
    156 type=CRED_DISP
    155 type=USER_START
    149 type=USER_ACCT
    149 type=LOGIN
    149 type=CRED_ACQ
     16 type=SERVICE_STOP
     16 type=SERVICE_START
      6 type=USER_CMD
      6 type=CRED_REFR
      3 type=NETFILTER_CFG
# grep 1631967301.772 /var/log/audit/audit.log
type=SYSCALL msg=audit(1631967301.772:1842167): arch=c000003e syscall=59 success=yes exit=0 a0=7f8c1c4c9260 a1=7f8c1c6db4e0 a2=7ffe80304318 a3=9 items=3 ppid=18688 pid=18689 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=3993 comm="check-pid" exe="/usr/bin/env" key="rootcmd"
type=EXECVE msg=audit(1631967301.772:1842167): argc=4 a0="/usr/bin/env" a1="bash" a2="/opt/zeek/share/zeekctl/scripts/helpers/check-pid" a3="21898"
type=PATH msg=audit(1631967301.772:1842167): item=0 name="/opt/zeek/share/zeekctl/scripts/helpers/check-pid" inode=4655459 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1631967301.772:1842167): item=1 name="/usr/bin/env" inode=4152 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1631967301.772:1842167): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=18925 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1631967301.772:1842167): proctitle=2F7573722F62696E2F656E760062617368002F6F70742F7A65656B2F73686172652F7A65656B63746C2F736372697074732F68656C706572732F636865636B2D706964003231383938
# grep 1631967301.772 /var/log/laurel/audit.log
#

SYSCALL is the ones where I would expect process activity Even if merging some of the different msgtype, same data should be found

[IppSec]

I believe you may be excluding the End Of Event (EOE) Records. If you use the Florian Roth auditd template it is excluded by default. It is this line in your /etc/audit/rules.d/audit.conf file.

## Ignore EOE records (End Of Event, not needed)
-a always,exclude -F msgtype=EOE

Just comment out the above, restart the services and you should be good to go

[hillu]

@juju4 Does this help?

[juju4]

Sorry, I believed I had updated.

Yes, it fixed the issue. But one thing that I was thinking are they are other types required/useful? CWD seems to be not required but useful for omigod per https://techcommunity.microsoft.com/t5/azure-sentinel/hunting-for-omi-vulnerability-exploitation-with-azure-sentinel/ba-p/2764093 and https://github.com/SigmaHQ/sigma/pull/2042/files This is also excluded in https://github.com/Neo23x0/auditd/blob/master/audit.rules#L70

Still got very different count, so not sure if something else or just the result of laurel merging logs

# jq -r 'keys | nth(1)' /var/log/laurel/audit.log | sort | uniq -c | sort -nr | head -20
   2636 OBJ_PID
   1905 PARENT_INFO
   1275 ID
    687 SECCOMP
     12 USER_START
     12 USER_END
     12 USER_ACCT
     12 LOGIN
      4 PATH
      2 SERVICE_STOP
      2 SERVICE_START
# cut -d ' ' -f 7 /var/log/audispd/audispd.log | sort | uniq -c | sort -nr | head -20
  26974 type=SYSCALL
  26974 type=PROCTITLE
  22666 type=PATH
  13163 type=OBJ_PID
   6026 type=EXECVE
   3358 type=SECCOMP
     70 type=USER_END
     70 type=CRED_DISP
     69 type=USER_START
     63 type=USER_ACCT
     63 type=LOGIN
     63 type=CRED_ACQ
      7 type=SERVICE_STOP
      7 type=SERVICE_START
      6 type=USER_CMD
      6 type=CRED_REFR
      3 type=NETFILTER_CFG

[hillu]

I think that the jq call does not do what you want. Using it on the sample piece of laurel output from https://github.com/threathunters-io/laurel/blob/master/practical-auditd-problems.md gives us just the EXECVE key. Removing the nth gives me:

[
  "CWD",
  "EXECVE",
  "ID",
  "PATH",
  "PROCTITLE",
  "SYSCALL"
]

… and apparently the ordering is not retained.

CWD, PROCTITLE, SYSCALL should give you the same counts. PATH should be off by a factor 2-3 because an EXECVE event usually goes together with multiple PATH lines which are grouped into one entry in the JSON log. The EXECVE count may be off if there are long command lines that are split across multiple EXECVE lines. (Merging those was one of the main motivations for writing Laurel in the first place.)

[juju4]

If comparing this way, got below which has still some big difference in count

# jq -r 'keys' /var/log/laurel/audit.log | egrep -v '(\]|\[)' | sort | uniq -c | sort -nr | head -20
    281   "ID",
    278   "SYSCALL"
    278   "PROCTITLE",
    278   "PARENT_INFO",
    199   "PATH",
    199   "CWD",
    152   "EXECVE",
     79   "OBJ_PID",
      2   "SECCOMP"
      1   "USER_END"
      1   "ID"
      1   "CRED_DISP",
# cut -d ' ' -f 8 /var/log/audispd/audispd.log | sort | uniq -c | sort -nr | head -20
  60448 type=SYSCALL
  60445 type=PROCTITLE
  48570 type=PATH
  31322 type=OBJ_PID
  26632 type=CWD
  14536 type=EXECVE
   2803 type=SECCOMP
    164 type=USER_END
    164 type=CRED_DISP
    163 type=USER_START
    155 type=USER_ACCT
    155 type=LOGIN
    155 type=CRED_ACQ
     32 type=USER_LOGIN
     23 type=SERVICE_START
     20 type=SERVICE_STOP
     10 type=NETFILTER_CFG
      8 type=USER_CMD
      8 type=CRED_REFR
# jq -r 'keys' /var/log/laurel/audit.log-20211002 | egrep -v '(\]|\[)' | sort | uniq -c | sort -nr | head -20
  11694   "ID",
  11280   "SYSCALL"
  11277   "PROCTITLE",
  11266   "PARENT_INFO",
   6739   "CWD",
   6738   "PATH",
   4320   "OBJ_PID",
   2410   "EXECVE",
    278   "SECCOMP"
     72   "ID"
     36   "USER_END"
     36   "CRED_DISP",
     35   "USER_START"
     21   "USER_ACCT"
     21   "LOGIN"
     21   "CRED_ACQ",
     14   "USER_CMD"
     14   "CRED_REFR",
      4   "SERVICE_STOP"
      4   "SERVICE_START"
# cut -d ' ' -f 8 /var/log/audispd/audispd.log-20211002 | sort | uniq -c | sort -nr | head -20
 175266 type=SYSCALL
 175265 type=PROCTITLE
 130816 type=PATH
  97193 type=OBJ_PID
  70442 type=CWD
  48698 type=EXECVE
   7197 type=SECCOMP
    527 type=USER_START
    527 type=USER_END
    527 type=CRED_DISP
    513 type=USER_ACCT
    513 type=LOGIN
    513 type=CRED_ACQ
     66 type=SERVICE_START
     65 type=SERVICE_STOP
     30 type=NETFILTER_CFG
     14 type=USER_CMD
     14 type=CRED_REFR
      1 type=USER_LOGIN
      1 type=SOCKADDR

(not sure why the change in cut field number)

[hillu]

alright. I'll be doing some remodelling to be able to implement event filtering fieatures, this will allow for some more diagnostics and statistics.

In the meantime, here's a stupid idea: Are you sure that you collected all the logs for laurel? In the default configuration, log file segments are rotated at 1 MB which translates to something between 1700 and 1800 messages on my boxes.

[juju4]

I had increased it to 10MB because I wanted logrotation to be done by logrotate every day and it seems 0 is not the right way the way to disable this option from laurel. looking at logdir, this seems enough for a quiet day but not enough when more activity like today when I'm playing on the system. increased to 1GB and will check again after few days.

yeah, some extra filtering options would be nice, auditd exclusions are not really the most granular/precise... Thanks

[juju4]

I added report with both audispd and laurel output in my ansible role pipeline One example here https://github.com/juju4/ansible-auditd/runs/3847691965?check_suite_focus=true#step:12:1 number seems approaching from a type perspective, less when digging by keyword

also for file integrity monitoring, audispd have nametype with CREATED, DELETED... but laurel output is always NORMAL

[hillu]

also for file integrity monitoring, audispd have nametype with CREATED, DELETED... but laurel output is always NORMAL

Not sure what you mean by that, can you provide an example?

[juju4]

# audispd report
*** sensitive_file CREATE+DELETE
Total unique files: 9
Total unique apt .#lk0x files: 0
      2 item=1 mode=0100600 rdev=00:00
      1 name="/etc/systemd/system/snap-core-11743.mount.Xg4Jpb9hHSCL~" ouid=0 nametype=DELETE
      1 name="/etc/systemd/system/snap-core-11743.mount.Xg4Jpb9hHSCL~" ouid=0 nametype=CREATE
      1 name="/etc/systemd/system/snap-core-11743.mount" ouid=0 nametype=CREATE
      1 name="/etc/systemd/system/snap-core-11420.mount" ouid=0 nametype=DELETE
      1 name="/etc/systemd/system/multi-user.target.wants/snap-core-11743.mount" ouid=0 nametype=CREATE

# laurel report (not same day)
*** sensitive_file -CREATE+DELETE- (name, nametype, exe)
      3 /var/www NORMAL /usr/bin/aide
      3 /var/www/html/.well-known NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/fim-test-touch-auditd1 NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/fim-test-touch3 NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/.fim-test-file5-long NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/.fim-test-file3-long NORMAL /usr/bin/aide
      3 /var/www/html/.well-known/.fim-test-file2 NORMAL /usr/bin/aide

[hillu]

Laurel does not do anything specific with the PATH entries that may contain different nametype entries. So if you paste the following into its STDIN…

type=SYSCALL msg=audit(1633859647.072:120613): arch=c000003e syscall=316 success=yes exit=0 a0=ffffff9c a1=7fff70af382e a2=ffffff9c a3=7fff70af3851 items=4 ppid=3219422 pid=3219423 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts9 ses=3 comm="mv" exe="/bin/mv" subj==unconfined key="sudo"
type=CWD msg=audit(1633859647.072:120613): cwd="/home/bengen/src/spyre"
type=PATH msg=audit(1633859647.072:120613): item=0 name="/etc/sudoers.d/" inode=38798702 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1633859647.072:120613): item=1 name="/etc/sudoers.d/" inode=38798702 dev=fd:00 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1633859647.072:120613): item=2 name="/etc/sudoers.d/x2goserver.disabled" inode=38819726 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1633859647.072:120613): item=3 name="/etc/sudoers.d/x2goserver" inode=38819726 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PROCTITLE msg=audit(1633859647.072:120613): proctitle=6D76002F6574632F7375646F6572732E642F7832676F7365727665722E64697361626C6564002F6574632F7375646F6572732E642F7832676F736572766572
type=EOE msg=audit(1633859647.072:120613): 

it should give you 4 PATH entries, like so:

{
  "ID": "1633859647.072:120613",
  "SYSCALL": {
    "arch": "0xc000003e",
    "syscall": 316,
    "success": "yes",
    "exit": 0,
    "items": 4,
    "ppid": 3219422,
    "pid": 3219423,
    "auid": 1000,
    "uid": 0,
    "gid": 0,
    "euid": 0,
    "suid": 0,
    "fsuid": 0,
    "egid": 0,
    "sgid": 0,
    "fsgid": 0,
    "tty": "pts9",
    "ses": 3,
    "comm": "mv",
    "exe": "/bin/mv",
    "subj": "=unconfined",
    "key": "sudo",
    "ARCH": "x86_64",
    "SYSCALL": "renameat2",
    "AUID": "bengen",
    "UID": "root",
    "GID": "root",
    "EUID": "root",
    "SUID": "root",
    "FSUID": "root",
    "EGID": "root",
    "SGID": "root",
    "FSGID": "root",
    "ARGV": [
      "0xffffff9c",
      "0x7fff70af382e",
      "0xffffff9c",
      "0x7fff70af3851"
    ]
  },
  "CWD": {
    "cwd": "/home/bengen/src/spyre"
  },
  "PATH": [
    {
      "item": 0,
      "name": "/etc/sudoers.d/",
      "inode": 38798702,
      "dev": "fd:00",
      "mode": "0o40755",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "PARENT",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    },
    {
      "item": 1,
      "name": "/etc/sudoers.d/",
      "inode": 38798702,
      "dev": "fd:00",
      "mode": "0o40755",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "PARENT",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    },
    {
      "item": 2,
      "name": "/etc/sudoers.d/x2goserver.disabled",
      "inode": 38819726,
      "dev": "fd:00",
      "mode": "0o100440",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "DELETE",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    },
    {
      "item": 3,
      "name": "/etc/sudoers.d/x2goserver",
      "inode": 38819726,
      "dev": "fd:00",
      "mode": "0o100440",
      "ouid": 0,
      "ogid": 0,
      "rdev": "00:00",
      "nametype": "CREATE",
      "cap_fp": "0x0",
      "cap_fi": "0x0",
      "cap_fe": 0,
      "cap_fver": "0x0",
      "cap_frootid": "0",
      "OUID": "root",
      "OGID": "root"
    }
  ],
  "PROCTITLE": {
    "ARGV": [
      "mv",
      "/etc/sudoers.d/x2goserver.disabled",
      "/etc/sudoers.d/x2goserver"
    ]
  }
}

[hillu]

@juju4 Do you still see problems with missing messages?

[hillu]

@juju4 ping?

[juju4]

Sorry for delay, busy times.

On github action side, I lost laurel report between build of Nov 29th and Dec 4th. just repaired it

On my test server, for some reasons, laurel logs are empty since some time while /var/log/audit/audit.log is fine.

# ps aux |grep laurel
_laurel   1072  0.0  0.1  10248  9488 ?        S<   Nov03  28:47 /usr/local/sbin/laurel --config /etc/laurel/config.toml
# killall laurel
# ps aux |grep laurel
_laurel   7661  0.6  0.0   1272   976 ?        S<   09:32   0:00 /usr/local/sbin/laurel --config /etc/laurel/config.toml
# ls -l /var/log/laurel/audit.log
-rw-------+ 1 _laurel _laurel 64550 Dec 18 09:33 /var/log/laurel/audit.log

trying to review during holidays

I see many commits since last 1.0.5 release on Nov 1st. any plan for a new one?

Thanks

[hillu]

I see many commits since last 1.0.5 release on Nov 1st. any plan for a new one?

Yeah, I somehow got sidetracked dealing with surprise features in Java software. ;-)

[juju4]

no problem. same for many people :)

[hillu]

I am closing this issue because we have not made an progress in almost two years. Feel free to reopen if you still see problems with a more current version.