search

gdestuynder / audisp-json

42 stars 17 forks source link

Duplicate key:values in audisp-json.log #22

Closed Phrozyn closed 5 years ago

Phrozyn commented 5 years ago

As I reviewed some events today in the log to understand the structure I noticed there were several fields being duplicated within the details structure of the event.

I'll add examples below:

Auditserial is duplicated twice at the top and bottom of socket events: { "category": "socket", "summary": "Socket", "severity": "INFO", "hostname": "ip-172-31-23-75.us-west-2.compute.internal", "processid": "0", "processname": "audisp-json", "timestamp": "2019-04-04T13:18:23+0000", "tags": [ "audisp-json", "2.2.4", "audit" ], "details": { "auditserial": "121", "session": "4294967295", "fsgid": "0", "sgid": "0", "egid": "0", "fsuid": "0", "suid": "0", "euid": "0", "gid": "0", "pid": "297", "process": "/sbin/auditctl", "tty": "(none)", "uid": "0", "user": "root", "originaluid": "4294967295", "auditkey": "net", "protocol": "9", "type": "3", "domain": "10", "processname": "auditctl", "auditserial": "121" } }

Execve can have auditserial up to 5 times: { "category": "execve", "summary": "Execve: /sbin/ldconfig -p", "severity": "INFO", "hostname": "ip-172-31-23-75.us-west-2.compute.internal", "processid": "0", "processname": "audisp-json", "timestamp": "2019-04-04T13:18:23+0000", "tags": [ "audisp-json", "2.2.4", "audit" ], "details": { "auditserial": "138", "rdev": "00:00", "ogid": "0", "ouid": "0", "mode": "0100755", "dev": "ca:01", "inode": "263305", "path": "/sbin/ldconfig", "auditserial": "138", "cwd": "/", "auditserial": "138", "command": "/sbin/ldconfig -p", "auditserial": "138", "session": "4294967295", "fsgid": "0", "sgid": "0", "egid": "0", "fsuid": "0", "suid": "0", "euid": "0", "gid": "0", "pid": "310", "process": "/sbin/ldconfig", "tty": "(none)", "uid": "0", "user": "root", "originaluid": "4294967295", "parentprocess": "cloud-init", "auditkey": "exec", "processname": "ldconfig", "auditserial": "138" } }

In the following several fields are duplicated: auditserial, rdev, ogid, ouid, mode, dev, inode, path { "category": "execve", "summary": "Execve: systemd-detect-virt --quiet --container", "severity": "INFO", "hostname": "ip-172-31-23-75.us-west-2.compute.internal", "processid": "0", "processname": "audisp-json", "timestamp": "2019-04-04T13:18:23+0000", "tags": [ "audisp-json", "2.2.4", "audit" ], "details": { "auditserial": "142", "rdev": "00:00", "ogid": "0", "ouid": "0", "mode": "0100755", "dev": "ca:01", "inode": "266643", "path": "/lib64/ld-linux-x86-64.so.2", "auditserial": "142", "rdev": "00:00", "ogid": "0", "ouid": "0", "mode": "0100755", "dev": "ca:01", "inode": "266643", "path": "/usr/bin/systemd-detect-virt", "auditserial": "142", "cwd": "/", "auditserial": "142", "command": "systemd-detect-virt --quiet --container", "auditserial": "142", "session": "4294967295", "fsgid": "0", "sgid": "0", "egid": "0", "fsuid": "0", "suid": "0", "euid": "0", "gid": "0", "pid": "312", "process": "/usr/bin/systemd-detect-virt", "tty": "(none)", "uid": "0", "user": "root", "originaluid": "4294967295", "parentprocess": "cloud-init", "auditkey": "exec", "processname": "systemd-detect-", "auditserial": "142" } }

In some cases more than twice: { "category": "execve", "summary": "Execve: /bin/sh /sbin/dhclient-script", "severity": "INFO", "hostname": "ip-172-31-23-75.us-west-2.compute.internal", "processid": "0", "processname": "audisp-json", "timestamp": "2019-04-04T13:18:23+0000", "tags": [ "audisp-json", "2.2.4", "audit" ], "details": { "auditserial": "173", "rdev": "00:00", "ogid": "0", "ouid": "0", "mode": "0100755", "dev": "ca:01", "inode": "267828", "path": "/lib64/ld-linux-x86-64.so.2", "auditserial": "173", "rdev": "00:00", "ogid": "0", "ouid": "0", "mode": "0100755", "dev": "ca:01", "inode": "267828", "path": "/bin/sh", "auditserial": "173", "rdev": "00:00", "ogid": "0", "ouid": "0", "mode": "0100755", "dev": "ca:01", "inode": "267828", "path": "/sbin/dhclient-script", "auditserial": "173", "cwd": "/", "auditserial": "173", "command": "/bin/sh /sbin/dhclient-script", "auditserial": "173", "session": "4294967295", "fsgid": "0", "sgid": "0", "egid": "0", "fsuid": "0", "suid": "0", "euid": "0", "gid": "0", "pid": "332", "process": "/bin/dash", "tty": "(none)", "uid": "0", "user": "root", "originaluid": "4294967295", "parentprocess": "dhclient", "auditkey": "exec", "processname": "dhclient-script", "auditserial": "173" } }

tristanweir commented 5 years ago

We validated that this data is not being duplicated in our MozDef instance.

the-smooth-operator commented 5 years ago

@tristanweir if you checked the data in MozDef (Elasticsearch) it is very probable that the duplicated keys are getting deleted somewhere in the collection/ingestion process. For example when I load the file in python it deduplicates the keys.

Some more details about the issue after analyzing a sample log file**:

For the sample analyzed which contains 16597 objects, I found the next:

alberto@banbao /tmp % python duplication.py | sort | uniq -c
  58903 duplicate key: 'auditserial'
  11860 duplicate key: 'dev'
  11860 duplicate key: 'inode'
  11860 duplicate key: 'mode'
  11860 duplicate key: 'ogid'
  11860 duplicate key: 'ouid'
  11860 duplicate key: 'path'
  11860 duplicate key: 'rdev'

If you want, I can provide you the sample file and the modifications I did to make it a valid JSON.

I have experienced the issue in different environments (Debian and RedHat based), describing the test setup:

**In order to load the log file for analysis, we have to convert it to a valid JSON document. audisp-json produces valid JSON objects, but is not wrapping them in an array. So, it needs an [ ] at the beginning and end and a , after each object.

gdestuynder commented 5 years ago

fixed in 2.2.5 note: we previously relied on the receiving library to dedup - not a great solution of course. now its already deduped