End-of-line characters prevent certain matches

[a-vine]

Zircolite seems to capture the \n end-of-line characters in the log fields it processes to perform its mapping from SIGMA rules. This can cause problems when mapping certain logs to Techniques.

For example, when I run the following command:

$ python3 zircolite.py --events audit.log --ruleset rules/rules_linux.json --auditd

I get the this (partial) output in the detected_events.json file:

    {
      "row_id": 2531,
      "node": "n12-vm3",
      "type": "CONFIG_CHANGE",
      "timestamp": "2021-05-09 12:22:00",
      "auid": "4294967295",
      "ses": "4294967295",
      "key": "susp_activity",
      "host": "offline",
      "OriginalLogfile": "audit.log-4RCGK4HT.json",
      "op": "add_rule",
      "res": "1\n",
      "list": "4"
    },
    {
      "row_id": 2535,
      "node": "n12-vm3",
      "type": "CONFIG_CHANGE",
      "timestamp": "2021-05-09 12:22:00",
      "auid": "4294967295",
      "ses": "4294967295",
      "key": "susp_activity",
      "host": "offline",
      "OriginalLogfile": "audit.log-4RCGK4HT.json",
      "op": "add_rule",
      "res": "1\n",
      "list": "4"
    },

Here's a concrete example (extracted from my log file) of a log that should be triggered by Zircolite but isn't:

node=n12-vm3 type=SYSCALL msg=audit(1620555801.506:817): arch=c000003e syscall=59 success=yes exit=0 a0=5611f5511fa8 a1=5611f5511ed8 a2=5611f5511f78 a3=4040 items=2 ppid=1347 pid=1384 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="curl" exe="/usr/bin/curl" key="susp_activity"

While the following rule exists in the rules/rules_linux.json file:

    {
        "title": "Suspicious C2 Activities",
        "id": "f7158a64-6204-4d6d-868a-6e6378b467e0",
        "description": "Detects suspicious activities as declared by Florian Roth in its 'Best Practice Auditd Configuration'.\nThis includes the detection of the following commands; wget, curl, base64, nc, netcat, ncat, ssh, socat, wireshark, rawshark, rdesktop, nmap.\nThese commands match a few techniques from the tactics \"Command and Control\", including not exhaustively the following; Application Layer Protocol (T1071), Non-Application Layer Protocol (T1095), Data Encoding (T1132)\n",
        "author": "Marie Euler",
        "tags": [
            "attack.command_and_control"
        ],
        "falsepositives": [
            "Admin or User activity"
        ],
        "level": "medium",
        "rule": [
            "SELECT * FROM logs WHERE key LIKE 'susp\\_activity' ESCAPE '\\'"
        ],
        "filename": "lnx_auditd_susp_c2_commands.yml"
    }

[wagga40]

Hi, thank you for the issue.

I think systematically stripping the trailing newline by adding a rstrip() here : https://github.com/wagga40/Zircolite/blob/0257ea78a59ce7687259fe968ad8186205b00956/zircolite.py#L1229

i.e event[attribute[0]] = attribute[1].rstrip() instead of event[attribute[0]] = attribute[1]

will solde the problem. I will update the code as soon as I can.

[wagga40]

Is the new version OK for this issue ?

[a-vine]

Yes, I just tested it and it works. Thanks for the fix.