double quotes in jsonl value prevent certain matches

[a-vine]

As each json value is surrounded by double quotes, when logs are converted to jsonl format and double quotes are present at the beginning and end of the string of a json value, this causes problems when triggering certain logs and mapping them to techniques.

For example, when I convert the following raw log into jsonl:

node=n12-vm3 type=CONFIG_CHANGE msg=audit(1620555720.138:183): auid=4294967295 ses=4294967295 op=add_rule key="susp_activity" list=4 res=1

I obtain the following:

{"node": "n12-vm3", "type": "CONFIG_CHANGE", "msg": "audit(1620555720.138:183):", "auid": "4294967295", "ses": "4294967295", "op": "add_rule", "key": "\"susp_activity\"", "list": "4", "res": "1"}

If I run the following command:

$ python3 zircolite.py --events audit.json --ruleset rules/rules_linux.json --jsonl

Considering the following rule in the rules/rules_linux.json file:

    {
        "title": "Suspicious C2 Activities",
        "id": "f7158a64-6204-4d6d-868a-6e6378b467e0",
        "description": "Detects suspicious activities as declared by Florian Roth in its 'Best Practice Auditd Configuration'.\nThis includes the detection of the following commands; wget, curl, base64, nc, netcat, ncat, ssh, socat, wireshark, rawshark, rdesktop, nmap.\nThese commands match a few techniques from the tactics \"Command and Control\", including not exhaustively the following; Application Layer Protocol (T1071), Non-Application Layer Protocol (T1095), Data Encoding (T1132)\n",
        "author": "Marie Euler",
        "tags": [
            "attack.command_and_control"
        ],
        "falsepositives": [
            "Admin or User activity"
        ],
        "level": "medium",
        "rule": [
            "SELECT * FROM logs WHERE key LIKE 'susp\\_activity' ESCAPE '\\'"
        ],
        "filename": "lnx_auditd_susp_c2_commands.yml"
    }

The log should be triggered by Zircolite but isn't. However, when I remove the double quotes present in the initial raw log (as below), the jsonl log is triggered:

{"node": "n12-vm3", "type": "CONFIG_CHANGE", "msg": "audit(1620555720.138:183):", "auid": "4294967295", "ses": "4294967295", "op": "add_rule", "key": "susp_activity", "list": "4", "res": "1"}

[wagga40]

If you use directly the auditd log with Zircolite (even without considering the newline problem mentionned in your other issue) : python3 zircolite.py -e auditd.log -r rules/rules_linux.json --auditd, it is detected : - Suspicious C2 Activities [medium] : 1 events.

The problem in this case comes from your auditd to jsonl conversion. When converting auditd logs to json, Zircolite ignore values with double quotes to avoid this kind of problem. This is why when you run Zircolite with the --keepflat option : python3 zircolite.py -e auditd.log -r rules/rules_linux.json --auditd --keepflat and the correction from your other issue (the newline problem), the generated jsonl (something like flattened_events_XXXX.json in the current directory) does not have the double quotes :

{
    "node": "n12-vm3",
    "type": "CONFIG_CHANGE",
    "timestamp": "2021-05-09 12:22:00",
    "auid": "4294967295",
    "ses": "4294967295",
    "op": "add_rule",
    "key": "susp_activity",
    "list": "4",
    "res": "1",
    "host": "offline",
    "OriginalLogfile": "auditd2.log-AR1CSN8Y.json"
}

[a-vine]

Thanks for the answers and responsiveness. :-)

In fact, I don't want to use the raw auditd log directly with Zircolite. I'm currently testing, but my long-term need is to process logs (auditd and windows) in jsonl format.

I think that when converting to another format, the raw log values should be strictly preserved (overall, compromised integrity makes data unreliable). That's why I've kept the quotes in the string.

So in my opinion, Zircolite should process the same strings for mapping, regardless of whether the logs are in raw or jsonl format.

[wagga40]

I will have to look into it.

Please note that the rule will never match in this case. In fact, the original rule logic is :

logsource:
    product: linux
    service: auditd
detection:
    selection:
        key: 'susp_activity'
    condition: selection

Since there is no startswith or endswith keywords, the field value must be strictly equal to susp_activity and will never match "susp_activity".

[a-vine]

However, key="susp_activity" matches when I run the following command on auditd raw logs:

$ python3 zircolite.py --events audit.log --ruleset rules/rules_linux.json --auditd

Below, an extract of the detected_events.json file:

{
  "title": "Suspicious C2 Activities",
  "id": "f7158a64-6204-4d6d-868a-6e6378b467e0",
  "description": "Detects suspicious activities as declared by Florian Roth in its 'Best Practice Auditd Configuration'.\nThis includes the detection of the following commands; wget, curl, base64, nc, netcat, ncat, ssh, socat, wireshark, rawshark, rdesktop, nmap.\nThese commands match a few techniques from the tactics \"Command and Control\", including not exhaustively the following; Application Layer Protocol (T1071), Non-Application Layer Protocol (T1095), Data Encoding (T1132)\n",
  "sigmafile": "lnx_auditd_susp_c2_commands.yml",
  "sigma": [
    "SELECT * FROM logs WHERE key LIKE 'susp\\_activity' ESCAPE '\\'"
  ],
  "rule_level": "medium",
  "tags": [
    "attack.command_and_control"
  ],
  "count": 24,
  "matches": [
    {
      "row_id": 1577,
      "node": "n12-vm1",
      "type": "CONFIG_CHANGE",
      "timestamp": "2021-05-09 12:16:36",
      "auid": "4294967295",
      "ses": "4294967295",
      "key": "susp_activity",
      "host": "offline",
      "OriginalLogfile": "audit.log-P6LS4BY4.json",
      "op": "add",
      "res": "1",
      "list": "4"
    },

And here is the triggered log:

$ sed -n 1577p audit.log
node=n12-vm1 type=CONFIG_CHANGE msg=audit(1620555396.106:173): auid=4294967295 ses=4294967295 op="add rule" key="susp_activity" list=4 res=1

[wagga40]

You are right, that's what I was saying in the comment above .

I took time to think, for me, your problem comes from your raw logs conversion to JSONL. Since the key field "records the user defined string associated with a rule that generated a particular event in the Audit log", it is a string. So the double quote must not be taken into account because the field value is a string.

So, either, your conversion from raw logs to JSONL must remove the double quotes to only take the value of the field (that's what I do in Zircolite here) or the sigma rule logic must be changed to something like :

logsource:
    product: linux
    service: auditd
detection:
    selection:
        key|contains: 'susp_activity'
    condition: selection

Which will produce a SQL query/Zircolite rule pretty much like this :

SELECT * FROM logs WHERE key LIKE '%susp\\_activity%' ESCAPE '\\'`

The second solution is not the best one; instead, it is better to focus on converting the logs correctly.