Is it possible to parse .json/.jsonl files that contain other log formats?

[Maspital]

Hey, I hope my question isn't badly phrased or has already been asked.

I am trying to obtain sigma hits for logs produced by a simulated network. Problem is that those are not in the .evtx format whatsoever, instead relevant logs from windows machines are collected by winlogbeat and then sent to an elasticsearch instance running on a separate Ubuntu server, from which they are extracted and then - at least in theory - analyzed.

A single event may look like the following

{
  "@timestamp": "2022-08-23T11:16:15.205Z",
  "agent": {
    "ephemeral_id": "e4ab12df-8c99-42e8-a157-b21175cd4141",
    "hostname": "CLIENT2",
    "id": "27aaa960-47e5-483a-a4b7-487e0ae70d81",
    "name": "CLIENT2",
    "type": "winlogbeat",
    "version": "7.10.2"
  },
  "destination": {
    "domain": "-",
    "ip": "x.x.x.x",
    "port": 25
  },
  "ecs": {
    "version": "1.5.0"
  },
  "event": {
    "action": "Network connection detected (rule: NetworkConnect)",
    "category": ["network"],
    "code": 3,
    "created": "2022-08-23T11:16:17.337Z",
    "kind": "event",
    "module": "sysmon",
    "provider": "Microsoft-Windows-Sysmon",
    "type": ["connection", "start", "protocol"]
  },
  "host": {
    "name": "CLIENT2.breach.local"
  },
  "log": {
    "level": "information"
  },
  [...]
}

(Formatted for some readability, the actual files are in .jsonl format aka contain several thousand events).

Is it possible to use chainsaw to parse data like this? As of right now, Chainsaw tells me it cannot find any event logs in the .jsonl files I provide. I suspect that I need to adapt the mapping file in some way, but I am not quire sure how.

Kind regards, Philipp

[alexkornitzer]

Hey @Maspital,

So Chainsaw supports JSON, you can see a super simple example in #94, but it does not support JSONL. The format is just newline delimited JSON so adding in support for that should be very simple. I'll see what I can do.

Alex

[Maspital]

That would be neat, thank you a lot. Guess my question was badly worded then, I'm still somewhat new to this :smile:

[alexkornitzer]

The initial implementation is live in the master branch if you want to give it a go. I have not optimised it yet though, so if the files are very large it will use a fair amount of RAM at the moment.

[Maspital]

That was incredibly fast. Seems to work!

[+] Loading detection rules from: sigma/
[+] Loaded 2111 detection rules (466 not loaded)
[+] Loading event logs from: test_events/winlogbeat_EmailEXEAttack.jsonl (extensions: *)
[+] Loaded 1 EVTX files (2.4 MB)
[+] Hunting: [========================================] 1/1  
[+] 0 Detections found on 0 documents

Guessing the last thing left to do for me is do adapt the mapping properly. In my log example above, what is the correct way to map to a field? E.g. how do I correctly tell Chainsaw where it can find the destination port in the example below:

  "destination": {
    "domain": "-",
    "ip": "x.x.x.x",
    "port": 25
  },

Guessing from the default mapping

      - from: DestinationPort
        to: Event.EventData.DestinationPort
        visible: false

I need to change it to something like this?

      - from: DestinationPort
        to: destination.port
        visible: false

Thank you so much for you time :) not sure if I should close this issue already, so please go ahead

[alexkornitzer]

Yep exactly, the wording in the mapping file is a tad confusing (and we can't change it now without making breaking changes) but it is in the context of the rule so it can be read like so: from the field in rule map it to the field in file.

I have also implemented the correct optimisations so the RAM problem I mentioned above should now not be an issue.