brimdata / zed

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.38k stars 67 forks source link

EVTX reader #2793

Open x62smith opened 3 years ago

x62smith commented 3 years ago

Windows security and system logs are processed as .evtx and I believe they are binary XML. These files contain a lot of information about what the system and the processes within are doing. A popular implementation is Windows Internals Sysmon. This give more detailed information with less logs and is configurable.

Bringing these logs into zed/Brim can opened up more information when along side Zeek/Suricata data. The Sysmon Type-3 Network logs specifically can be joined with Zeek data via a 5-Tuple match. Once combine it will show what application are doing on the network and that same Type-3 Network log can be joined with the other Sysmon logs via the Process ID giving a much broader insight into what's going on in the network and host.

https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon

pcap_evtx_files.zip

philrz commented 1 year ago

Another community user recently asked about this. In their own words:

so you know in brim, how you can drag and drop a pcap into it? I think it will be a really huge deal if you somehow allowed something similar except for windows eventlog files

I know that might not be a strict goal for zui but it'll be pretty impactful, specifically for the security folks

right now one of my workflows is evtx -> json -> zed and it's working pretty great - but then I remembered brim has some functionality that lets you just go straight from pcap to an explore window

I see there's some existing Go libraries such as https://github.com/0xrawsec/golang-evtx that we could potentially leverage.

philrz commented 8 months ago

A community user recently pointed at some other tools that can do this conversion. While we're waiting for this functionality to arrive at the Zed side, I just tried out the workflow using such a tool. When combined with some other recent Zui features it's fairly seamless.

For instance, the tool https://github.com/omerbenamram/evtx can output as JSON and they have binary versions available for download. On my Macbook where I've got the pbcopy command available, I can import the sample data from the start of this issue by first doing:

$ ./evtx_dump-v0.8.1-x86_64-apple-darwin -o json --dont-show-record-number Sysmon_SlothFulMedia.evtx | pbcopy

Once it's in my paste buffer like that, I can use the File > Paste Data pull-down menu option in Zui (or associated hot key) and the data will immediately come up in Preview & Load for optional shaping and then adding into a pool. So, that's not quite as fast as a direct drag into the app, but it's darn close. For OSes that may lack the equivalent of pbcopy, it's not too much harder to have the evtx_dump tool put its output into a temporary file and then drag that into Zui.

Regardless, we'll eventually get to checking out one of the Go libraries for handling this even quicker. Thanks everyone for being patient.