philrz
commented
9 months ago This topic came up again recently in a community Slack thread when a user asked:
if you make it so people can drag/drop windows evtx (event log) files directly into zui, and it shows up loaded in a pool kind of the way it does for pcaps - I suspect alot of dfir folks are going to start using it
We've been aware of this general EVTX requirement for a while and there's a Zed issue https://github.com/brimdata/zed/issues/2793 tracking its eventual implementation. Indeed, it would seem ideal if support for the format were implemented at the Zed layer so that way the data could also be read with the Zed CLI tooling and drag & drop support into Zui would "come for free".
The reason it's not been implemented yet by the core Zed dev team is simply because there's been other priorities. As that issue points out, there's already existing EVTX readers that could easily turn the format into JSON and that output could be piped onward into Zed tooling, such as a zed load into the lake running behind Zui. However, that's still just short of the 100% drag & drop experience envisioned by the user.
As a proof-of-concept for Zui's plugin system, perhaps we could create an example plugin for EVTX to show a user how to wire up conversion of some arbitrary format using 3rd-party tooling that the app can call out to for drag & drop load of such a format. Even if EVTX is something we eventually support "officially" starting at the Zed layer, this would provide an example users could follow for other formats that would likely never be supported within Zed itself, e.g., a user's proprietary formats.
Such a plugin could also include a way to apply shaping logic as part of the drag & drop, e.g., to turn string-based IP addresses into Zed ip types, string-based timestamps to time types, etc.
When discussing the role of the Brimcap plugin as part of #2785, I thought about how the plugin system was first introduced in #1573 to handle pcaps but hasn't yet been used for other data sources. Given the project's widened scope far beyond just security data, it seems like it would be cool to write a plugin based on some non-security data source that our user base might find interesting. I'd hope it would make a nice way to give more visibility to the plugin system (maybe inspire users to start developing stuff?) and just be useful out-of-the-box and to demo in blog posts & videos.
When discussing this as a team, the first data source I thought of is GitHub. While I've not yet done any prototyping, I could imagine some potentially interesting things it could do in terms of data presentation, e.g., stacked bars in the histogram for open PRs or Issues per repo, correlations between a PR and its related commits, etc. Lots TBD here when we actually take up the effort, but @jameskerr seemed to like the sound of the idea so I figured I'd put it in the backlog for later.