Feature Request: ET(W)L Parser

[johnmccash]

See Nicole Ibrahim's presentation: "Windows Forensics: Event Trace Logs", presented at the SANS 2018 DFIR Summit. (Posted at https://www.sans.org/summit-archives/file/summit_archive_1528388048.pdf)

[joachimmetz]

@Onager and myself talked about supporting this format before, I see we did not file an issue yet. Thx for filing.

[joachimmetz]

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363668(v=vs.85).aspx

Looks like this one of those rabit hole Windows formats that allow you to define customs event formats https://msdn.microsoft.com/en-us/library/windows/desktop/aa364099(v=vs.85).aspx

[johnmccash]

I was hoping the portion containing the timestamps might be standardized. If that's the case, then you could just parse that, and return an entry with a reference to the file where it was found. That would provide information about where additional data related to things that were going on during a specific timeframe might be found. Thoughts?

[johnmccash]

I asked Nicole after the presentation whether all these records contain a timestamps, and she said they did. That's what gave me hope that the timestamps might be independently parseable, and the rest of the record could be ignored if it wasn't one of the more common formats.

[joachimmetz]

Thoughts?

I would need to take a closer look at the format first. IMHO having some context around a timestamp is useful.

[johnmccash]

Was just discussing this with Nicole, and she confirms it should be possible to parse the beginning portion of the record, including the timestamps and a number of other fields, and include either the remainder of the record, or the full record as unparsed hex code, which can be extracted by the examiner and used as input to some other tool for full extraction.

[joachimmetz]

I understand this could be technically possible, but the question here would be how should plaso handle such "events".

What do we know about the providers/consumers, do you know of any documentation/research on them? Any indication of where the ETWL event schemas are stored?

[johnmccash]

There's a python parser for some of them at http://www.hecfblog.com/2018/06/etw-event-tracing-for-windows-and-etl.html?m=1

[johnmccash]

Nicole also indicated these files are eminently carveable, and appear frequently in unallocated space, so we may want to parse these records there as well...

[johnmccash]

Oh, and here's a link to Nicole's python parser for these files: https://github.com/gcpartners/ETLParser

[joachimmetz]

Oh, and here's a link to Nicole's python parser for these files: https://github.com/gcpartners/ETLParser

I think you meant: https://github.com/sebmarchand/pyetw which looks like to use the Windows subsystem and possibly not portable.

ETLParser only seems to be an executable and a readme, no license and not portable.

So as is looks now, there is little information about the actual format available. So this would mean that to write a ETWL parser format analysis would need to be done.

so we may want to parse these records there as well...

Recovery of event from unallocated space is on the long-term wish list.

[joachimmetz]

Another reference: http://www.hexacorn.com/blog/2018/06/09/logman-the-windows-volverine/

[Onager]

Possibly useful library to build from: https://github.com/TracingTools/libtrace/

[Onager]

Some additional information and binary templates: https://github.com/forensenellanebbia/xways-forensics/blob/master/my-templates/ETL_Header_x64.tpl

[Zhentar]

I've been putting together an 010 Editor binary template for parsing ETL files; my work so far is available here: https://gist.github.com/Zhentar/b77174141734c9407cb08ffd4392b809 For Windows Vista+ traces, it fully parses the buffer structure & all known event header types, which I believe makes it the most coherent documentation of the file structure outside of the Windows source tree* **. It doesn't do any event data parsing, but that's a very different, and mostly better documented, task as it's required even by the public APIs.

* Not that that's a particularly high bar to meet ** https://www.geoffchappell.com/ certainly has me beat on completeness (indeed, it was my primary reference for writing this), but its structure isn't ETL file parsing oriented

[aaptel]

I was googling how to parse ETL files and found this thread. @Zhentar fantastic work!! I've opened an ETL file of mine in 010 with your script and in MessageAnalyzer side-by-side to double check.

I thought there were some issues because the SeqNo/MessageNumber in Message Analyzer do not match the indexes in 010 but actually after looking for specific entries via their ProviderId or header size I was able to match them: the offset can be quite big, e.g. SeqNo 17 ends up as the event at EtwBuffer[1].events[113] in 010. I haven't checked yet but my guess is MessageAnalyzer SeqNo are generated after loading and sorting all events by their timestamp. I'll try to update but now it's time to head to bed lol.

[aaptel]

@Zhentar I've noticed that your GUID struct in ETL.bt is only 12 bytes long but GUID are actually 16 bytes long.

[Zhentar]

Thanks :)

I haven't checked yet but my guess is MessageAnalyzer SeqNo are generated after loading and sorting all events by their timestamp.

Yes, that is almost certainly correct. By default, kernel tracing uses per-CPU buffers (IIRC since Windows 8), so there's no guarantee of chronological ordering across buffers. Don't think there's even such a guarantee within individual buffers, though it probably requires a relogger altering events to end up that way.

[aaptel]

Another thing I found out: WMI_HEADER_BUFFER timestamp is the "base" timestamp (in FILETIME format) and all events within (at least for the type EVENT_HEADER) values are offsets to that time. So to get individual events timestamps you have to do wmi_timestamp + event_timestamp

[aaptel]

I wrote a script to extract and convert network traffic from ETL files to PCAP.

https://github.com/aaptel/etl2pcap

[aaptel]

To anyone looking to parse actual events after getting past the initial basic blocks from @Zhentar description: open the trace in MessageAnalyzer and completely expand the details of the type of events you are interested in.

In the bottom pane you can see each field type and offsets:

The offsets and lenghts are in bit so don't forget to divide by 8 to get bytes.

[bf]

Thanks for the detailed posts. I see ETL/ETW file format still being used by Microsoft Intune Diagnostic logs that are uploaded to the Intune Admin Web Portal, so the issue is still relevant in 2023.

Is there a current plan to implement this? Thank you!

[joachimmetz]

Is there a current plan to implement this? Thank you!

So as is looks now, there is little information about the actual format available. So this would mean that to write a ETWL parser format analysis would need to be done.

Has this changed?

[bf]

It seems there are quite some windows-based GitHub projects offering a viewer but I am not sure what kind of backend they are using.

---

From: Joachim Metz @.> Sent: Thursday, September 14, 2023 4:45:59 PM To: log2timeline/plaso @.> Cc: Benjamin Flesch @.>; Comment @.> Subject: Re: [log2timeline/plaso] Feature Request: ET(W)L Parser (#1933)

Is there a current plan to implement this? Thank you!

So as is looks now, there is little information about the actual format available. So this would mean that to write a ETWL parser format analysis would need to be done.

Has this changed?

— Reply to this email directly, view it on GitHubhttps://github.com/log2timeline/plaso/issues/1933#issuecomment-1719599861, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AADRNGM5GQB6M7RQWOZYAH3X2MKCPANCNFSM4FD3BCFQ. You are receiving this because you commented.Message ID: @.***>

Mit freundlichen Grüßen, With Kind Regards, Dr. Benjamin Flesch

DCB offers MSP/MSSP services for costumers with Microsoft E3 or E5 licenses. We solve security, forensics and data science problems.

This email is confidential and may be privileged. If you have received it in error, please notify us immediately and then delete it. Please do not copy it, disclose its contents or use it for any purpose.

DCB Deutsche Cyberberatung GmbH www.deutsche-cyberberatung.dehttps://www.deutsche-cyberberatung.de/ Donnersbergweg 1 DE-67059 Ludwigshafen am Rhein Amtsgericht Ludwigshafen HRB 66581 VAT-ID: DE274414961

[joachimmetz]

unfortunately without the necessary research/documentation and representative test files this is a low priority