search

libyal / libevtx

Library and tools to access the Windows XML Event Log (EVTX) format
GNU Lesser General Public License v3.0
190 stars 49 forks source link

Unescaped ampersand character in EventXML attribute value output? #30

Closed LatinSuD closed 6 months ago

LatinSuD commented 1 year ago

Hi, I was trying to parse a file and run into an invalid character

The sample was taken from a Microsoft-Windows-DriverFrameworks-UserMode%4Operational.evtx

The XML looks more or less like this:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
  </System>
  <UserData>
    <UMDFHostDeviceArrivalBegin instance="WPDBUSENUMROOT\UMB\2&37C117B&0&STORAGE#VOLUME#_??_USBSTOR#DISK" lifetime="{ABABAB-ABAB-ABAB-ABABA-ABABABAB}" xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="http://www.microsoft.com/DriverFrameworks/UserMode/Event"/>
  </UserData>
</Event>

The & ampersand should be escaped with &amp;

joachimmetz commented 1 year ago

First of all EventXML is not proper XML 1.0 also see https://osdfir.blogspot.com/2021/10/common-misconceptions-about-windows.html "Misconception 6: Event XML is proper XML 1.0"

How does the output compare against the EventXML generated by EventViewer?

LatinSuD commented 1 year ago

This is the output by MS (i only modified some uids)

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Events>
    <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
        <System>
            <Provider Name='Microsoft-Windows-DriverFrameworks-UserMode' Guid='{XXXXXXXXXXXXXXXXXX}'/>
            <EventID>2003</EventID>
            <Version>1</Version>
            <Level>4</Level>
            <Task>33</Task>
            <Opcode>1</Opcode>
            <Keywords>0x8000000000000000</Keywords>
            <TimeCreated SystemTime='2013-06-28TXXXXXXXX'/>
            <EventRecordID>6</EventRecordID>
            <Correlation/>
            <Execution ProcessID='2888' ThreadID='2588'/>
            <Channel>Microsoft-Windows-DriverFrameworks-UserMode/Operational</Channel>
            <Computer>XXXXXXXXXXXXXXXXXX</Computer>
            <Security UserID='S-1-5-19'/>
        </System>
        <UserData>
            <UMDFHostDeviceArrivalBegin instance='WPDBUSENUMROOT\UMB\2&amp;37C186B&amp;0&amp;STORAGE#VOLUME#_??_USBSTOR#DISK&amp;VEN_CORSAIR&amp;PROD_FLASH_VOYAGER&amp;REV_0.00#XXXXXXXXXX&amp;0#' lifetime='{XXXXXXXXXXXXXXXXXXX}' xmlns:auto-ns2='http://schemas.microsoft.com/win/2004/08/events' xmlns='http://www.microsoft.com/DriverFrameworks/UserMode/Event'/>
        </UserData>
    </Event>
</Events>

The funny thing, after I reviewed evtxexport output again, is that the first event had the XML escaped, but only the first one.

joachimmetz commented 1 year ago

I'll have a closer look when time permits, to see if I can reproduce it

koromodako commented 1 year ago

Hi,

I'm encountering the same issue as the one described in "Misconception 6: Event XML is proper XML 1.0" but this time with a <Data>[...]<lambda_[hexadecimal_string]>[...]</Data> in Microsoft-Windows-Store/Operational for provider Microsoft-Windows-Install-Agent.

Microsoft event viewer also displays invalid XML but exporting events as XML produces valid XML 1.0.

I think that libevtx can generate valid XML 1.0 from BinXml by escaping characters using the following rules.

Char Replacement
< &lt;
> &gt;
& &amp;

I do not foresee any harm of doing that kind of substitution except additional overhead (relatively small I suppose)

But if it means having valid XML 1.0, it may be worth the change.

joachimmetz commented 1 year ago

But if it means having valid XML 1.0, it may be worth the change.

EventXML is not proper XML also see https://osdfir.blogspot.com/2021/10/common-misconceptions-about-windows.html for more examples

I think that libevtx can generate valid XML 1.0 from BinXml by escaping characters using the following rules.

this will be insufficient

Microsoft event viewer also displays invalid XML but exporting events as XML produces valid XML 1.0.

Can you elaborate what you mean with this? libevtx tries to mimic the behavior of event viewer (which is the reference implementation AFAIK)

koromodako commented 1 year ago

Can you elaborate what you mean with this? libevtx tries to mimic the behavior of event viewer (which is the reference implementation AFAIK)

Here is the procedure I followed, I found a problematic event in Microsoft-Windows-Store/Operational (it seems that event id 2006 is prone to this kind of inconsistency) and looking at the event in Windows Event Viewer XML View I see:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
  <System>
    <Provider Name='Microsoft-Windows-Install-Agent' Guid='{e0c6f6de-258a-50e0-ac1a-103482d118bc}'/>
    <EventID>2006</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>2002</Task>
    <Opcode>14</Opcode>
    <Keywords>0x8000000020000000</Keywords>
    <TimeCreated SystemTime='2023-09-16T13:45:13.1802522Z'/>
    <EventRecordID>416224</EventRecordID>
    <Correlation/>
    <Execution ProcessID='16072' ThreadID='13912'/>
    <Channel>Microsoft-Windows-Store/Operational</Channel>
    <Computer>COMPUTERNAME</Computer>
    <Security UserID='S-1-5-18'/>
  </System>
  <EventData>
    <Data Name='Message'>result: Microsoft.WindowsStore, CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US, blocked = false</Data>
    <Data Name='Function'>Windows::ApplicationModel::Store::Preview::InstallControl::AppInstallManagerImpl::IsStoreBlockedByPolicyAsync::<lambda_8780db9c3f23660ab3645582dbfa3321>::operator ()</Data>
    <Data Name='Error Code'>0</Data>
    <Data Name='Source'>onecoreuap\enduser\winstore\installservice\lib\appinstallcontrol.cpp</Data>
    <Data Name='Line Number'>1784</Data>
    <Data Name='CorrelationVector'>NULL</Data>
    <Data Name='ProductId'>NULL</Data>
  </EventData>
</Event>

And libevtx seems to give the same data which crashes Python's etree.ElementTree.fromstring because this is not valid XML 1.0 data due to the unclosed tag <lambda_8780db9c3f23660ab3645582dbfa3321>.

Yet when I use Windows Event Viewer feature called Save all events as... in the panel on the right and then in the save file dialog I select Save as type: Xml (Xml File) (*.xml) instead of Event Files (*.evtx), Windows Event Viewer produces a valid XML 1.0 file containing the same event but this time with escaped characters:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
  <System>
    <Provider Name='Microsoft-Windows-Install-Agent' Guid='{e0c6f6de-258a-50e0-ac1a-103482d118bc}'/>
    <EventID>2006</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>2002</Task>
    <Opcode>14</Opcode>
    <Keywords>0x8000000020000000</Keywords>
    <TimeCreated SystemTime='2023-09-16T13:45:13.1802522Z'/>
    <EventRecordID>416224</EventRecordID>
    <Correlation/>
    <Execution ProcessID='16072' ThreadID='13912'/>
    <Channel>Microsoft-Windows-Store/Operational</Channel>
    <Computer>COMPUTERNAME</Computer>
    <Security UserID='S-1-5-18'/>
  </System>
  <EventData>
    <Data Name='Message'>result: Microsoft.WindowsStore, CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US, blocked = false</Data>
    <Data Name='Function'>Windows::ApplicationModel::Store::Preview::InstallControl::AppInstallManagerImpl::IsStoreBlockedByPolicyAsync::&lt;lambda_8780db9c3f23660ab3645582dbfa3321&gt;::operator ()</Data>
    <Data Name='Error Code'>0</Data>
    <Data Name='Source'>onecoreuap\enduser\winstore\installservice\lib\appinstallcontrol.cpp</Data>
    <Data Name='Line Number'>1784</Data>
    <Data Name='CorrelationVector'>NULL</Data>
    <Data Name='ProductId'>NULL</Data>
  </EventData>
</Event>

Microsoft developers may have chosen to display something invalid but human readable in XML View. When exporting events as XML it produces valid XML 1.0 data, some characters are escaped &lt;lambda_8780db9c3f23660ab3645582dbfa3321&gt;.

this will be insufficient

I don't think so because it seems that Microsoft Event Viewer does this substitution when using the Save all events as... feature and selecting XML as the saved file format.

joachimmetz commented 1 year ago 
Yet when I use Windows Event Viewer feature called Save all events as... in the panel on the right and then in the save file dialog I select Save as type: Xml (Xml File) (*.xml) instead of Event Files (*.evtx), Windows Event Viewer produces a valid XML 1.0 file containing the same event but this time with escaped characters:

Interesting observation, that would imply that EventXML is not a single format. I would need to double check my notes, but this might have changed with different version of EventViewer. The challenge is that most people use wevtutil to export Event Logs, wondering what its behavior is.

I don't think so because it seems that Microsoft Event Viewer does this substitution when using the Save all events as... feature and selecting XML as the saved file format.

ah so there are more issues such as control characters and very likely invalid Unicode characters that Windows can represent but a strict UTF-8 XML 1.0 parser does not allow.

koromodako commented 1 year ago

The challenge is that most people use wevtutil to export Event Logs, wondering what its behavior is.

Interesting I had not tested until now and it turns out to give the same result as Save all events as... using XML format. I used the following command to perform the export

wevtutil.exe qe "Microsoft-Windows-Store/Operational" | Out-File -Encoding utf8 .\test.xml

ah so there are more issues such as control characters and very likely invalid Unicode characters that Windows can represent but a strict UTF-8 XML 1.0 parser does not allow.

Yes indeed, the issue might be trickier to solve than I expected it to be. Though, if the goal is to mimic eventvwr/wevtutil behavior, a fix might be needed eventually.

By the way, many thanks for libyal project, it is a really valuable resource when studying Windows forensic artifacts and automating processing of these artifacts

joachimmetz commented 6 months ago

Observed behavior Windows 10 EventViewer XML view represents a & character in an element value as its character, "save as XML" and wevtutil will convert to &amp; (same for &lt; and &gt;).

However evtxexport already escapes EventXML element values.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Store" Guid="{9C2A37F3-E5FD-5CAE-BCD1-43DAFEEE1FF0}"/>
    <EventID>8001</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>8001</Task>
    <Opcode>14</Opcode>
    <Keywords>0x8000100000000000</Keywords>
    <TimeCreated SystemTime="2023-06-17T20:36:48.215941000Z"/>
    <EventRecordID>7695892690</EventRecordID>
    <Correlation/>
    <Execution ProcessID="6980" ThreadID="2592"/>
    <Channel>Microsoft-Windows-Store/Operational</Channel>
    <Computer>test</Computer>
    <Security UserID="S-1-5-18"/>
  </System>
  <EventData>
    <Data Name="Message">Decoding: {"entitlementId":"7947ee06-ffd8-4878-989d-7f5798510f2e","entitlementSatisfaction":"Device","isOffline":true,"leaseEnforceme
nt":"None","leaseUri":"https://licensing.md.mp.microsoft.com/v7.0/licenses/?beneficiaryId=msa%3a985156904265117&amp;contentId=eee53744-2bb9-bca2-a50a-c6a1c5b0
a0ed&amp;entitlementId=7947ee06-ffd8-4878-989d-7f5798510f2e&amp;market=US&amp;policyType=Device","keyIds":["0f8e2cd5-b8eb-7a22-b9e9-9b1183fa0a84"],"kind":"Con
tent","packages":[{"packageIdentifier":"eee53744-2bb9-bca2-a50a-c6a1c5b0a0ed","packageType":"msix","productAddOns":[],"productId":"9NMPJ99VJBWV","skuId":"0010
"}],"pollAt":"2022-08-23T23:35:03.9798794+00:00","refreshOnStartup":false,"version":7}</Data>
    <Data Name="Function">DecodeCustomPolicy</Data>
    <Data Name="Source">onecoreuap\enduser\winstore\licensemanager\lib\clipdocument.cpp</Data>
    <Data Name="Line Number">50</Data>
  </EventData>
</Event>

So the issue you describe only appears to be specific EventXML attribute values

The funny thing, after I reviewed evtxexport output again, is that the first event had the XML escaped, but only the first one.

unable to reproduce with current version see output above

joachimmetz commented 6 months ago

Observed behavior Windows 10 EventViewer XML view represents a & character in an attribute value as its character wevtutil will convert to &amp;.

It looks like evtxexport does not, so I'll update that behavior.

joachimmetz commented 6 months ago

Changes in https://github.com/libyal/libfwevt/commit/18edea0655cb08cb778120248c0ca61a013e6874