Integrate winevt-kb extract and export functionality into plaso tool

joachimmetz commented 8 years ago

Per https://github.com/libyal/winevt-kb/issues/10 request to keep winevt-kb tooling and plaso closely synced. Best approach is to have the functionality to extract Windows EventLog resources embedded in Plaso.

[x] ~~add pyexe and pywrc dependencies to gift, l2tdevtools and l2tbinaries~~
- ~~GIFT~~
- ~~l2tbinaries win32~~
- ~~l2tbinaries win64~~
- ~~l2tbinaries macosx~~
[x] ~~add WindowsEventLogProviders artifact definition - https://github.com/ForensicArtifacts/artifacts/pull/422~~
[x] ~~add PreprocessingWarning~~
[x] ~~release new version of ForensicArtifacts - https://github.com/ForensicArtifacts/artifacts/releases/tag/20210620~~
[x] ~~extract and store Windows EventLog providers in knowledge base (WindowsEventLogProviderArtifact, WindowsEventLogProvidersPlugin) - https://github.com/log2timeline/plaso/pull/3755~~
[x] ~~Move event log providers out of system configuration - https://github.com/log2timeline/plaso/pull/3819~~
[x] ~~extract EventLog message strings - https://github.com/log2timeline/plaso/pull/3853~~
- looks like pefile has no native support to read the message strings https://github.com/erocarrera/pefile/blob/wiki/ReadingResourceStrings.md
- winevtrc model:
- message files table (columns: file_id, path), message string table [name: file_id+lcid] (columns: message id, string)
- only extract message strings from files defined by event providers
- only extract message strings corresponding to the preferred language
[x] ~~move winevtrc-db and "native" winevtrc into output helper class - https://github.com/log2timeline/plaso/pull/3856~~
[x] ~~output extracted message strings - https://github.com/log2timeline/plaso/pull/3860~~
[x] ~~add log2timeline option to set preferred language - https://github.com/log2timeline/plaso/pull/3869~~
[x] ~~add log2timeline option to disable message string extraction - https://github.com/log2timeline/plaso/pull/3871~~
[x] extract message strings when the winevt or winevtx parser is enabled - https://github.com/log2timeline/plaso/pull/3882
- for now issue a warning if pe parser is disabled

Next steps and improvements captured in https://github.com/log2timeline/plaso/issues/163

rodgermoore commented 7 years ago

Hi @joachimmetz . Is this going to be implemented in the 1.6.0 release? Would be nice 🥇

joachimmetz commented 7 years ago

@rodgermoore this is not likely to get done before the next mid year release (1.6.0)

A couple of things we've dropped the x.y.z version schema and are working on a more frequent release schedule.
We have to overhaul the storage, which should allow us to directly store this and similar information into the plaso storage file

joachimmetz commented 6 years ago

Storage changes have a higher priority at the moment, but should make it easier to implement this in plaso during extraction.

joachimmetz commented 6 years ago

Unlikely to be implemented before Jan 25, bumping release

Onager commented 6 years ago

Not going to make release, removing milestone.

Please do not re-assign to a milestone until this issue has been cleaned up.

joachimmetz commented 3 years ago

In [1]: import pefile                                                                                                                                                                                                                         

In [2]: p = pefile.PE('msobjs.dll.mui')                                                                                                                                                                                                       

In [3]: offset = p.DIRECTORY_ENTRY_RESOURCE.entries[1].directory.entries[0].directory.entries[0].data.struct.OffsetToData                                                                                                                     

In [4]: size = p.DIRECTORY_ENTRY_RESOURCE.entries[1].directory.entries[0].directory.entries[0].data.struct.Size                                                                                                                               

In [5]: data = p.get_memory_mapped_image()[offset:offset + size]

data contains MESSAGE_RESOURCE_DATA

joachimmetz commented 3 years ago

Have a closer look at Microsoft-Windows-Resource-Exhaustion-Detector event identifiers don't specify qualifiers eg. 0x03eb but MUI message file does use them 0xb00003eb

Looks like such message files use 0xb0000000 for the event log messages - should the upper 4 bits be ignored? https://docs.microsoft.com/en-us/windows/win32/eventlog/event-identifiers

Microsoft-Windows-Resource-Exhaustion-Detector has a "traditional" event log provider definition but also defines

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\{9988748e-c2e8-4054-85f6-0c3e1cad2470}
(default): [REG_SZ] Microsoft-Windows-Resource-Exhaustion-Detector
MessageFileName: [REG_EXPAND_SZ] %SystemRoot%\system32\radardt.dll
ResourceFileName: [REG_EXPAND_SZ] %SystemRoot%\system32\radardt.dll

Certain sources reference Microsoft-Windows-Help but have no "traditional" event log provider definition

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Help" Guid="{DE513A55-C345-438B-9A74-E18CAC5C5CC5}"/>
    <EventID>2001</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>21</Task>
    <Opcode>1</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2012-05-16T11:45:44.347200000Z"/>
    <EventRecordID>1</EventRecordID>
    <Correlation/>
    <Execution ProcessID="932" ThreadID="1068"/>
    <Channel>Microsoft-Windows-Help/Operational</Channel>
    <Computer>test-PC</Computer>
    <Security UserID="S-1-5-18"/>
  </System>
  <EventData/>
</Event>

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\{de513a55-c345-438b-9a74-e18cac5c5cc5}
(default): [REG_SZ] Microsoft-Windows-Help
MessageFileName: [REG_EXPAND_SZ] %SystemRoot%\System32\apds.dll
ResourceFileName: [REG_EXPAND_SZ] %SystemRoot%\System32\apds.dll

joachimmetz commented 3 years ago

how to handle an event provider defined as event source:

Microsoft-Windows-Resource-Exhaustion-Detector  System          %SystemRoot%\system32\radardt.dll

and winevt publisher

Microsoft-Windows-Resource-Exhaustion-Detector                  %SystemRoot%\system32\radardt.dll

Looks like the 0xb0000000 mask is also used

10073|124|1033|2952791017|The Windows Resource Exhaustion Detector started.|0xb00003e9
10074|124|1033|2952791018|The Windows Resource Exhaustion Detector stopped.|0xb00003ea
10075|124|1033|2952791019|The Windows Resource Exhaustion Detector received a notification that the computer is low on virtual memory.|0xb00003eb
10076|124|1033|2952791021|The Windows Resource Exhaustion Detector failed to start due to an error.|0xb00003ed
10077|124|1033|2952791022|The Windows Resource Exhaustion Detector failed to stop due to an error.|0xb00003ee
10078|124|1033|2952791023|The Windows Resource Exhaustion Detector experienced a memory allocation failure.|0xb00003ef
10079|124|1033|2952791024|Windows failed to diagnose a low virtual memory condition.|0xb00003f0

joachimmetz commented 3 years ago

Regarding the use of 0xb0000000 it seems to be mixed e.g.

Log source              : Microsoft-Windows-Application-Experience
Identifier              : {eef54e71-0661-422d-9a98-82fd4940b820}
Log type                : Application
Event message files     : %SystemRoot%\system32\aeevts.dll

2012-05-16T11:04:41.000000+00:00,Content Modification Time,EVT,WinEVTX,[201 / 0x00c9] Source Name: Microsoft-Windows-Application-Experience [Provider identifier: {eef54e71-0661-422d-9a98-82fd4940b820}] [Message identifier: 0x000000c9] Strings: [] Computer Name: test-PC Record Number: 1486 Event Level: 4,winevtx,NTFS:\Windows\System32\winevt\Logs\System.evtx,-

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Application-Experience" Guid="{EEF54E71-0661-422D-9A98-82FD4940B820}"/>
    <EventID>201</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2012-05-16T11:04:41.440000000Z"/>
    <EventRecordID>1486</EventRecordID>
    <Correlation/>
    <Execution ProcessID="768" ThreadID="2784"/>
    <Channel>System</Channel>
    <Computer>test-PC</Computer>
    <Security UserID="S-1-5-18"/>
  </System>
  <EventData/>
</Event>

138669|103|1033|201|The Program Compatibility Assistant service started successfully.|0x000000c9
138691|103|1033|2952790223|The Program Compatibility Assistant was requested to monitor {0:s}, but ignored the request because the application is excluded in the registry.|0xb00000cf

joachimmetz commented 3 years ago

Also seen 0xb1000000:

140344|128|1033|2969570237|A summary of the Client Side Caching counters has been generated. The counter list can be found in the event details.|0xb1000bbd

  <System>
    <Provider Name="Microsoft-Windows-BranchCacheSMB" Guid="{4A933674-FB3D-4E8D-B01D-17EE14E91A3E}"/>
    <EventID>3005</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000800</Keywords>
    <TimeCreated SystemTime="2012-05-16T09:59:40.199600000Z"/>
    <EventRecordID>1</EventRecordID>
    <Correlation/>
    <Execution ProcessID="880" ThreadID="1076"/>
    <Channel>Microsoft-Windows-BranchCacheSMB/Operational</Channel>
    <Computer>37L4247D28-05</Computer>
    <Security UserID="S-1-5-18"/>
  </System>

Is this related to channels?

Key path: CMI-CreateHive{3D971F19-49AB-4000-8D39-A6D9C673D809}\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\{4a933674-fb3d-4e8d-b01d-17ee14e91a3e}\ChannelReferences\0
Name: 0
Last written time: Jul 14, 2009 07:51:33.220737800 UTC

Value: 0 (default)
Type: string (REG_SZ)
Data size: 90
Data: Microsoft-Windows-BranchCacheSMB/Operational

Value: 1 Id
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 16

Value: 2 Flags
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 0

Key path: CMI-CreateHive{3D971F19-49AB-4000-8D39-A6D9C673D809}\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\{4a933674-fb3d-4e8d-b01d-17ee14e91a3e}\ChannelReferences\1
Name: 1
Last written time: Jul 14, 2009 07:51:33.220737800 UTC

Value: 0 (default)
Type: string (REG_SZ)
Data size: 84
Data: Microsoft-Windows-BranchCacheSMB/Analytic

Value: 1 Id
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 17

Value: 2 Flags
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 0

joachimmetz commented 3 years ago

Maybe related to 2 event providers using the same message resource file?

Log source              : Microsoft-Windows-OfflineFiles
Identifier              : {95353826-4fbe-41d4-9c42-f521c6e86360}
Log type                : System
Event message files     : %systemroot%\system32\cscsvc.dll

Log source              : Microsoft-Windows-BranchCacheSMB
Identifier              : {4a933674-fb3d-4e8d-b01d-17ee14e91a3e}
Event message files     : %systemroot%\system32\cscsvc.dll

Link to WEVT_TEMPLATE PE/COFF resource? Is possible

libfwevt_event_read: identifier                                         : 0x0001
libfwevt_event_read: version                                            : 0
libfwevt_event_read: channel                                            : 16
libfwevt_event_read: level                                              : 4
libfwevt_event_read: opcode                                             : 0
libfwevt_event_read: task                                               : 0
libfwevt_event_read: keywords                                           : 0x4000000000000010
libfwevt_event_read: message identifier                                 : 0xb0000001
libfwevt_event_read: template offset                                    : 0x00000000
libfwevt_event_read: opcode offset                                      : 0x000033b4
libfwevt_event_read: level offset                                       : 0x000033fc
libfwevt_event_read: task offset                                        : 0x00000000
libfwevt_event_read: unknown3                                           : 0x00000001
libfwevt_event_read: unknown4                                           : 0x00003c28
libfwevt_event_read: flags                                              : 0x000000bc

More research is needed here added to https://github.com/log2timeline/plaso/issues/163

joachimmetz commented 3 years ago

EventLog provides with multiple names:

Log source              : Microsoft-Windows-WMI
Identifier              : {1edeee53-0afe-4609-b846-d8c0b2075b1f}
Event message files     : %SystemRoot%\system32\wbem\WinMgmtR.dll

Log source              : WinMgmt
Identifier              : {1edeee53-0afe-4609-b846-d8c0b2075b1f}
Log type                : Application

joachimmetz commented 3 years ago

Note to self to check for unsupported EventLog providers

grep '\[Message identifier: ' output.log | grep -v ' Message string: ' | sed 's/^.* Source Name: //;s/ \[Message identifier: .*$//' | sort | uniq

log2timeline / plaso

Integrate winevt-kb extract and export functionality into plaso tool #636