Open john-corcoran opened 6 years ago
hey @john-corcoran
Thanks for raising this issue. The easiest way to triage it will be to share the .evtx file. I'd be happy to take a peek if you send to my email: willi.ballenthin@gmail.com
file structure looks ok:
λ C:\Users\user\env2\Scripts\python ..\scripts\evtx_info.py data\issue_46.evtx
Information from file header:
Format version : 3.1
Flags : 0x00000000
File is : clean
Log is full : no
Current chunk : 3 of 16
Oldest chunk : 5
Next record# : 521
Check sum : pass
Information from chunks:
Chunk file (first/last) log (first/last) Header Data
- ----- --------------------- --------------------- ------ ------
1 417 441 417 441 pass pass
2 442 466 442 466 pass pass
3 467 499 467 499 pass pass
* 4 500 520 500 520 pass pass
> 5 104 129 104 129 pass pass
6 130 150 130 150 pass pass
7 151 180 151 180 pass pass
8 181 205 181 205 pass pass
9 206 226 206 226 pass pass
10 227 252 227 252 pass pass
11 253 283 253 283 pass pass
12 284 304 284 304 pass pass
13 305 335 305 335 pass pass
14 336 360 336 360 pass pass
15 361 387 361 387 pass pass
16 388 416 388 416 pass pass
Seems like the very first record in the evtx file is problematic:
magic: 0x2a2a
size: 0x17a0
num: 417
ts: 2017-11-14T11:32:41.189631
substitutions count: 20
substitutions count: 1124080761
Clearly 1124080761 is not a reasonable number of substitutions.
there's a root note being parsed at 0x1836, with substitution count being parsed at 0x1844:
19:44:21,0731 [root :1046][INFO ] root node: 0x1836
19:44:21,0732 [root :1047][INFO ] ofs: 0xe (abs: 0x1844)
19:44:21,0732 [root :1048][INFO ] sub count: 1124080761
but this doesn't look quite right:
1830h: 00 05 12 00 00 00 01 5C 11 00 00 3F 08 00 00 00 .......\...?....
1840h: 00 00 00 B4 79 1C 00 43 00 72 00 79 00 70 00 74 ...´y..C.r.y.p.t
1850h: 00 52 00 65 00 74 00 72 00 69 00 65 00 76 00 65 .R.e.t.r.i.e.v.e
1860h: 00 4F 00 62 00 6A 00 65 00 63 00 74 00 42 00 79 .O.b.j.e.c.t.B.y
1870h: 00 55 00 72 00 6C 00 57 00 69 00 72 00 65 00 00 .U.r.l.W.i.r.e..
1880h: 00 02 41 CB 00 00 00 8B 08 00 00 00 00 00 00 4F ..AË...‹.......O
the dword at 0x1844 is 0x43001C79, which is crazy. a better interpretation of these bytes is the byte at 0x1845 (value: 0x1C) is number of wchar in the utf-16le string (number of chars: 0x1C) that begins at 0x1847.
need to trace down where the parsing is faulty that leads to these incorrect fields.
file structure:
λ C:\Users\user\env2\Scripts\python ..\scripts\evtx_structure.py data\issue_46.evtx | head -n 400
File header
magic: ElfFile
oldest_chunk: 0x4
current_chunk_number: 0x3
next_record_number: 0x209
header_size: 0x80
minor_version: 0x1
major_version: 0x3
header_chunk_size: 0x1000
chunk_count: 0x10
flags: 0x0
checksum: 0x1d883bed
verify: True
dirty: False
full: False
Chunk
offset: 0x1000
magic: ElfChnk
file_first_record_number: 0x1a1
file_last_record_number: 0x1b9
log_first_record_number: 0x1a1
log_last_record_number: 0x1b9
header_size: 0x80
last_record_offset: 0xdf80
next_record_offset: 0xfe90
data_checksum: 0x20c08819
header_checksum: 0xf0aadf4cL
verify: True
templates: 1
Record
offset: 0x1200
magic: 0x2a2a
size: 0x17a0
number: 0x1a1
timestamp: 2017-11-14 11:32:41.189631
verify: True
RootNode(offset=0x18)
StreamStartNode(offset=0x18)
TemplateInstanceNode(offset=0x1c, resident=True, length=0x571)
TemplateNode(offset=0x26)
StreamStartNode(offset=0x3e)
OpenStartElementNode(offset=0x42) --> Event
AttributeNode(offset=0x65) --> xmlns
ValueNode(offset=0x7e)
WstringTypeNode(offset=0x80) --> http://schemas.microsoft.com/win/2004/08/events/event
CloseStartElementNode(offset=0xec)
OpenStartElementNode(offset=0xed) --> System
CloseStartElementNode(offset=0x10e)
OpenStartElementNode(offset=0x10f) --> Provider
AttributeNode(offset=0x138) --> Name
ValueNode(offset=0x14f)
WstringTypeNode(offset=0x151) --> Microsoft-Windows-CAPI2
AttributeNode(offset=0x181) --> Guid
ValueNode(offset=0x198)
WstringTypeNode(offset=0x19a) --> {5bbca4a8-b209-48dc-a8c7-b23d3e5216fb}
CloseEmptyElementNode(offset=0x1e8)
OpenStartElementNode(offset=0x1e9) --> EventID
AttributeNode(offset=0x210) --> Qualifiers
ConditionalSubstitutionNode(offset=0x233)
CloseStartElementNode(offset=0x237)
ConditionalSubstitutionNode(offset=0x238)
CloseElementNode(offset=0x23c)
OpenStartElementNode(offset=0x23d) --> Version
CloseStartElementNode(offset=0x260)
ConditionalSubstitutionNode(offset=0x261)
CloseElementNode(offset=0x265)
OpenStartElementNode(offset=0x266) --> Level
CloseStartElementNode(offset=0x285)
ConditionalSubstitutionNode(offset=0x286)
CloseElementNode(offset=0x28a)
OpenStartElementNode(offset=0x28b) --> Task
CloseStartElementNode(offset=0x2a8)
ConditionalSubstitutionNode(offset=0x2a9)
CloseElementNode(offset=0x2ad)
OpenStartElementNode(offset=0x2ae) --> Opcode
CloseStartElementNode(offset=0x2cf)
ConditionalSubstitutionNode(offset=0x2d0)
CloseElementNode(offset=0x2d4)
OpenStartElementNode(offset=0x2d5) --> Keywords
CloseStartElementNode(offset=0x2fa)
ConditionalSubstitutionNode(offset=0x2fb)
CloseElementNode(offset=0x2ff)
OpenStartElementNode(offset=0x300) --> TimeCreated
AttributeNode(offset=0x32f) --> SystemTime
ConditionalSubstitutionNode(offset=0x352)
CloseEmptyElementNode(offset=0x356)
OpenStartElementNode(offset=0x357) --> EventRecordID
CloseStartElementNode(offset=0x386)
ConditionalSubstitutionNode(offset=0x387)
CloseElementNode(offset=0x38b)
OpenStartElementNode(offset=0x38c) --> Correlation
AttributeNode(offset=0x3bb) --> ActivityID
ConditionalSubstitutionNode(offset=0x3de)
AttributeNode(offset=0x3e2) --> RelatedActivityID
ConditionalSubstitutionNode(offset=0x413)
CloseEmptyElementNode(offset=0x417)
OpenStartElementNode(offset=0x418) --> Execution
AttributeNode(offset=0x443) --> ProcessID
ConditionalSubstitutionNode(offset=0x464)
AttributeNode(offset=0x468) --> ThreadID
ConditionalSubstitutionNode(offset=0x487)
CloseEmptyElementNode(offset=0x48b)
OpenStartElementNode(offset=0x48c) --> Channel
CloseStartElementNode(offset=0x4af)
ValueNode(offset=0x4b0)
WstringTypeNode(offset=0x4b2) --> Microsoft-Windows-CAPI2/Operational
CloseElementNode(offset=0x4fa)
OpenStartElementNode(offset=0x4fb) --> Computer
CloseStartElementNode(offset=0x520)
ValueNode(offset=0x521)
WstringTypeNode(offset=0x523) --> exploited
CloseElementNode(offset=0x537)
OpenStartElementNode(offset=0x538) --> Security
AttributeNode(offset=0x561) --> UserID
ConditionalSubstitutionNode(offset=0x57c)
CloseEmptyElementNode(offset=0x580)
CloseElementNode(offset=0x581)
OpenStartElementNode(offset=0x582) --> UserData
CloseStartElementNode(offset=0x5a7)
ConditionalSubstitutionNode(offset=0x5a8)
CloseElementNode(offset=0x5ac)
CloseElementNode(offset=0x5ad)
EndOfStreamNode(offset=0x5ae)
Substitutions(offset=0x5af)
UnsignedByteTypeNode(offset=0x603) --> 2
UnsignedByteTypeNode(offset=0x604) --> 2
UnsignedWordTypeNode(offset=0x605) --> 53
UnsignedWordTypeNode(offset=0x607) --> 53
NullTypeNode(offset=0x609)
Hex64TypeNode(offset=0x609) --> 0x4000000000000036
FiletimeTypeNode(offset=0x611) --> 2017-11-14 11:32:41.189631
NullTypeNode(offset=0x619)
UnsignedDwordTypeNode(offset=0x619) --> 2856
UnsignedDwordTypeNode(offset=0x61d) --> 608
UnsignedQwordTypeNode(offset=0x621) --> 417
UnsignedByteTypeNode(offset=0x629) --> 0
SIDTypeNode(offset=0x62a) --> S-1-5-18
NullTypeNode(offset=0x636)
NullTypeNode(offset=0x636)
NullTypeNode(offset=0x636)
NullTypeNode(offset=0x636)
NullTypeNode(offset=0x636)
NullTypeNode(offset=0x636)
BXmlTypeNode(offset=0x636) -->
RootNode(offset=0x636)
ERROR: 0
Record
offset: 0x29a0
magic: 0x2a2a
size: 0x790
number: 0x1a2
timestamp: 2017-11-14 11:32:41.189631
verify: True
RootNode(offset=0x18)
StreamStartNode(offset=0x18)
TemplateInstanceNode(offset=0x1c, resident=False)
Substitutions(offset=0x26)
UnsignedByteTypeNode(offset=0x7a) --> 4
UnsignedByteTypeNode(offset=0x7b) --> 2
UnsignedWordTypeNode(offset=0x7c) --> 41
UnsignedWordTypeNode(offset=0x7e) --> 41
NullTypeNode(offset=0x80)
Hex64TypeNode(offset=0x80) --> 0x4000000000000005
FiletimeTypeNode(offset=0x88) --> 2017-11-14 11:32:41.189631
NullTypeNode(offset=0x90)
UnsignedDwordTypeNode(offset=0x90) --> 2856
UnsignedDwordTypeNode(offset=0x94) --> 608
UnsignedQwordTypeNode(offset=0x98) --> 418
UnsignedByteTypeNode(offset=0xa0) --> 0
SIDTypeNode(offset=0xa1) --> S-1-5-18
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
BXmlTypeNode(offset=0xad) -->
RootNode(offset=0xad)
ERROR: Tried to parse beyond the end of the file (read: 0x160c1006, buffer length: 0x101000)
Record
offset: 0x3130
magic: 0x2a2a
size: 0x1738
number: 0x1a3
timestamp: 2017-11-14 11:32:41.189631
verify: True
RootNode(offset=0x18)
StreamStartNode(offset=0x18)
TemplateInstanceNode(offset=0x1c, resident=False)
Substitutions(offset=0x26)
UnsignedByteTypeNode(offset=0x7a) --> 4
UnsignedByteTypeNode(offset=0x7b) --> 2
UnsignedWordTypeNode(offset=0x7c) --> 11
UnsignedWordTypeNode(offset=0x7e) --> 11
NullTypeNode(offset=0x80)
Hex64TypeNode(offset=0x80) --> 0x4000000000000003
FiletimeTypeNode(offset=0x88) --> 2017-11-14 11:32:41.189631
NullTypeNode(offset=0x90)
UnsignedDwordTypeNode(offset=0x90) --> 2856
UnsignedDwordTypeNode(offset=0x94) --> 608
UnsignedQwordTypeNode(offset=0x98) --> 419
UnsignedByteTypeNode(offset=0xa0) --> 0
SIDTypeNode(offset=0xa1) --> S-1-5-18
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
NullTypeNode(offset=0xad)
BXmlTypeNode(offset=0xad) -->
RootNode(offset=0xad)
ERROR: Tried to parse beyond the end of the file (read: 0x5341006, buffer length: 0x101000)
I have the same problem, but in this instance, I can't share the EVTX files for confidentiality reasons. I have the same parsing problem. I tried to follow some of your steps above for leads, and I will paste the output below. I'm running python 3.7.1.
python D:\tools\Parse-Evtx\scripts\evtx_info.py "Microsoft-Windows-TerminalServices-LocalSessionManager%4Operational.evtx"
Information from file header:
Format version : 3.1
Flags : 0x00000001
File is : dirty
Log is full : no
Current chunk : 8 of 9
Oldest chunk : 1
Next record# : 1018
Check sum : pass
Traceback (most recent call last):
File "D:\tools\Parse-Evtx\scripts\evtx_info.py", line 114, in <module>
main()
File "D:\tools\Parse-Evtx\scripts\evtx_info.py", line 57, in main
chunk_count = sum([1 for c in fh.chunks() if c.verify()])
File "D:\tools\Parse-Evtx\scripts\evtx_info.py", line 57, in <listcomp>
chunk_count = sum([1 for c in fh.chunks() if c.verify()])
File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 333, in verify
return self.check_magic() and \
File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 308, in check_magic
return self.magic() == "ElfChnk\x00"
File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 211, in explicit_length_handler
return f(offset, length)
File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 475, in unpack_string
return self.unpack_binary(offset, length).decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 4: ordinal not in range(128)
Here is the excerpt from evtx_structure.py where it failed.
Chunk
offset: 0x91000
magic: ☻ ☻
file_first_record_number: 0x20048
file_last_record_number: 0x12c65ab
log_first_record_number: 0x2
log_last_record_number: 0x0
header_size: 0x20800
last_record_offset: 0x3
next_record_offset: 0x20000
data_checksum: 0x0
header_checksum: 0x0
verify: False
Unexpected token encountered
Unexpected token encountered
Traceback (most recent call last):
File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 273, in unpack_byte
return struct.unpack_from("<B", self._buf, o)[0]
struct.error: unpack_from requires a buffer of at least 1 bytes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 185, in <module>
main()
File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 180, in main
for line in formatter.format_header(log.get_file_header()):
File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 66, in format_header
for line in self.format_chunk(chunk):
File "D:\tools\Parse-Evtx\scripts\evtx_structure.py", line 89, in format_chunk
yield self._l("templates: %d" % (len(chunk.templates())))
File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 411, in templates
self._load_templates()
File "C:\Python3\lib\site-packages\Evtx\Evtx.py", line 380, in _load_templates
token = self.unpack_byte(ofs - 10)
File "C:\Python3\lib\site-packages\Evtx\BinaryParser.py", line 275, in unpack_byte
raise OverrunBufferException(o, len(self._buf))
Evtx.BinaryParser.OverrunBufferException: Tried to parse beyond the end of the file (read: 0x43dbfc, buffer length: 0x101000)
Experiencing the same exact issues. Any recommendations?
@williballenthin I believe this evt(x) file causes the same exception: https://send.firefox.com/download/1804ced0a1ad523c/#A_E9qDRE3rU1poCUk_zq4g
I hope this helps.
(it's produced by wusa.exe)
Hi,
I'm seeing exception Evtx.BinaryParser.OverrunBufferException raised unexpectedly when parsing event logs "Microsoft-Windows-Windows Firewall With Advanced Security%4Firewall.evtx", "Microsoft-Windows-WindowsUpdateClient%4Operational.evtx", and "Microsoft-Windows-CAPI2%4Operational.evtx" (all extracted from a Windows 7 testing instance) using evtx_dump.py.
For Microsoft-Windows-Windows Firewall With Advanced Security%4Firewall.evtx, the same behaviour is seen across Windows with Python 2.7 and Ubuntu with Python 2.7 / 3.5. Traceback from Ubuntu with Python 3.5 as follows:
For Microsoft-Windows-WindowsUpdateClient%4Operational.evtx, Python 3.5 on Ubuntu gives Evtx.BinaryParser.OverrunBufferException, while Python 2.7 on Windows / Ubuntu gives a MemoryError. Python 3.5 traceback as follows:
Python 2.7 traceback (from Ubuntu) as follows:
And similarly for Microsoft-Windows-CAPI2%4Operational.evtx, Python 3.5 on Ubuntu gives Evtx.BinaryParser.OverrunBufferException, while Python 2.7 on Windows / Ubuntu BSODs / has process 'Killed'. Traceback for Python 3.5 on Ubuntu as follows:
I can forward the EVTX files in question if that assists?