Error when running on a disk image

[johnmccash]

Wili, I get the following error when attempting to run EVTXtract on a disk image (after successfully using it to extract templates from a large number of evtx files that were found on that disk image).

Any thoughts on what's going on?

python ~/EVTXtract/EVTXtract-master/find_evtx_chunks.py /mnt/ewf/ewf1 Project Template --progress

Progress: [=====================================================] Time: 1:23:01

Found 9400 valid chunks.

python ~/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py /mnt/ewf/ewf1 Project Template --progress

2014-06-10 13:41:08,810 WARNING root Flushing the existing template file due to exception. 2014-06-10 13:41:09,728 WARNING root Flushing the existing state file due to exception. Traceback (most recent call last): File "/home/likewise-open/SAD/john.mccash/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 183, in main extract_valid_evtx_records_and_templates(state, templates, buf, progress_class=args.progress_class) File "/path/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 167, in extract_valid_evtx_records_and_templates extract_chunk(buf, chunk_offset, state, templates) File "/path/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 160, in extract_chunk raise e XMLSyntaxError: Opening and ending tag mismatch: Unknown line 13 and string, line 13, column 27 Traceback (most recent call last): File "/path/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 190, in main() File "/path/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 183, in main extract_valid_evtx_records_and_templates(state, templates, buf, progress_class=args.progress_class) File "/path/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 167, in extract_valid_evtx_records_and_templates extract_chunk(buf, chunk_offset, state, templates) File "/path/EVTXtract/EVTXtract-master/extract_valid_evtx_records_and_templates.py", line 160, in extract_chunk raise e lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: Unknown line 13 and string, line 13, column 27

[johnmccash]

I tried running it with --verbose with essentially no difference. Running it with --vverbose, I get the following just before the error: . . . 2014-06-10 14:36:19,261 DEBUG extract_records Considering record at offset 12405511696 2014-06-10 14:36:19,267 DEBUG extract_records Considering record at offset 12405512656 2014-06-10 14:36:19,272 DEBUG extract_records Considering record at offset 12405513032 2014-06-10 14:36:19,273 INFO extract_records Unknown exception processing record at 0x2e36d1b48: Opening and ending tag mismatch: Unknown line 13 and string, line 13, column 27 2014-06-10 14:36:19,275 DEBUG template_database Using existing template file: Template.db.json 2014-06-10 14:36:19,301 WARNING root Flushing the existing template file due to exception. 2014-06-10 14:36:22,660 WARNING root Flushing the existing state file due to exception. . . .

[johnmccash]

If I had to guess, I'd suspect it was probably trying to process a corrupted chunk, which just happened to match the checksum, as though it were valid. What do you think?

[ihatecomputers]

I am having this exact issue. The evidence file in question is a singlepart raw disc image created with dd. find_evtx_chunks.py works without a hitch, but all subsequent scripts drop errors. Note that errors do not occur immediately; after the progress bar reaches around the 20% mark, everything suddenly explodes.

---

C:\Users\R\Desktop\a353 REGISTRY\EVTXtract-master\EVTXtract-master>python f
ind_evtx_chunks.py I:\A353dd.raw A353Test A353Test --progress
Progress: [===================================================] Time: 00:00:02
# Found 112 valid chunks.

C:\Users\R\Desktop\a353 REGISTRY\EVTXtract-master\EVTXtract-master>python e
xtract_valid_evtx_records_and_templates.py I:\A353dd.raw A353Test A353Test --pro
gress
2014-07-30 10:15:26,638 WARNING root Flushing the existing template file due to
exception.
2014-07-30 10:15:26,720 WARNING root Flushing the existing state file due to exc
eption.
Traceback (most recent call last):
  File "extract_valid_evtx_records_and_templates.py", line 183, in main
    extract_valid_evtx_records_and_templates(state, templates, buf, progress_cla
ss=args.progress_class)
  File "extract_valid_evtx_records_and_templates.py", line 167, in extract_valid
_evtx_records_and_templates
    extract_chunk(buf, chunk_offset, state, templates)
  File "extract_valid_evtx_records_and_templates.py", line 134, in extract_chunk

    for record in chunk.records():
  File "C:\Python27\lib\site-packages\Evtx\Evtx.py", line 414, in records
    record = self.first_record()
  File "C:\Python27\lib\site-packages\Evtx\Evtx.py", line 411, in first_record
    return Record(self._buf, self._offset + 0x200, self)
  File "C:\Python27\lib\site-packages\Evtx\Evtx.py", line 437, in __init__
    raise InvalidRecordException()
InvalidRecordException: Parse Exception(Invalid record structure)
Traceback (most recent call last):
  File "extract_valid_evtx_records_and_templates.py", line 190, in <module>
    main()
  File "extract_valid_evtx_records_and_templates.py", line 183, in main
    extract_valid_evtx_records_and_templates(state, templates, buf, progress_cla
ss=args.progress_class)
  File "extract_valid_evtx_records_and_templates.py", line 167, in extract_valid
_evtx_records_and_templates
    extract_chunk(buf, chunk_offset, state, templates)
  File "extract_valid_evtx_records_and_templates.py", line 134, in extract_chunk

    for record in chunk.records():
  File "C:\Python27\lib\site-packages\Evtx\Evtx.py", line 414, in records
    record = self.first_record()
  File "C:\Python27\lib\site-packages\Evtx\Evtx.py", line 411, in first_record
    return Record(self._buf, self._offset + 0x200, self)
  File "C:\Python27\lib\site-packages\Evtx\Evtx.py", line 437, in __init__
    raise InvalidRecordException()
Evtx.Evtx.InvalidRecordException: Parse Exception(Invalid record structure)

[ihatecomputers]

Also, following along with johnmccash's --vverbose log above:

2014-07-30 10:21:26,826 DEBUG extract_records Considering record at offset 250739272
2014-07-30 10:21:26,832 DEBUG extract_records Considering record at offset 250739760
2014-07-30 10:21:26,839 DEBUG extract_records Considering chunk at offset 305197056
2014-07-30 10:21:26,861 DEBUG template_database Using existing template file: A353Test.db.json
2014-07-30 10:21:26,869 WARNING root Flushing the existing template file due to exception.
2014-07-30 10:21:27,033 WARNING root Flushing the existing state file due to exception.

Note that in my entire --vverbose log, there is NO "unknown exception" thrown (I've searched the whole thing and not just the tail end.

[williballenthin]

Thanks for the updated info on this bug. I'm in the middle of a move right now, but I'd like to get the issue fixed as soon as I can. I hope this will be in a week or two. Please let me know if you notice anything else unusual, and we can squash them all in one go.

[awfr]

Any updates on this issue? I am having the same issue running on a single part disk image. find_evtx_chunks.py ran fine. Thank you.

C:\Python27>c:\python27\python c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py n:\14FR-08_00-01_750GB_HDD.001 14FR-08 14FR-08 --progress 2014-09-23 14:24:41,546 WARNING root Flushing the existing template file due to exception. 2014-09-23 14:24:41,601 WARNING root Flushing the existing state file due to exception. Traceback (most recent call last): File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 183, in main extract_valid_evtx_records_and_templates(state, templates, buf, progress_class=args.progress_class) File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 167, in extract_valid_evtx_records_and_templates extract_chunk(buf, chunk_offset, state, templates) File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 134, in extract_chunk for record in chunk.records(): File "c:\python27\lib\site-packages\Evtx\Evtx.py", line 414, in records record = self.first_record() File "c:\python27\lib\site-packages\Evtx\Evtx.py", line 411, in first_record return Record(self._buf, self._offset + 0x200, self) File "c:\python27\lib\site-packages\Evtx\Evtx.py", line 437, in init raise InvalidRecordException() InvalidRecordException: Parse Exception(Invalid record structure) Traceback (most recent call last): File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 190, in main() File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 183, in main extract_valid_evtx_records_and_templates(state, templates, buf, progress_class=args.progress_class) File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 167, in extract_valid_evtx_records_and_templates extract_chunk(buf, chunk_offset, state, templates) File "c:\py\EVTXtract\extract_valid_evtx_records_and_templates.py", line 134, in extract_chunk for record in chunk.records(): File "c:\python27\lib\site-packages\Evtx\Evtx.py", line 414, in records record = self.first_record() File "c:\python27\lib\site-packages\Evtx\Evtx.py", line 411, in first_record return Record(self._buf, self._offset + 0x200, self) File "c:\python27\lib\site-packages\Evtx\Evtx.py", line 437, in init raise InvalidRecordException() Evtx.Evtx.InvalidRecordException: Parse Exception(Invalid record structure)

[williballenthin]

I've pushed a change that will suppress this bug, and should allow processing to continue. I was able to reproduce it on a test image myself, so debugging it will be easier now.

Leaving this issue open until I can verify there's not another underlying issue (the unmatched tags in potentially corrupted chunk).

[bytesnoop]

Hello Will,

I am experiencing the same issues. find_evtx_chunks.py runs fine but extract_valid_evtx_records_and_templates.py throws the error outlined in previous posts. The error happens about 4 minutes in but what it does extract is great. Any updates on this issue? Thank you for your work.

[williballenthin]

I've recently rewritten much of EVTXtract to avoid bugs like these. Please retry on the latest version. This bug should be fixed.