search

log2timeline / plaso

Super timeline all the things
https://plaso.readthedocs.io
Apache License 2.0
1.73k stars 353 forks source link

winevtx: unable to parse evtx files without dependency identifier and string integer and FILETIME values #4876

Closed tomchop closed 6 months ago

tomchop commented 7 months ago

Describe the problem:

log2timeline is unable to parse the evtx files provided in the FCSC cybersecurity challenge (run by ANSSI). More info: france-cybersecurity-challenge.fr (seems to be in french only)

To Reproduce:

The version of Plaso you used:

20240317

The operating system you are running Plaso on (Not the operating system of the image/files you're trying to analyze):

For example: Docker images.

Steps to reproduce the behavior including command line and arguments and output:

wget https://files.france-cybersecurity-challenge.fr/dl/soc-simulator/soc_events.zip
unzip soc_events.zip
docker run --rm -v "$(pwd):/data" -w /data log2timeline/plaso log2timeline soc_events --storage_file soc.plaso
docker run --rm -v "$(pwd):/data" -w /data log2timeline/plaso psort soc.plaso -w soc.csv

Expected behavior:

The EVTX files being parsed by plaso

Actual behavior:

Only the filestat events get output in the CSV file:

$ wc -l soc.csv
1354 soc.csv
$ grep -v filestat soc.csv
datetime,timestamp_desc,source,source_long,message,parser,display_name,tag

Debug output/tracebacks:

the pinfo output of the plaso file 

 docker run --rm -v "$(pwd):/data" -w /data log2timeline/plaso pinfo soc.plaso

************************** Plaso Storage Information ***************************
            Filename : soc.plaso
      Format version : 20230327
Serialization format : json
--------------------------------------------------------------------------------

*********************************** Sessions ***********************************
5c28fe93-ac0c-433b-ab6d-e2d31d6b6f1b : 2024-04-16T13:36:48.806632+00:00
--------------------------------------------------------------------------------

******************************** Event sources *********************************
Total : 451
--------------------------------------------------------------------------------

************************* Events generated per parser **************************
Parser (plugin) name : Number of events
--------------------------------------------------------------------------------
            filestat : 1353
               Total : 1353
--------------------------------------------------------------------------------

No events labels stored.

******************* Extraction warnings generated per parser *******************
Parser (plugin) name : Number of warnings
--------------------------------------------------------------------------------
             winevtx : 1698310
--------------------------------------------------------------------------------

************** Path specifications with most extraction warnings ***************
Number of warnings : Pathspec
--------------------------------------------------------------------------------
              8607 : type: OS, location: /data/soc_events/20220706T143356.evtx
              6925 : type: OS, location: /data/soc_events/20220706T095008.evtx
              6706 : type: OS, location: /data/soc_events/20220706T130407.evtx
              6680 : type: OS, location: /data/soc_events/20220705T151955.evtx
              6676 : type: OS, location: /data/soc_events/20220705T134811.evtx
              6660 : type: OS, location: /data/soc_events/20220705T134544.evtx
              6657 : type: OS, location: /data/soc_events/20220705T132936.evtx
              6656 : type: OS, location: /data/soc_events/20220705T133524.evtx
              6655 : type: OS, location: /data/soc_events/20220705T130540.evtx
              6652 : type: OS, location: /data/soc_events/20220705T143505.evtx
--------------------------------------------------------------------------------

******************** Recovery warnings generated per parser ********************
Parser (plugin) name : Number of warnings
--------------------------------------------------------------------------------
             winevtx : 51066
--------------------------------------------------------------------------------

*************** Path specifications with most recovery warnings ****************
Number of warnings : Pathspec
--------------------------------------------------------------------------------
               567 : type: OS, location: /data/soc_events/20220704T215510.evtx
               552 : type: OS, location: /data/soc_events/20220706T105527.evtx
               539 : type: OS, location: /data/soc_events/20220704T165547.evtx
               536 : type: OS, location: /data/soc_events/20220706T011031.evtx
               526 : type: OS, location: /data/soc_events/20220705T142203.evtx
               523 : type: OS, location: /data/soc_events/20220706T145524.evtx
               500 : type: OS, location: /data/soc_events/20220706T032138.evtx
               498 : type: OS, location: /data/soc_events/20220704T204049.evtx
               475 : type: OS, location: /data/soc_events/20220705T231039.evtx
               463 : type: OS, location: /data/soc_events/20220705T021043.evtx
--------------------------------------------------------------------------------

No analysis reports stored.
tomchop commented 7 months ago

Warnings contain

************************** Extraction warning: 210200 **************************
           Message : unable to parse event record: 868 with error:
                     pyevtx_file_get_record_by_index: unable to retrieve
                     record: 868. libfwevt_xml_document_read_element: invalid
                     element data offset value out of bounds.
                     libfwevt_xml_document_read_fragment: unable to read
                     element. libfwevt_xml_document_read_with_template_values:
                     unable to read fragment header.
                     libfwevt_xml_document_read: unable to read XML document.
                     libevtx_record_values_read_xml_document: unable to read
                     binary XML document. libevtx_io_handle_read_chunk: unable
                     to read record values XML document.
                     libfdata_list_get_element_value: unable to read element
                     data at offset: 11511464 (0x00afa6a8).
                     libfdata_list_get_element_value_by_index: unable to
                     retrieve element value. libevtx_file_get_record_by_index:
                     unable to retrieve record values: 868.
      Parser chain : winevtx
Path specification : type: OS, location: /data/soc_events/20220705T160016.evtx
joachimmetz commented 7 months ago

Looking at one file (20220706T143356.evtx), it looks like it is "dirty"

evtxinfo 20220706T143356.evtx
evtxinfo 20240204

Windows Event Viewer Log (EVTX) information:
    Version             : 3.1
    Number of records       : 8607
    Number of recovered records : 27
    Flags:
        Is dirty
ibfwevt_xml_document_read_fragment_header: data offset                 : 0x00000218
libfwevt_xml_document_read_fragment_header: fragment header data:
00000000: 0f 01 01 00                                        ....

libfwevt_xml_document_read_fragment_header: type                        : 0x0f
libfwevt_xml_document_read_fragment_header: major version               : 1
libfwevt_xml_document_read_fragment_header: minor version               : 1
libfwevt_xml_document_read_fragment_header: flags                       : 0x00

libfwevt_xml_document_read_element: data offset                         : 0x0000021c
libfwevt_xml_document_read_element: element data:
00000000: 41 ac 0c 00 00 25 02                               A....%.

libfwevt_xml_document_read_element: type                                : 0x41
libfwevt_xml_document_read_element: dependency identifier               : 3244 (0x0cac)
libfwevt_xml_document_read_element: size                                : 35979264
libfwevt_xml_document_read_element: name offset                         : 0x00000000

Given what is known about the format the element data size 00 00 25 02 is off

Test skipping dependency identifier

libfwevt_xml_document_read_element: data offset                         : 0x0000021c
libfwevt_xml_document_read_element: element data:
00000000: 41 ac 0c 00 00                                     A....

libfwevt_xml_document_read_element: type                                : 0x41
libfwevt_xml_document_read_element: size                                : 3244
libfwevt_xml_document_read_element: name offset                         : 0x00000225

libfwevt_xml_document_read_name: data offset                            : 0x00000225
libfwevt_xml_document_read_name: name header data:
00000000: 00 00 00 00 ba 0c 05 00                            ........

This does not appears to be related to the dirty flag being set and format version, given the dependency identifier appears to be missing from all records

joachimmetz commented 7 months ago

Also looks like SystemTime is a string and not a FILETIME

From: 20220706T143356.evtx)

libfwevt_xml_tag_name_debug_print: name                                 : SystemTime

libfwevt_xml_document_read_value: data offset                           : 0x0000051d
libfwevt_xml_document_read_value: value data:
00000000: 05 01 1e 00                                        ....

libfwevt_xml_document_read_value: type                                  : 0x05
libfwevt_xml_document_read_value: value type                            : 0x01 (UTF-16 string)
libfwevt_xml_document_read_value: number of characters                  : 30
libfwevt_xml_document_read_value: data offset                           : 0x00000521
libfwevt_xml_document_read_value: value data:
00000000: 32 00 30 00 32 00 32 00  2d 00 30 00 37 00 2d 00   2.0.2.2. -.0.7.-.
00000010: 30 00 36 00 54 00 31 00  32 00 3a 00 32 00 34 00   0.6.T.1. 2.:.2.4.
00000020: 3a 00 34 00 30 00 2e 00  36 00 30 00 38 00 31 00   :.4.0... 6.0.8.1.
00000030: 31 00 35 00 35 00 30 00  30 00 5a 00               1.5.5.0. 0.Z.

libfwevt_xml_tag_value_debug_print: value                               : 2022-07-06T12:24:40.608115500Z

From Application.evtx

libfwevt_xml_tag_name_debug_print: name                                 : SystemTime

libfwevt_xml_document_read_optional_substitution: data offset           : 0x000005a5
libfwevt_xml_document_read_optional_substitution: optional substitution data:
00000000: 0e 06 00 11                                        ....

libfwevt_xml_document_read_optional_substitution: type                  : 0x0e
libfwevt_xml_document_read_optional_substitution: identifier            : 6
libfwevt_xml_document_read_optional_substitution: value type            : 0x11 (Filetime)

libfwevt_xml_document_substitute_template_value: value: 06 offset       : 0x0000081b
libfwevt_xml_document_substitute_template_value: value: 06 size         : 8
libfwevt_xml_document_substitute_template_value: value: 06 type         : 0x11 (Filetime)
libfwevt_xml_document_substitute_template_value: value: 06 data:
00000000: e8 7d ac a5 3a d0 d6 01                            .}..:...

libfwevt_xml_tag_value_debug_print: value                               : 2020-12-12T03:55:36.023396000Z

Looks like various other values like version are stored as strings as well.

joachimmetz commented 7 months ago

@tomchop it would be good to understand which tool / setting created these evtx files given they are in a slightly different format.

joachimmetz commented 6 months ago

Support for variant of evtx added to libevtx 20240427, closing issue.