search

log2timeline / plaso

Super timeline all the things
https://plaso.readthedocs.io
Apache License 2.0
1.66k stars 327 forks source link

winreg parser bails out on single corrupt key - please make it more resilient #3571

Closed MikeHofmann closed 3 years ago

MikeHofmann commented 3 years ago

Description of problem:

Followup from #3238

log2timeline.py parses only the Service from the SYSTEM hive CurrentControlSet but ignores other ControlSets from past working configurations.

Command line and arguments:

log2timeline.py --debug --workers 1 system.plaso system
psort.py -o l2tcsv -w system.csv system.plaso

has been used to generate the timeline.

log2timeline.py sees the ControlSet002:

zcat log2timeline-20210429T074209.log.gz | grep ControlSet002 | wc -l
1031

but no Services are parsed:

zcat log2timeline-20210429T074209.log.gz | grep 'ControlSet002\\Services' | wc -l
0

See below for more details.

Source data:

I exported a SYSTEM Hive from a Windows XP (sigh) System. I extracted the corresponding Software Keys using xways by hand:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion]
"ProductName"="Microsoft Windows XP"
"CurrentVersion"="5.1"

The hive seems healthy:

ls -al system
-rwxrwxrwx 1 2143290499 2143289857 16515072 Feb  8 10:26 system

file system
system: MS Windows registry file, NT/2000 or above

and i can export from a past ControlSet002 a malicious service using:

reged -x system 'HKEY_LOCAL_MACHINE\SYSTEM' '\ControlSet002\Services\corspdeft' export.reg
cat export.reg

giving:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\corspdeft]
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000000
"ImagePath"=hex(2):22,00,43,00,3a,00,5c,00,57,00,49,00,4e,00,44,00,4f,00,57,00,\
  53,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,63,00,6f,\
  00,72,00,73,00,70,00,64,00,65,00,66,00,74,00,2e,00,65,00,78,00,65,00,22,00,\
  00,00
"DisplayName"="corspdeft"
"ObjectName"="LocalSystem"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\corspdeft\Security]
"Security"=hex:01,00,14,80,90,00,00,00,9c,00,00,00,14,00,00,00,30,00,00,00,02,\
  00,1c,00,01,00,00,00,02,80,14,00,ff,01,0f,00,01,01,00,00,00,00,00,01,00,00,\
  00,00,02,00,60,00,04,00,00,00,00,00,14,00,fd,01,02,00,01,01,00,00,00,00,00,\
  05,12,00,00,00,00,00,18,00,ff,01,0f,00,01,02,00,00,00,00,00,05,20,00,00,00,\
  20,02,00,00,00,00,14,00,8d,01,02,00,01,01,00,00,00,00,00,05,0b,00,00,00,00,\
  00,18,00,fd,01,02,00,01,02,00,00,00,00,00,05,20,00,00,00,23,02,00,00,01,01,\
  00,00,00,00,00,05,12,00,00,00,01,01,00,00,00,00,00,05,12,00,00,00

Plaso version:

log2timeline.py --version
plaso - log2timeline version 20210412

Operating system Plaso is running on:

Installed using latest docker image

Installation method:

see above

If multiple methods were used please indicate.

joachimmetz commented 3 years ago

I'm unable to reproduce this with the SYSTEM file in test data:

2010-11-10T18:18:32.5312500+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2] Type: Kernel Device Driver (0x1) Start: Manual (3) Image path: \SystemRoot\system32\DRIVERS\lsi_sas2.sys Error control: Normal (1) DriverPackageId: [REG_SZ] lsi_sas2.inf_x86_neutral_e12a5c4cfbe49204 Group: [REG_SZ] SCSI Miniport,winreg/windows_services,OS:test_data/SYSTEM,-
2010-11-10T18:18:32.5312500+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2] Type: Kernel Device Driver (0x1) Start: Manual (3) Image path: \SystemRoot\system32\DRIVERS\lsi_sas2.sys Error control: Normal (1) DriverPackageId: [REG_SZ] lsi_sas2.inf_x86_neutral_e12a5c4cfbe49204 Group: [REG_SZ] SCSI Miniport,winreg/windows_services,OS:test_data/SYSTEM,-
2012-04-04T11:47:08.7656250+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS] Type: Kernel Device Driver (0x1) Start: Boot (0) Image path: \SystemRoot\system32\drivers\lsi_sas.sys Error control: Normal (1) Group: [REG_SZ] SCSI Miniport Tag: [REG_DWORD_LE] 64,winreg/windows_services,OS:test_data/SYSTEM,-
2012-04-04T11:47:08.7656250+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS] Type: Kernel Device Driver (0x1) Start: Boot (0) Image path: \SystemRoot\system32\drivers\lsi_sas.sys Error control: Normal (1) Group: [REG_SZ] SCSI Miniport Tag: [REG_DWORD_LE] 64,winreg/windows_services,OS:test_data/SYSTEM,-
joachimmetz commented 3 years ago

@MikeHofmann can you provide the output of log2timeline.py --troubles and debug logs

Also see: https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html

joachimmetz commented 3 years ago

@MikeHofmann log2timeline-20210429T074209.log.gz is the log file, what does you psort output tell you?

joachimmetz commented 3 years ago

@MikeHofmann also can you try dynamic output instead of l2tcsv. The amount of information l2tcsv can present is limited also see https://github.com/log2timeline/plaso/issues/3570

joachimmetz commented 3 years ago

Then again for me l2tcsv does show both control sets:

11/02/2006,12:49:55,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters\PnpInter...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters\PnpInterface] 5: [REG_DWORD_LE] 1,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
12/16/2008,02:28:41,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters] BusType...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters] BusTyp...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters\PnpInte...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters\PnpInterface] 5: [REG_DWORD_LE] 1,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters] BusTyp...,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters\PnpInte...,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters\PnpInterface] 5: [REG_DWORD_LE] 1,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS\Parameters] BusType...,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
MikeHofmann commented 3 years ago

@MikeHofmann log2timeline-20210429T074209.log.gz is the log file, what does you psort output tell you?

Attaching both logs (had to redact some path informations)

log2timeline-20210429T074209.log.redacted.gz psort-20210429T074256.log.gz

@MikeHofmann can you provide the output of log2timeline.py --troubles

2021-04-29 09:00:57,708 [INFO] (MainProcess) PID:472 <data_location> Determined data location: /usr/share/plaso
Using Python version 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0]

Path: /usr/bin/log2timeline.py

plaso - log2timeline version 20210412

Checking availability and versions of dependencies.
[OK]            artifacts version: 20210404
[OK]            bencode
[OK]            certifi version: 2020.06.20
[OK]            chardet version: 3.0.4
[OK]            cryptography version: 3.3.1
[OK]            dateutil version: 2.8.1
[OK]            defusedxml version: 0.7.1
[OK]            dfdatetime version: 20200824
[OK]            dfvfs version: 20210213
[OK]            dfwinreg version: 20201006
[OK]            dtfabric version: 20200621
[OK]            elasticsearch version: 7.9.1
[OK]            future version: 0.18.2
[OK]            idna version: 2.9
[OK]            lz4 version: 3.1.3
[OK]            pefile version: 2019.4.18
[OK]            psutil version: 5.8.0
[OK]            pybde version: 20210327
[OK]            pycreg version: 20200725
[OK]            pyesedb version: 20200418
[OK]            pyevt version: 20200926
[OK]            pyevtx version: 20200709
[OK]            pyewf version: 20140811
[OK]            pyfsapfs version: 20201107
[OK]            pyfsext version: 20210129
[OK]            pyfshfs version: 20201104
[OK]            pyfsntfs version: 20201115
[OK]            pyfsxfs version: 20210403
[OK]            pyfvde version: 20191221
[OK]            pyfwnt version: 20200723
[OK]            pyfwsi version: 20201204
[OK]            pylnk version: 20200810
[OK]            pyluksde version: 20200205
[OK]            pymsiecf version: 20200710
[OK]            pyolecf version: 20201004
[OK]            pyparsing version: 2.4.7
[OK]            pyqcow version: 20201213
[OK]            pyregf version: 20201007
[OK]            pyscca version: 20200717
[OK]            pysigscan version: 20201117
[OK]            pysmdev version: 20201204
[OK]            pysmraw version: 20201210
[OK]            pytsk3 version: 20210130
[OK]            pytz
[OK]            pyvhdi version: 20201204
[OK]            pyvmdk version: 20200926
[OK]            pyvsgpt version: 20210207
[OK]            pyvshadow version: 20201222
[OK]            pyvslvm version: 20200817
[OK]            redis version: 3.5.3
[OK]            requests version: 2.24.0
[OK]            six version: 1.15.0
[OK]            urllib3 version: 1.25.9
[OK]            xlsxwriter version: 1.3.8
[OK]            yaml version: 5.3.1
[OK]            yara version: 4.0.5
[OK]            zmq version: 22.0.3

Also see: https://plaso.readthedocs.io/en/latest/sources/user/Troubleshooting.html

@MikeHofmann also can you try dynamic output instead of l2tcsv. The amount of information l2tcsv can present is limited also

noted. We use this for analyzing in Erics TimeLineExplorer and it's easier to quickly grep some information. But if we upload the .plaso file to timesketch the services is also not visible.

joachimmetz commented 3 years ago

At first glance I do not see anything in Plaso that hints at an error.

MikeHofmann commented 3 years ago

Hmmh, these tools seems to be older when i install them inside the log2timeline-container. They don't feature the -H or -K options.

apt-get install libregf-utils

regfinfo -V
regfinfo 20190303

Copyright (C) 2009-2019, Joachim Metz.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Report bugs to <joachim.metz@gmail.com>.

regfexport -V
regfexport 20190303

Copyright (C) 2009-2019, Joachim Metz.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Report bugs to <joachim.metz@gmail.com>.

When used without any cli-options on the Hive, i do get some errors on stderr:

regfinfo system

Unable to print file information.
libregf_key_item_read_named_key: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read_node_data: unable to read named key at offset: 16215544 (0x00f76df8).
libfdata_tree_get_node_value: unable to read node at offset: 0x00f76df8.
libfdata_tree_node_get_node_value: unable to retrieve node value.
libregf_key_get_utf8_name_size: unable to retrieve key item.
info_handle_key_fprint: unable to retrieve key name size.
info_handle_key_fprint: unable to print sub key: 16 info.
info_handle_key_fprint: unable to print sub key: 42 info.
info_handle_key_fprint: unable to print sub key: 5 info.
info_handle_key_fprint: unable to print sub key: 0 info.
info_handle_key_fprint: unable to print sub key: 1 info.
info_handle_file_fprint: unable to print root key info.

I do see ControlSet002\Control but not ControlSet002\Services (ControlSet001\Services exists)

joachimmetz commented 3 years ago

Hmmh, these tools seems to be older when i install them inside the log2timeline-container.

Can you try libregf-tools (instead of any older versions libregf-utils from Ubuntu/Debian) from the GIFT PPA that should be enabled inside the container

joachimmetz commented 3 years ago

This would indicate libregf is not able to find all the data it needs.

libregf_key_item_read_named_key: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read_node_data: unable to read named key at offset: 16215544 (0x00f76df8).
libfdata_tree_get_node_value: unable to read node at offset: 0x00f76df8.
libfdata_tree_node_get_node_value: unable to retrieve node value.
libregf_key_get_utf8_name_size: unable to retrieve key item.
MikeHofmann commented 3 years ago

Can you try libregf-tools (instead of any older versions libregf-utils from Ubuntu/Debian) from the GIFT PPA that should be enabled inside the container

Nope, the PPA wasnt enabled, had to:

add-apt-repository ppa:gift/stable
apt-get install libregf-tools

Tools are newer, but still regfinfo missing the -H option. However regfexport now has -K.

regfinfo -h
regfinfo 20201007

Use regfinfo to determine information about a Windows NT
Registry File (REGF).

Usage: regfinfo [ -c codepage ] [ -hvV ] source

        source: the source file

        -c:     codepage of ASCII strings, options: ascii, windows-874,
                windows-932, windows-936, windows-949, windows-950,
                windows-1250, windows-1251, windows-1252 (default),
                windows-1253, windows-1254, windows-1255, windows-1256
                windows-1257 or windows-1258
        -h:     shows this help
        -v:     verbose output to stderr
        -V:     print version
regfexport -h
regfexport 20201007

Use regfexport to export information from a Windows NT
Registry File (REGF).

Usage: regfexport [ -c codepage ] [ -K key_path ] [ -l logfile ]
                  [ -hvV ] source

        source: the source file

        -c:     codepage of ASCII strings, options: ascii, windows-874,
                windows-932, windows-936, windows-949, windows-950,
                windows-1250, windows-1251, windows-1252 (default),
                windows-1253, windows-1254, windows-1255, windows-1256
                windows-1257 or windows-1258
        -h:     shows this help
        -K:     show information about a specific key path.
        -l:     logs information about the exported items
        -v:     verbose output to stderr
        -V:     print version

When using regfinfo i still get errors on stderr:

nable to print file information.
libregf_hive_bins_list_get_cell_at_offset: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read_named_key: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read: unable to read named key at offset: 16215544 (0x00f76df8).
libregf_key_initialize: unable to read key item at offset: 16215544 (0x00f76df8).
libregf_key_get_sub_key: unable to initialize sub key: 16.
info_handle_key_fprint: unable to retrieve sub key: 16.
info_handle_key_fprint: unable to print sub key: 42 info.
info_handle_key_fprint: unable to print sub key: 5 info.
info_handle_key_fprint: unable to print sub key: 0 info.
info_handle_key_fprint: unable to print sub key: 1 info.
info_handle_file_fprint: unable to print root key info.

but regfexport yields results:

regfexport -K '\ControlSet002\services' system | wc -l
44688

regfexport -K '\ControlSet002\services' system | grep corspdeft
Key path: \ControlSet002\services\Services\corspdeft
Key: corspdeft
Data: "C:\WINDOWS\system32\corspdeft.exe"
Data: corspdeft
Key path: \ControlSet002\services\Services\corspdeft\Security

While i do find the services\Services output troubling. The lowercase services shouldn't be there.

is the entire SYSTEM file there? not that it did got truncated on copy or export?

Yes, same problem exists when i process the entire EWF Image. I just exported the SYSTEM hive for simplicity. Also note that reged found the Keys as well as xways own registry viewer.

did log2timeline.py indicate any parsing warnings (pinfo.py should be able to give you that information)

Yes, one warning present:

           Message : pyregf_key_get_sub_key_by_index: unable to retrieve sub
                     key: 16. libregf_hive_bins_list_get_cell_at_offset: unable
                     to retrieve hive bin cell at offset: 16215544
                     (0x00f76df8). libregf_key_item_read_named_key: unable to
                     retrieve hive bin cell at offset: 16215544 (0x00f76df8).
                     libregf_key_item_read: unable to read named key at offset:
                     16215544 (0x00f76df8). libregf_key_initialize: unable to
                     read key item at offset: 16215544 (0x00f76df8).
                     libregf_key_get_sub_key: unable to initialize sub key: 16.
      Parser chain : winreg
joachimmetz commented 3 years ago

Tools are newer, but still regfinfo missing the -H option.

Maybe something I added more recent than 20201007

While i do find the services\Services output troubling.

To determine if this is corruption or some format edge case

I assume you cannot share the file? If not could you compile the latest from source with verbose and debug output https://github.com/libyal/libregf/wiki/Troubleshooting#verbose-and-debug-output

regfexport -v -K '\ControlSet002\services' system 2>debug.log would still provide a lot of details, so double check it, but should likely be sufficient to pin point a possible root cause

The lowercase services shouldn't be there.

Don't know I would need to see the actual data

MikeHofmann commented 3 years ago

I assume you cannot share the file?

No, this agency is not known for sharing. :smile:

If not could you compile the latest from source with verbose and debug output

I quickly slapped a docker-container together and compiled (CPPCFLAGS=-g ./configure --enable-shared=no --enable-verbose-output --enable-debug-output) and used this on the hive. The debug.log is huge, probably more than github.com can handle. And i would need to filter out a lot of information (probably some grep -vE "^0" ?). But i think this wouldn't help, as regfexport is able to parse the hive without any problems:

Key path: \ControlSet002\services\Services\corspdeft
Name: corspdeft
Last written time: Jan 26, 2021 08:42:04.942000000 UTC

Value: 0 Type
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 16

Value: 1 Start
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 2

Value: 2 ErrorControl
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 0

Value: 3 ImagePath
Type: expandable string (REG_EXPAND_SZ)
Data size: 72
Data: "C:\WINDOWS\system32\corspdeft.exe"

Value: 4 DisplayName
Type: string (REG_SZ)
Data size: 20
Data: corspdeft

Value: 5 ObjectName
Type: string (REG_SZ)
Data size: 24
Data: LocalSystem
joachimmetz commented 3 years ago

The debug.log is huge, probably more than github.com can handle.

Can you compress it with gzip and mail it to me ?

And i would need to filter out a lot of information (probably some grep -vE "^0"

any special chars in the key path ?

joachimmetz commented 3 years ago

But i think this wouldn't help, as regfexport is able to parse the hive without any problems:

One thing that could be happening is that winreg stops parsing because the exception is raised. I'll have a look if I can make this more error resilient.

MikeHofmann commented 3 years ago

Can you compress it with gzip and mail it to me ? i just mailed you something.

Whats seems odd, that my compiled version doesn't complain nor does the exporter from ppa:gift/stable, it's only from the docker-container. Could be the build process for the container using some old libraries?

joachimmetz commented 3 years ago

The version detected in the --troubles output is:

[OK]            pyregf version: 20201007

which was the latest version when the container was released

joachimmetz commented 3 years ago

i just mailed you something.

thx, received will have a look later

Also made some changes to make winreg more resilient to corrupt files https://github.com/log2timeline/plaso/pull/3572/files

MikeHofmann commented 3 years ago

Also made some changes to make winreg more resilient to corrupt files https://github.com/log2timeline/plaso/pull/3572/files

Tested with this change, this fixes the issue. Thx. Any idea when this will reach the container on hub.docker.com?

MikeHofmann commented 3 years ago

Was a little too hasty here. CurrentControlSet should be a link to the ControlSet as indicated in HKLM\System\Select\Current. The timeline includes some services under the CurrentControlSet Keys, but i can't figure out from which ControlSet.

The SYSTEM hive has three ControlSets (ControlSet001, ControlSet002 and ControlSet004). Each having a different services configuration. For example ControlSet001 has a unique service tifsfilter that the other two don't have. ControlSet002 has corspdeft and ControlSet004 has 157786025 which the respective other ControlSets don't have:

grep -E "(tifsfilter|corspdeft|157786025)" system.csv 
05/14/2009,06:57:13,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter\Security] Securi...,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter\Security] Security: [REG_BINARY] (168 bytes),2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
02/26/2010,15:44:28,UTC,M...,REG,Registry Key - Service,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter] Type: File Syst...,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter] Type: File System Driver (0x2) Start: Auto Start (2) Image path: system32\DRIVERS\tifsfilt.sys Error control: Normal (1) DisplayName: [REG_SZ] Acronis True Image FS Filter Group: [REG_SZ] Filter Tag: [REG_DWORD_LE] 7,2,OS:/export/system,-,-,winreg/windows_services,name: tifsfilter; sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
02/26/2010,15:44:43,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\Enum\Root\LEGACY_TIFSFILTER\0000] Cl...,[HKEY_LOCAL_MACHINE\System\ControlSet001\Enum\Root\LEGACY_TIFSFILTER\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] Acronis True Image FS Filter Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] tifsfilter,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
03/02/2011,14:03:42,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_TIFSFILTER\0000] Cl...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_TIFSFILTER\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] Acronis True Image FS Filter Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] tifsfilter,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
03/02/2011,14:03:42,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_TIFSFILTER\0000] Cl...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_TIFSFILTER\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] Acronis True Image FS Filter Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] tifsfilter,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,10:12:00,UTC,....,REG,AppCompatCache Registry Entry,File Last Modification Time,-,-,Path: \??\C:\WINDOWS\157786025.exe,[HKEY_LOCAL_MACHINE\System\ControlSet004\Control\Session Manager\AppCompatibility] Cached entry: 76 Path: \??\C:\WINDOWS\157786025.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,11:29:14,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025\Security] Securit...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025\Security] Security: [REG_BINARY] (168 bytes),2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,11:29:15,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025] NextInst...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025] NextInstance: [REG_DWORD_LE] 1,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,11:29:15,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025\0000] Cla...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] 157786025 Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] 157786025,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
04/22/2020,09:27:38,UTC,.A..,REG,AppCompatCache Registry Entry,Last Time Executed,-,-,Path: \??\C:\WINDOWS\157786025.exe,[HKEY_LOCAL_MACHINE\System\ControlSet004\Control\Session Manager\AppCompatibility] Cached entry: 76 Path: \??\C:\WINDOWS\157786025.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
06/06/2020,16:00:33,UTC,M...,REG,Registry Key - Service,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025] Type: Service - ...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025] Type: Service - Own Process (0x10) Start: Auto Start (2) Image path: %SystemRoot%\157786025.exe Error control: Ignore (0),2,OS:/export/system,-,-,winreg/windows_services,name: 157786025; object_name: LocalSystem; sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:41:56,UTC,....,REG,AppCompatCache Registry Entry,File Last Modification Time,-,-,Path: \??\C:\WINDOWS\system32\corspdeft.exe,[HKEY_LOCAL_MACHINE\System\ControlSet002\Control\Session Manager\AppCompatibility] Cached entry: 46 Path: \??\C:\WINDOWS\system32\corspdeft.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft\Security] Securit...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft\Security] Security: [REG_BINARY] (168 bytes),2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_CORSPDEFT\0000] Cla...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_CORSPDEFT\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] corspdeft Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] corspdeft,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,M...,REG,Registry Key - Service,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft] Type: Service - ...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft] Type: Service - Own Process (0x10) Start: Auto Start (2) Image path: "C:\WINDOWS\system32\corspdeft.exe" Error control: Ignore (0) DisplayName: [REG_SZ] corspdeft,2,OS:/export/system,-,-,winreg/windows_services,name: corspdeft; object_name: LocalSystem; sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,.A..,REG,AppCompatCache Registry Entry,Last Time Executed,-,-,Path: \??\C:\WINDOWS\system32\corspdeft.exe,[HKEY_LOCAL_MACHINE\System\ControlSet002\Control\Session Manager\AppCompatibility] Cached entry: 46 Path: \??\C:\WINDOWS\system32\corspdeft.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae

But CurrentControlSet doesn't have any of these services. It has another set of services, which doesn't match any ControlSet.

MikeHofmann commented 3 years ago

Oh and pinfo.py now gives multiple warnings when having the fix applied:

********************************** Warning: 0 **********************************
           Message : unable to process path specification with error: name
                     'subkey_index' is not defined
      Parser chain : 
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

********************************** Warning: 1 **********************************
           Message : unable to process path specification with error: name
                     'subkey_index' is not defined
      Parser chain : 
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

********************************** Warning: 2 **********************************
           Message : unable to process path specification with error: name
                     'subkey_index' is not defined
      Parser chain : 
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

********************************** Warning: 3 **********************************
           Message : in key:
                     HKEY_LOCAL_MACHINE\System\ControlSet002\Control\Class\{71A27CDD-812A-11D0-BEC7-08002BE2092F}
                     error: pyregf_key_get_sub_key_by_index: unable to retrieve
                     sub key: 16. libregf_hive_bins_list_get_cell_at_offset:
                     unable to retrieve hive bin cell at offset: 16215544
                     (0x00f76df8). libregf_key_item_read_named_key: unable to
                     retrieve hive bin cell at offset: 16215544 (0x00f76df8).
                     libregf_key_item_read: unable to read named key at offset:
                     16215544 (0x00f76df8). libregf_key_initialize: unable to
                     read key item at offset: 16215544 (0x00f76df8).
                     libregf_key_get_sub_key: unable to initialize sub key: 16.
      Parser chain : winreg
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------
joachimmetz commented 3 years ago

Any idea when this will reach the container on hub.docker.com?

In the next release (https://github.com/log2timeline/plaso/milestone/33), end of June likely

The timeline includes some services under the CurrentControlSet Keys, but i can't figure out from which ControlSet.

Can you provide an example entry?

But CurrentControlSet doesn't have any of these services. It has another set of services, which doesn't match any ControlSet.

CurrentControlSet of what? How are you looking at CurrentControlSet?

MikeHofmann commented 3 years ago

CurrentControlSet of what? How are you looking at CurrentControlSet?

Uff, i think i need a break. I was operating under the assumption, that log2timeline includes the virtual CurrentControlSet into the timeline as well. And i couldn't find the services i was looking for by grepping in the psorted .csv, while at the same time looking at the file using less and because of that my mind mixed a few columns on screen.

TLDR: ControlSet[0-9]{3} is included in the timeline, CurrentControlSet is not. As it should be.

I think this can be closed now. Do you keep this open until everything is merged or can i close it?

joachimmetz commented 3 years ago

Keep it open for now, is a reminder for me to add test coverage for a corrupted Windows NT Registry file.

MikeHofmann commented 3 years ago

Just some statistical ramblings, no action needed:

I was wondering, on how often our images would run into a error: pyregf_key_get_sub_key_by_index: unable to retrieve. So i took the time and tested a few images. All from different sources (different organization/person, different examiner, different imager-tool, different hardware, etc.). I had the hopes in finding some common reason, why the registry is damaged. I'm afraid this looks totally random to me.

ImageID CurrentBuildNr ProductName regf Warnings present
35 17763 Windows 10 Pro no
45 18363 Windows 10 Enterprise no
46 14393 Windows Server 2016 Standard no
47 9600 Windows Server 2012 R2 Standard no
48 7601 Windows 7 Ultimate no
49 7601 Windows 7 Professional no
50 9600 Windows Server 2012 R2 Standard no
51 17763 Windows 10 Pro yes
52 2600 Microsoft Windows XP yes
53 6003 Windows Server 2008 Enterprise yes
54 14393 Windows Server 2016 Standard no
55 9600 Windows Server 2012 R2 Standard no
56 7601 Windows Server 2008 R2 Standard no
57 7601 Windows 7 Enterprise no
59 9600 Windows Server 2012 R2 Datacenter no
60 7601 Windows 7 Home Premium no
61 17134 Windows 10 Pro no
joachimmetz commented 3 years ago

All from different sources (different organization/person, different examiner, different imager-tool, different hardware, etc.).

MikeHofmann commented 3 years ago

were these all offline acquisitions ? or all some live acquisitions ? do you know if these systems were cleanly shutdown or some also pulled the plug ?

Unfortunately these images are below average quality from a forensics standpoint. From the three images above: one was done with FTK Imager, another one with Logicube Falcon and the last just a .vmdk. No written aquisition reports, sometimes a log file from the aquisition/imager tool, no name of examiner, etc. Probably need to parse bootstat.dat (a very nice to have feature for l2t btw) to check for a clean shutdown.

joachimmetz commented 3 years ago

Probably need to parse bootstat.dat (a very nice to have feature for l2t btw) to check for a clean shutdown.

https://github.com/log2timeline/plaso/issues/3578

joachimmetz commented 3 years ago

Test file added https://github.com/log2timeline/plaso/pull/3576, closing issue