sf-jonstewart
commented
1 year ago Thanks! Will investigate.
Open scudette opened 1 year ago
sf-jonstewart
commented
1 year ago Thanks! Will investigate.
reece394
commented
6 days ago I have the same issue when collecting from a live system using KAPE with Server 2022. I noticed when I shut down my VM and copied the files from the disk when it was not being used it parsed fine so it seems to have something to do with the Search Index / database being open whilst it is being copied
reece394
commented
6 days ago C:\Users\TestUser\Documents\sidr\target\debug>sidr -f csv C:\\Server20222
Processing ESE db: C:\\Server20222\C\programdata\microsoft\search\data\applications\windows\Windows.edb
thread 'main' panicked at src\ese.rs:168:73:
called `Result::unwrap()` on an `Err` value: SimpleError { err: "wrong checksum: 1551933666, calculated 1089402065" }
stack backtrace:
0: 0x7ff6ab4abd1d - std::backtrace_rs::backtrace::dbghelp64::trace
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:91
1: 0x7ff6ab4abd1d - std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
2: 0x7ff6ab4abd1d - std::sys::backtrace::_print_fmt
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:65
3: 0x7ff6ab4abd1d - std::sys::backtrace::impl$0::print::impl$0::fmt
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:40
4: 0x7ff6ab4bde29 - core::fmt::rt::Argument::fmt
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\fmt\rt.rs:173
5: 0x7ff6ab4bde29 - core::fmt::write
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\fmt\mod.rs:1182
6: 0x7ff6ab4a8f41 - std::io::Write::write_fmt<std::sys::pal::windows::stdio::Stderr>
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\io\mod.rs:1827
7: 0x7ff6ab4adac7 - std::panicking::default_hook::closure$1
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:269
8: 0x7ff6ab4ad6b9 - std::panicking::default_hook
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:296
9: 0x7ff6ab4ae202 - std::panicking::rust_panic_with_hook
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:800
10: 0x7ff6ab4ae046 - std::panicking::begin_panic_handler::closure$0
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:674
11: 0x7ff6ab4ac40f - std::sys::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::closure_env$0,never$>
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:168
12: 0x7ff6ab4adc26 - std::panicking::begin_panic_handler
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:665
13: 0x7ff6ab5b6444 - core::panicking::panic_fmt
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\panicking.rs:74
14: 0x7ff6ab5b68a0 - core::result::unwrap_failed
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\result.rs:1679
15: 0x7ff6ab2c4073 - enum2$<core::result::Result<ese_parser_lib::ese_parser::EseParser<std::io::buffered::bufreader::BufReader<std::fs::File> >,simple_error::SimpleError> >::unwrap
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c\library\core\src\result.rs:1102
16: 0x7ff6ab2c4073 - sidr::ese::ese_generate_report
at C:\Users\TestUser\Documents\sidr\src\ese.rs:168
17: 0x7ff6ab2c9f23 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:36
18: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
19: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
20: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
21: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
22: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
23: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
24: 0x7ff6ab2ca166 - sidr::dump
at C:\Users\TestUser\Documents\sidr\src\main.rs:32
25: 0x7ff6ab2ca8c0 - sidr::main
at C:\Users\TestUser\Documents\sidr\src\main.rs:125
26: 0x7ff6ab2cca73 - core::ops::function::FnOnce::call_once<enum2$<core::result::Result<tuple$<>,simple_error::SimpleError> > (*)(),tuple$<> >
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c\library\core\src\ops\function.rs:250
27: 0x7ff6ab2cf9e6 - std::sys::backtrace::__rust_begin_short_backtrace<enum2$<core::result::Result<tuple$<>,simple_error::SimpleError> > (*)(),enum2$<core::result::Result<tuple$<>,simple_error::SimpleError> > >
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c\library\std\src\sys\backtrace.rs:152
28: 0x7ff6ab2d3ff6 - std::rt::lang_start::closure$0<enum2$<core::result::Result<tuple$<>,simple_error::SimpleError> > >
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c\library\std\src\rt.rs:162
29: 0x7ff6ab4a38f9 - std::rt::lang_start_internal::closure$2
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\rt.rs:141
30: 0x7ff6ab4a38f9 - std::panicking::try::do_call
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:557
31: 0x7ff6ab4a38f9 - std::panicking::try
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:521
32: 0x7ff6ab4a38f9 - std::panic::catch_unwind
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panic.rs:350
33: 0x7ff6ab4a38f9 - std::rt::lang_start_internal
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\rt.rs:141
34: 0x7ff6ab2d3fca - std::rt::lang_start<enum2$<core::result::Result<tuple$<>,simple_error::SimpleError> > >
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c\library\std\src\rt.rs:161
35: 0x7ff6ab2cbd69 - main
36: 0x7ff6ab5b3f20 - invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
37: 0x7ff6ab5b3f20 - __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
38: 0x7ffa8f3f4ed0 - BaseThreadInitThunk
39: 0x7ffa9054e39b - RtlUserThreadStartI did some more experimenting with this and in this case the database having the state Dirty Shutdown causes this. For anybody else having this error to fix this you will need the entire folder of programdata\microsoft\search\data\applications\windows as this contains the transaction logs. Make a copy of the folder as this process will modify the database! Then change directory into where the database are stored esentutl /mh Windows.edb - to check if the database is dirty esentutl /r edb /i - This is to do soft recovery using the edb.jtx transaction log files stored in the folder esentutl /p Windows.edb - This repairs the Windows.edb database and allows for sidr to parse it.
Below is the commands I ran to aid troubleshooting. Seems like error handling will be needing implemented for this case.
PS C:\SERVER20222\C\programdata\microsoft\search\data\applications\windows> esentutl /mh Windows.edb
Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
Database: Windows.edb
DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x5c80a0e2
Actual Checksum: 0x5c80a0e2
Fields:
File Type: Database
Checksum: 0x5c80a0e2
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,180,400 (attached by 9360)
Engine ulVersion: 0x620,180,400 (efvCurrent = 9360)
Created ulVersion: 0x620,20
DB Signature: Create time:10/16/2024 19:40:50.362 Rand:3090127869 Computer:
cbDbPage: 32768
dbtime: 3450 (0xd7a)
State: Dirty Shutdown
Log Required: 4-4 (0x4-0x4) [Min. Req. Pre-Redo: 0 (0x0)]
Gen. Max. Req. Creation Time: 10/16/2024 19:40:57.830 UTC
Log Committed: 0-4 (0x0-0x4)
Gen. Max. Com. Creation Time: 10/16/2024 19:40:57.830 UTC
Log Consistent: 4 (0x4) [Pre-Redo: 0 (0x0)]
Log Recovering: 0 (0x0)
Shadowed: Yes
Last Objid: 27
Scrub Dbtime: 0 (0x0)
Scrub Date: 00/00/1900 00:00:00.000 LOC
Repair Count: 0
Repair Date: 00/00/1900 00:00:00.000 LOC
Old Repair Count: 0
Last Consistent: (0x4,64,9D8) 10/16/2024 22:18:41.116 UTC
Last Attach: (0x4,66,268) 10/17/2024 08:03:32.978 UTC
Last Detach: (0x0,0,0) 00/00/1900 00:00:00.000 LOC
Last ReAttach: (0x0,0,0) 00/00/1900 00:00:00.000 LOC
Dbid: 1
Log Signature: Create time:10/16/2024 19:40:50.346 Rand:3152147356 Computer:
OS Version: (10.0.20348 SP 0 NLS ffffffff.ffffffff)
Previous Full Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Previous Incremental Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Previous Copy Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Previous Differential Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Current Full Backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
Current Shadow copy backup:
Log Gen: 0-0 (0x0-0x0)
Mark: (0x0,0,0)
Mark: 00/00/1900 00:00:00.000 LOC
cpgUpgrade55Format: 0
cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0
Last Resize: (0x4,66,268) (Matches "Last Attach", no subsequent resize)
Extend Count: 1
Extend Date: 10/16/2024 19:40:50.393 UTC
Shrink Count: 0
Shrink Date: 00/00/1900 00:00:00.000 LOC
Trim Count: 0
ECC Fix Success Count: none
Old ECC Fix Success Count: none
ECC Fix Error Count: none
Old ECC Fix Error Count: none
Bad Checksum Error Count: none
Old bad Checksum Error Count: none
Last Database Maintenance Finish Date: 00/00/1900 00:00:00.000 LOC
Current Database Maintenance Start Date: 00/00/1900 00:00:00.000 LOC
Highest Continuous Database Maintenance Page: 0
Highest Database Maintenance Page: 0
Database Header Flush Signature: Create time:10/17/2024 08:03:32.978 Rand:1357645417 Computer:
Flush Map Header Flush Signature: Create time:10/17/2024 08:03:32.978 Rand:1475634610 Computer:
Operation completed successfully in 0.47 seconds.
PS C:\SERVER20222\C\programdata\microsoft\search\data\applications\windows> esentutl /r edb /i
Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating RECOVERY mode...
Logfile base name: edb
Log files: <current directory>
System files: <current directory>
Performing soft recovery...
Restore Status (% complete)
0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................
Operation completed successfully in 30.110 seconds.
PS C:\SERVER20222\C\programdata\microsoft\search\data\applications\windows> esentutl /p Windows.edb
Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating REPAIR mode...
Database: Windows.edb
Temp. Database: TEMPREPAIR6976.EDB
Checking database integrity.
The database is not up-to-date. This operation may find that
this database is corrupt because data from the log files has
yet to be placed in the database.
To ensure the database is up-to-date please use the 'Recovery' operation.
Scanning Status (% complete)
0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................
Scanning the database.
Scanning Status (% complete)
0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................
Repairing damaged tables.
Scanning Status (% complete)
0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................
Repair completed. Database corruption has been repaired!
Note:
It is recommended that you immediately perform a full backup
of this database. If you restore a backup made before the
repair, the database will be rolled back to the state
it was in at the time of that backup.
Operation completed successfully with 595 (JET_wrnDatabaseRepaired, Database corruption has been repaired) after 9.516 seconds.
PS C:\SERVER20222\C\programdata\microsoft\search\data\applications\windows>
C:\Users\TestUser\Documents\sidr\target\debug>sidr -f csv C:\\Server20222
Processing ESE db: C:\\Server20222\C\programdata\microsoft\search\data\applications\windows\Windows.edb
thread 'main' panicked at src\ese.rs:168:73:
called `Result::unwrap()` on an `Err` value: SimpleError { err: "wrong checksum: 1551933666, calculated 1089402065" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
C:\Users\TestUser\Documents\sidr\target\debug>sidr -f csv C:\\Server20222
Processing ESE db: C:\\Server20222\C\programdata\microsoft\search\data\applications\windows\Windows.edb
C:\Users\TestUser\Documents\sidr\target\debug\WIN-RAAM6FJLDR1_File_Report_20241018_144020.152772700.csv
C:\Users\TestUser\Documents\sidr\target\debug\WIN-RAAM6FJLDR1_Internet_History_Report_20241018_144020.153671600.csv
C:\Users\TestUser\Documents\sidr\target\debug\WIN-RAAM6FJLDR1_Activity_History_Report_20241018_144020.153832800.csv
Thanks for presenting about this project at VeloCon ! It was a great presentation!
When I went to play with it on my machine I could not get it to work - it was crashing with
This file parses fine using Velociraptor's SQLiteHunter and the underlying parse_ese(). You can verify with the binary from https://github.com/Velocidex/go-ese
eseparser dump Windows.edb
Here is the sample file. Windows.edb.gz