search

sleuthkit / autopsy

Autopsy® is a digital forensics platform and graphical interface to The Sleuth Kit® and other digital forensics tools. It can be used by law enforcement, military, and corporate examiners to investigate what happened on a computer. You can even use it to recover photos from your camera's memory card.
http://www.sleuthkit.org/autopsy/
2.42k stars 596 forks source link

4.15.0 crashes when adding APFS volume #5903

Open mattgenious opened 4 years ago

mattgenious commented 4 years ago

Both 4.14.0 and 4.15.0 crashes with no error message when trying to add an APFS volume. I am running windows 10 1909 and have tried adding the harddrive directly and have also produced an image and tried on that -> same result. I have tried running normally and as admin. It crashes while running through the filesystem in the add data source dialog.

APriestman commented 4 years ago

We're going to need more information. Is there any chance the image is small-ish and shareable? Also check to see if you have any hs_err_pidXXXX.log files in (I believe) Users/(you)/AppData/Roaming/autopsy.

mattgenious commented 4 years ago

The image is 120gb so not so easily shareable, I will check the location you mentioned for log files when I get home 👍

APriestman commented 4 years ago

It seems like the log file we need might be in a different spot: If Autopsy has been installed to the system drive (usually C:) the file should end up in the %TEMP% folder. If Autopsy is installed elsewhere it should end up in the root of the Autopsy installation (e.g. if I install to "E:\Program Files" you should be able to find it in "E:\Program Files\Autopsy-4.15.0").

mattgenious commented 4 years ago

Found it, it was a bit big so I put it in a pastebin https://pastebin.com/XpiUXnw0

APriestman commented 4 years ago

Thanks. Unfortunately it doesn't really help - I was hoping the line it crashed on would be in our code so we'd at least have a spot to look at but it's not. I don't think we're going to be able to do anything further.

mattgenious commented 4 years ago

What would be the command to run in TSK to see if it works there? Maybe that could yield more debug information? Would it be feasible for me to debug the code manually, what IDE do you guys use?

mattgenious commented 4 years ago

Oooh, wait I misread, so it seems the fault is with the JRE right?

APriestman commented 4 years ago

If you're willing to do some debugging that'd be great. On Windows, we build TSK in Visual Studio. Instructions are here: https://github.com/sleuthkit/sleuthkit/blob/develop/win32/BUILDING.txt

No I don't think it's in the JRE. It's more that it's crashing in some library method and I'm not going to be able to trace that back to our code.

About TSK tools - you could just run tsk_gettimes or tsk_loaddb to attempt to run on all the files in the image (tsk_loaddb is basically what Autopsy runs to add the image to the database). There's a -v flag for verbose output, but it tends to print too much to be helpful. If you have the ability to run on linux, then running valgrind could help pinpoint the issue..

You could also do things a bit more step-by-step like this (forgive some of the auto-formatting in this section):

Run mmls to see the partitions (if you don't see paritions just go to the next step and don't enter an offset):

mmls.exe apfs_one_vol.dmg GUID Partition Table (EFI) Offset Sector: 0 Units are in 512-byte sectors

  Slot      Start        End          Length       Description

000: Meta 0000000000 0000000000 0000000001 Safety Table 001: ------- 0000000000 0000000039 0000000040 Unallocated 002: Meta 0000000001 0000000001 0000000001 GPT Header 003: Meta 0000000002 0000000033 0000000032 Partition Table 004: 000 0000000040 0000097663 0000097624 disk image 005: ------- 0000097664 0000097696 0000000033 Unallocated

We want that disk image one, which has an offset of 40. Next we have to run pstat to get info about the APFS pools:

pstat -o 40 apfs_one_vol.dmg POOL CONTAINER INFORMATION

Container cb1365d5-76ab-4559-be83-77f389c254e2

Type: APFS

NX Block Number: 0 NX oid: 1 NX xid: 12 Checkpoint Descriptor Block: 7

Capacity Ceiling (Size): 49983488 B Capacity In Use: 1724416 B Capacity Available: 48259072 B

Block Size: 4096 B Number of Blocks: 12203 Number of Free Blocks: 11782 +-> Volume 8f8dda38-0894-49f6-a943-da1401ddd148 =========================================== APSB Block Number: 418 APSB oid: 1026 APSB xid: 12 Name (Role): Test APFS 1 (No specific role) Capacity Consumed: 737280 B Capacity Reserved: None Capacity Quota: None Case Sensitive: No Encrypted: No Formatted by: newfs_apfs (945.260.7)
Created: 2019-07-23 09:40:48.754498461 (Eastern Daylight Time)
Changed: 2019-07-23 09:44:42.771863706 (Eastern Daylight Time)
Unmount Logs
------------
Timestamp Log String
2019-07-23 09:44:42.848757968 (Eastern Daylight Time) apfs_kext (945.260.7)
Root Files
-------------
[ 23] file1.txt
[ 19] .DS_Store
[ 16] .fseventsd
[ 18] folder1

+-> Unallocated Container Blocks | ============================ | 0x000001a5-0x00002faa

And now we can use fls to look at that pool block (we got the pool block number from the line " APSB Block Number: 418"):

fls -B 418 -o 40 apfs_one_vol.dmg r/r 23: file1.txt r/r 19: .DS_Store d/d 16: .fseventsd d/d 18: folder1

mattgenious commented 4 years ago

Thanks, I will have a look at all of this when I have time, maybe in the next week or so 👍

mattgenious commented 4 years ago

mmls gave me:

GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Safety Table
001:  -------   0000000000   0000000039   0000000040   Unallocated
002:  Meta      0000000001   0000000001   0000000001   GPT Header
003:  Meta      0000000002   0000000033   0000000032   Partition Table
004:  000       0000000040   0000409639   0000409600   EFI System Partition
005:  001       0000409640   0236978135   0236568496   Customer
006:  -------   0236978136   0236978175   0000000040   Unallocated
pstat.exe -o 40 ..\..\..\new.img

returns

Invalid image offset (0)

but

pstat.exe -o 409640 ..\..\..\new.img

returns

POOL CONTAINER INFORMATION
--------------------------------------------

Container 16a6a189-2750-43cb-9647-76c57f7d9508
==============================================
Type: APFS

NX Block Number: 21362651
NX oid: 1
NX xid: 1658573
Checkpoint Descriptor Block: Not Found

Capacity Ceiling (Size): 121123069952 B

Block Size:            4096 B
Number of Blocks:      29571062
|
+-> Volume 2438d4bd-d13f-3fcc-9682-bd54a285af33
|   ===========================================
|   APSB Block Number: 474503
|   APSB oid: 1027
|   APSB xid: 1658573
|   Name (Role): Macintosh HD - data (Unknown)
|   Capacity Consumed: 91944067072 B
|   Capacity Reserved: None
|   Capacity Quota: None
|   Case Sensitive: No
|   Encrypted: No
|   Formatted by: hfs_convert (748.57.19)
|
|   Created: 2014-10-09 08:43:03.000000000 (Romance Daylight Time)
|   Changed: 2020-05-13 20:38:33.409444658 (Romance Daylight Time)
|
|   Unmount Logs
|   ------------
|   Timestamp                            Log String
|   2020-05-13 20:36:43.099048464 (Romance Daylight Time)  apfs_kext (1412.101.1)
|   2020-05-08 22:32:42.347596000 (Romance Daylight Time)  fsck_apfs (1412.101.1)
|   2020-05-08 22:30:30.992950935 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:01:49.223069544 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 21:54:33.375867713 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 21:53:12.253282756 (Romance Daylight Time)  apfs_kext compiled @ Apr 24 201
|   2020-05-08 21:44:22.855313322 (Romance Daylight Time)  apfs_kext compiled @ Apr 24 201
|   2020-01-04 09:32:39.015968297 (Romance Standard Time)  apfs_kext compiled @ Apr 24 201
|
|   Root Files
|   -------------
|  [8598671392] sw
|  [      19] .HFS+ Private Directory Data
|  [8598671393] home
|  [8598671394] usr
|  [  604869] .Spotlight-V100
|  [8589934669] .DS_Store
|  [  401309] .PKInstallSandboxManager
|  [8598671724] .installer-compatibility
|  [13713718] .PKInstallSandboxManager-SystemSoftware
|  [  609441] Brugeroplysninger
|  [8598671725] .TempReceipt.bom
|  [8589934670] .file
|  [8598671726] Library
|  [  605097] .Trashes
|  [8598689864] System
|  [8598660272] .OSInstallerMessages
|  [8598691961] mnt
|  [  605055] .fseventsd
|  [8598671232] private
|  [  614036] .DocumentRevisions-V100
|  [8589934671] .vol
|  [  168917] Users
|  [8598692671] Applications
|  [8598694917] opt
|  [8598694918] Volumes
|  [8598694919] .TemporaryItems
|  [ 1479058] .apdisk
|  [8598694920] cores
|
+-> Volume cb8f51da-26ad-49ae-a741-1bfe879d70d3
|   ===========================================
|   APSB Block Number: 466471
|   APSB oid: 264524
|   APSB xid: 1658564
|   Name (Role): Preboot (Preboot)
|   Capacity Consumed: 81285120 B
|   Capacity Reserved: None
|   Capacity Quota: None
|   Case Sensitive: No
|   Encrypted: No
|   Formatted by: newfs_apfs (748.57.19)
|
|   Created: 2018-05-24 20:17:11.631786614 (Romance Daylight Time)
|   Changed: 2020-05-13 20:24:13.644362661 (Romance Daylight Time)
|
|   Unmount Logs
|   ------------
|   Timestamp                            Log String
|   2020-05-13 20:38:27.331675570 (Romance Daylight Time)  apfs_kext (1412.101.1)
|   2020-05-08 22:30:29.049671219 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:29:38.861336024 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:19:49.477524727 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:03:37.297273220 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:03:07.446034738 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:01:37.332254031 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:01:37.024445464 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|
|   Root Files
|   -------------
|  [    3102] 4E7843B3-0155-4651-A671-B5D31BF7BB9A
|
+-> Volume 24d752ec-14fd-4b66-8e18-f772081a5335
|   ===========================================
|   APSB Block Number: 472823
|   APSB oid: 264526
|   APSB xid: 1658549
|   Name (Role): Recovery (Recovery)
|   Capacity Consumed: 528076800 B
|   Capacity Reserved: None
|   Capacity Quota: None
|   Case Sensitive: No
|   Encrypted: No
|   Formatted by: newfs_apfs (748.57.19)
|
|   Created: 2018-05-24 20:17:12.843820063 (Romance Daylight Time)
|   Changed: 2020-05-13 20:38:15.489615626 (Romance Daylight Time)
|
|   Unmount Logs
|   ------------
|   Timestamp                            Log String
|   2020-05-13 20:38:15.498878448 (Romance Daylight Time)  apfs_kext (1412.101.1)
|   2020-05-08 22:29:38.938679695 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:19:49.564498172 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:19:15.993909116 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:18:34.359571077 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:05:33.703863086 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:04:14.853253264 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 22:03:53.306559477 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|
|   Root Files
|   -------------
|  [     136] 4E7843B3-0155-4651-A671-B5D31BF7BB9A
|
+-> Volume 74cb7b67-0f9a-4ef1-b649-6e984549814d
|   ===========================================
|   APSB Block Number: 474493
|   APSB oid: 305668
|   APSB xid: 1658573
|   Name (Role): VM (VM)
|   Capacity Consumed: 1074814976 B
|   Capacity Reserved: None
|   Capacity Quota: None
|   Case Sensitive: No
|   Encrypted: No
|   Formatted by: newfs_apfs (748.57.19)
|
|   Created: 2018-05-24 20:35:19.552372276 (Romance Daylight Time)
|   Changed: 2020-05-13 20:38:33.183203120 (Romance Daylight Time)
|
|   Unmount Logs
|   ------------
|   Timestamp                            Log String
|   2020-05-13 20:38:33.185547990 (Romance Daylight Time)  apfs_kext (1412.101.1)
|   2020-05-08 22:30:29.546991673 (Romance Daylight Time)  apfs_kext compiled @ Feb 19 202
|   2020-05-08 21:53:11.539518881 (Romance Daylight Time)  apfs_kext compiled @ Apr 24 201
|   2020-05-08 21:43:52.696604834 (Romance Daylight Time)  apfs_kext compiled @ Apr 24 201
|   2020-01-04 09:32:37.150857928 (Romance Standard Time)  apfs_kext compiled @ Apr 24 201
|   2020-01-03 09:59:45.587808289 (Romance Standard Time)  apfs_kext compiled @ Apr 24 201
|   2019-12-27 10:37:21.645835281 (Romance Standard Time)  apfs_kext compiled @ Apr 24 201
|   2019-07-16 09:11:51.503240274 (Romance Daylight Time)  apfs_kext compiled @ Dec 20 201
|
|   Root Files
|   -------------
|  [    1116] kernelcore
|  [      16] sleepimage
|
+-> Volume 4e7843b3-0155-4651-a671-b5d31bf7bb9a
|   ===========================================
|   APSB Block Number: 462304
|   APSB oid: 1592406
|   APSB xid: 1658475
|   Name (Role): Macintosh HD (System)
|   Capacity Consumed: 11001556992 B
|   Capacity Reserved: None
|   Capacity Quota: None
|   Case Sensitive: No
|   Encrypted: No
|   Formatted by: diskmanagementd (1412.101.1)
|
|   Created: 2020-05-08 22:36:21.164413784 (Romance Daylight Time)
|   Changed: 2020-05-13 20:36:43.067285098 (Romance Daylight Time)
|
|   Unmount Logs
|   ------------
|   Timestamp                            Log String
|   2020-05-13 20:36:43.082681904 (Romance Daylight Time)  apfs_kext (1412.101.1)
|
|   Root Files
|   -------------
|  [1152921500311879711] usr
|  [1152921500311902383] .DS_Store
|  [1152921500311902384] bin
|  [1152921500311902457] sbin
|  [1152921500311902559] .file
|  [1152921500311902560] etc
|  [1152921500312400543] var
|  [1152921500312400544] Library
|  [1152921500311902561] System
|  [1152921500312400546] .VolumeIcon.icns
|  [1152921500311879696] .fseventsd
|  [1152921500312400545] private
|  [1152921500312400547] .vol
|  [1152921500312400554] Users
|  [1152921500312400552] Applications
|  [1152921500312400548] opt
|  [1152921500312400549] dev
|  [1152921500312400550] Volumes
|  [1152921500312400551] tmp
|  [1152921500312400553] cores

So it seems the issue is with the efi system partition??? I'm having a little trouble understanding the output here hahaha

APriestman commented 4 years ago

No the EFI partition is probably fine - it's not a pool so pstat (pool stat) failed, but that doesn't mean it's corrupt or anything.

That customer volume is an APFS pool. pstat gives you the starting block and other information about each APFS volume contained in the pool. It looks like you have five. You can use other tsk tools to poke around in them, but it seems like this is a pretty large image and you wouldn't want to look around the whole thing manually. https://wiki.sleuthkit.org/index.php?title=TSK_Tool_Overview

Can you try running tsk_loaddb on your image? It should just take the image file name as a parameter. I would expect it to crash (though I also kind of thought pstat might crash so who knows).

Right now the only thing standing out to me is that the inodes (the numbers in brackets) are very large in that final partition.

mattgenious commented 4 years ago

I am trying to run tsk_loaddb targeting an empty case db from autopsy as the db, it did not seem to work but now I am trying with verbose and it sure is working for a long time so maybe it will work. Is it correctly understood that I should target the db file called [CASENAME].db in the root of the case directory?

mattgenious commented 4 years ago

it does not seem to be working, it created a db file at 1024KB size and a db-journal file of size 101KB without any arguments other than the image, so it seems to not want to load. The last of the verbose stuff was

raw_read: byte offset: 209725440 len: 65536
raw_read: found in image 0 relative offset: 209725440 len: 65536
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
fatfs_is_83_name: name[0] is invalid
tsk_fs_dir_find_orphans: De-duping orphan files and directories
raw_read: byte offset: 87710109696 len: 65536
raw_read: found in image 0 relative offset: 87710109696 len: 65536
raw_read: byte offset: 87710175232 len: 65536
raw_read: found in image 0 relative offset: 87710175232 len: 65536
raw_read: byte offset: 87710240768 len: 65536
raw_read: found in image 0 relative offset: 87710240768 len: 65536
raw_read: byte offset: 87710306304 len: 65536
raw_read: found in image 0 relative offset: 87710306304 len: 65536
raw_read: byte offset: 87710371840 len: 65536
raw_read: found in image 0 relative offset: 87710371840 len: 65536
raw_read: byte offset: 87710437376 len: 65536
raw_read: found in image 0 relative offset: 87710437376 len: 65536
raw_read: byte offset: 87710502912 len: 65536
raw_read: found in image 0 relative offset: 87710502912 len: 65536
raw_read: byte offset: 87710568448 len: 65536
raw_read: found in image 0 relative offset: 87710568448 len: 65536
raw_read: byte offset: 87710633984 len: 65536
raw_read: found in image 0 relative offset: 87710633984 len: 65536
raw_read: byte offset: 87710699520 len: 65536
raw_read: found in image 0 relative offset: 87710699520 len: 65536
raw_read: byte offset: 87710765056 len: 65536
raw_read: found in image 0 relative offset: 87710765056 len: 65536
raw_read: byte offset: 87710830592 len: 65536
raw_read: found in image 0 relative offset: 87710830592 len: 65536
raw_read: byte offset: 87710896128 len: 65536
raw_read: found in image 0 relative offset: 87710896128 len: 65536
raw_read: byte offset: 87710961664 len: 65536
raw_read: found in image 0 relative offset: 87710961664 len: 65536
raw_read: byte offset: 87711027200 len: 65536
raw_read: found in image 0 relative offset: 87711027200 len: 65536
raw_read: byte offset: 87711092736 len: 65536
raw_read: found in image 0 relative offset: 87711092736 len: 65536
raw_read: byte offset: 87711158272 len: 65536
raw_read: found in image 0 relative offset: 87711158272 len: 65536
raw_read: byte offset: 87711223808 len: 65536
raw_read: found in image 0 relative offset: 87711223808 len: 65536
raw_read: byte offset: 2153271296 len: 65536
raw_read: found in image 0 relative offset: 2153271296 len: 65536
raw_read: byte offset: 2120400896 len: 65536
raw_read: found in image 0 relative offset: 2120400896 len: 65536
raw_read: byte offset: 2146418688 len: 65536
raw_read: found in image 0 relative offset: 2146418688 len: 65536
raw_read: byte offset: 2153259008 len: 65536
raw_read: found in image 0 relative offset: 2153259008 len: 65536
raw_read: byte offset: 2103332864 len: 65536
raw_read: found in image 0 relative offset: 2103332864 len: 65536
raw_read: byte offset: 2102353920 len: 65536
raw_read: found in image 0 relative offset: 2102353920 len: 65536
raw_read: byte offset: 2103504896 len: 65536
raw_read: found in image 0 relative offset: 2103504896 len: 65536
raw_read: byte offset: 2109018112 len: 65536
raw_read: found in image 0 relative offset: 2109018112 len: 65536
raw_read: byte offset: 2127880192 len: 65536
raw_read: found in image 0 relative offset: 2127880192 len: 65536
raw_read: byte offset: 2137190400 len: 65536
raw_read: found in image 0 relative offset: 2137190400 len: 65536
raw_read: byte offset: 2223505408 len: 65536
raw_read: found in image 0 relative offset: 2223505408 len: 65536
raw_read: byte offset: 2121211904 len: 65536
raw_read: found in image 0 relative offset: 2121211904 len: 65536
raw_read: byte offset: 2127142912 len: 65536
raw_read: found in image 0 relative offset: 2127142912 len: 65536
APFS dir_open_meta: Processing directory 2
raw_read: byte offset: 2106466304 len: 65536
raw_read: found in image 0 relative offset: 2106466304 len: 65536
raw_read: byte offset: 2105032704 len: 65536
raw_read: found in image 0 relative offset: 2105032704 len: 65536
raw_read: byte offset: 2140667904 len: 65536
raw_read: found in image 0 relative offset: 2140667904 len: 65536
raw_read: byte offset: 2140508160 len: 65536
raw_read: found in image 0 relative offset: 2140508160 len: 65536
raw_read: byte offset: 2149478400 len: 65536
raw_read: found in image 0 relative offset: 2149478400 len: 65536
raw_read: byte offset: 2298404864 len: 65536
raw_read: found in image 0 relative offset: 2298404864 len: 65536
raw_read: byte offset: 2320961536 len: 65536
raw_read: found in image 0 relative offset: 2320961536 len: 65536
raw_read: byte offset: 2104893440 len: 65536
raw_read: found in image 0 relative offset: 2104893440 len: 65536
raw_read: byte offset: 2535567360 len: 65536
raw_read: found in image 0 relative offset: 2535567360 len: 65536
raw_read: byte offset: 2147266560 len: 65536
raw_read: found in image 0 relative offset: 2147266560 len: 65536
raw_read: byte offset: 2132889600 len: 65536
raw_read: found in image 0 relative offset: 2132889600 len: 65536
raw_read: byte offset: 2142863360 len: 65536
raw_read: found in image 0 relative offset: 2142863360 len: 65536
raw_read: byte offset: 2132381696 len: 65536
raw_read: found in image 0 relative offset: 2132381696 len: 65536
raw_read: byte offset: 2127781888 len: 65536
raw_read: found in image 0 relative offset: 2127781888 len: 65536
raw_read: byte offset: 2104225792 len: 65536
raw_read: found in image 0 relative offset: 2104225792 len: 65536
APFS dir_open_meta: Processing directory 8598671392
raw_read: byte offset: 2105241600 len: 65536
raw_read: found in image 0 relative offset: 2105241600 len: 65536
raw_read: byte offset: 2234396672 len: 65536
raw_read: found in image 0 relative offset: 2234396672 len: 65536
raw_read: byte offset: 2138263552 len: 65536
raw_read: found in image 0 relative offset: 2138263552 len: 65536
raw_read: byte offset: 2272632832 len: 65536
raw_read: found in image 0 relative offset: 2272632832 len: 65536
raw_read: byte offset: 2103762944 len: 65536
raw_read: found in image 0 relative offset: 2103762944 len: 65536
raw_read: byte offset: 2253017088 len: 65536
raw_read: found in image 0 relative offset: 2253017088 len: 65536
raw_read: byte offset: 2149318656 len: 65536
raw_read: found in image 0 relative offset: 2149318656 len: 65536
raw_read: byte offset: 2383138816 len: 65536
raw_read: found in image 0 relative offset: 2383138816 len: 65536
APFS dir_open_meta: Processing directory 19
APFS dir_open_meta: Processing directory 8598671393
APFS dir_open_meta: Processing directory 8598671394
APFS dir_open_meta: Processing directory 8598671395
APFS dir_open_meta: Processing directory 8598671396
APFS dir_open_meta: Processing directory 8598671397
raw_read: byte offset: 2152841216 len: 65536
raw_read: found in image 0 relative offset: 2152841216 len: 65536
raw_read: byte offset: 2416484352 len: 65536
raw_read: found in image 0 relative offset: 2416484352 len: 65536
raw_read: byte offset: 2416582656 len: 65536
raw_read: found in image 0 relative offset: 2416582656 len: 65536
APFS dir_open_meta: Processing directory 8598671405
APFS dir_open_meta: Processing directory 8598671410
APFS dir_open_meta: Processing directory 8598671415
APFS dir_open_meta: Processing directory 8598671416
raw_read: byte offset: 2152783872 len: 65536
raw_read: found in image 0 relative offset: 2152783872 len: 65536
raw_read: byte offset: 2489106432 len: 65536
raw_read: found in image 0 relative offset: 2489106432 len: 65536
raw_read: byte offset: 2128330752 len: 65536
raw_read: found in image 0 relative offset: 2128330752 len: 65536
raw_read: byte offset: 2204164096 len: 65536
raw_read: found in image 0 relative offset: 2204164096 len: 65536
raw_read: byte offset: 2673422336 len: 65536
raw_read: found in image 0 relative offset: 2673422336 len: 65536
raw_read: byte offset: 2186678272 len: 65536
raw_read: found in image 0 relative offset: 2186678272 len: 65536
raw_read: byte offset: 2138873856 len: 65536
raw_read: found in image 0 relative offset: 2138873856 len: 65536
raw_read: byte offset: 2321731584 len: 65536
raw_read: found in image 0 relative offset: 2321731584 len: 65536
raw_read: byte offset: 2695442432 len: 65536
raw_read: found in image 0 relative offset: 2695442432 len: 65536
raw_read: byte offset: 2218799104 len: 65536
raw_read: found in image 0 relative offset: 2218799104 len: 65536
raw_read: byte offset: 2206597120 len: 65536
raw_read: found in image 0 relative offset: 2206597120 len: 65536
raw_read: byte offset: 9274503168 len: 65536
raw_read: found in image 0 relative offset: 9274503168 len: 65536
raw_read: byte offset: 2150670336 len: 65536
raw_read: found in image 0 relative offset: 2150670336 len: 65536
raw_read: byte offset: 6013865984 len: 65536
raw_read: found in image 0 relative offset: 6013865984 len: 65536
raw_read: byte offset: 2142859264 len: 65536
raw_read: found in image 0 relative offset: 2142859264 len: 65536
raw_read: byte offset: 2490724352 len: 65536
raw_read: found in image 0 relative offset: 2490724352 len: 65536
raw_read: byte offset: 2135158784 len: 65536
raw_read: found in image 0 relative offset: 2135158784 len: 65536
raw_read: byte offset: 2241642496 len: 65536
raw_read: found in image 0 relative offset: 2241642496 len: 65536
raw_read: byte offset: 2152747008 len: 65536
raw_read: found in image 0 relative offset: 2152747008 len: 65536
raw_read: byte offset: 2516582400 len: 65536
raw_read: found in image 0 relative offset: 2516582400 len: 65536
raw_read: byte offset: 2148855808 len: 65536
raw_read: found in image 0 relative offset: 2148855808 len: 65536
raw_read: byte offset: 2443321344 len: 65536
raw_read: found in image 0 relative offset: 2443321344 len: 65536
raw_read: byte offset: 2132656128 len: 65536
raw_read: found in image 0 relative offset: 2132656128 len: 65536
raw_read: byte offset: 2132107264 len: 65536
raw_read: found in image 0 relative offset: 2132107264 len: 65536
raw_read: byte offset: 2175311872 len: 65536
raw_read: found in image 0 relative offset: 2175311872 len: 65536
raw_read: byte offset: 2235912192 len: 65536
raw_read: found in image 0 relative offset: 2235912192 len: 65536
raw_read: byte offset: 2189639680 len: 65536
raw_read: found in image 0 relative offset: 2189639680 len: 65536
raw_read: byte offset: 2551181312 len: 65536
raw_read: found in image 0 relative offset: 2551181312 len: 65536
raw_read: byte offset: 2416414720 len: 65536
raw_read: found in image 0 relative offset: 2416414720 len: 65536
APFS dir_open_meta: Processing directory 8598671441
raw_read: byte offset: 2126393344 len: 65536
raw_read: found in image 0 relative offset: 2126393344 len: 65536
APFS dir_open_meta: Processing directory 8598671450
APFS dir_open_meta: Processing directory 8598671461
raw_read: byte offset: 2416480256 len: 65536
raw_read: found in image 0 relative offset: 2416480256 len: 65536
APFS dir_open_meta: Processing directory 8598671466
APFS dir_open_meta: Processing directory 8598671503
raw_read: byte offset: 2117402624 len: 65536
raw_read: found in image 0 relative offset: 2117402624 len: 65536
raw_read: byte offset: 10899042304 len: 65536
raw_read: found in image 0 relative offset: 10899042304 len: 65536
raw_read: byte offset: 2150772736 len: 65536
raw_read: found in image 0 relative offset: 2150772736 len: 65536
raw_read: byte offset: 2112569344 len: 65536
raw_read: byte offset: 2112671744 len: 65536
raw_read: found in image 0 relative offset: 2112671744 len: 65536
raw_read: byte offset: 2226282496 len: 65536
raw_read: found in image 0 relative offset: 2226282496 len: 65536
raw_read: byte offset: 2136940544 len: 65536
raw_read: found in image 0 relative offset: 2136940544 len: 65536
raw_read: byte offset: 2152366080 len: 65536
raw_read: found in image 0 relative offset: 2152366080 len: 65536
APFS dir_open_meta: Processing directory 8590614732
raw_read: byte offset: 2118275072 len: 65536
raw_read: found in image 0 relative offset: 2118275072 len: 65536
raw_read: byte offset: 2536128512 len: 65536
raw_read: found in image 0 relative offset: 2536128512 len: 65536
raw_read: byte offset: 2241495040 len: 65536
raw_read: found in image 0 relative offset: 2241495040 len: 65536
raw_read: byte offset: 2171645952 len: 65536
raw_read: found in image 0 relative offset: 2171645952 len: 65536
raw_read: byte offset: 2241875968 len: 65536
raw_read: found in image 0 relative offset: 2241875968 len: 65536
raw_read: byte offset: 2449293312 len: 65536
raw_read: found in image 0 relative offset: 2449293312 len: 65536
raw_read: byte offset: 2150125568 len: 65536
raw_read: found in image 0 relative offset: 2150125568 len: 65536
raw_read: byte offset: 2319593472 len: 65536
raw_read: found in image 0 relative offset: 2319593472 len: 65536
APFS dir_open_meta: Processing directory 15910380
APFS dir_open_meta: Processing directory 15910381
APFS dir_open_meta: Processing directory 15910383
raw_read: byte offset: 2242179072 len: 65536
raw_read: found in image 0 relative offset: 2242179072 len: 65536
raw_read: byte offset: 2327941120 len: 65536
raw_read: found in image 0 relative offset: 2327941120 len: 65536
raw_read: byte offset: 2138681344 len: 65536
raw_read: found in image 0 relative offset: 2138681344 len: 65536
raw_read: byte offset: 2199588864 len: 65536
raw_read: found in image 0 relative offset: 2199588864 len: 65536
raw_read: byte offset: 2241314816 len: 65536
raw_read: found in image 0 relative offset: 2241314816 len: 65536
raw_read: byte offset: 2144620544 len: 65536
raw_read: found in image 0 relative offset: 2144620544 len: 65536
raw_read: byte offset: 2149507072 len: 65536
raw_read: found in image 0 relative offset: 2149507072 len: 65536
raw_read: byte offset: 2129301504 len: 65536
raw_read: found in image 0 relative offset: 2129301504 len: 65536
raw_read: byte offset: 2221506560 len: 65536
raw_read: found in image 0 relative offset: 2221506560 len: 65536
raw_read: byte offset: 4701835264 len: 65536
raw_read: found in image 0 relative offset: 4701835264 len: 65536
decmpfs_file_read_compressed_attr: Compressed data is inline in the attribute, will load this as the default DATA attribute.
mattgenious commented 4 years ago

fls gives the following:

fls.exe -B 462304 -o 409640 ..\..\..\new.img
d/d 1152921500311879711:        usr
r/r 1152921500311902383:        .DS_Store
d/d 1152921500311902384:        bin
d/d 1152921500311902457:        sbin
r/r 1152921500311902559:        .file
l/l 1152921500311902560:        etc
l/l 1152921500312400543:        var
d/d 1152921500312400544:        Library
d/d 1152921500311902561:        System
l/l 1152921500312400546:        .VolumeIcon.icns
d/d 1152921500311879696:        .fseventsd
d/d 1152921500312400545:        private
d/d 1152921500312400547:        .vol
d/d 1152921500312400554:        Users
d/d 1152921500312400552:        Applications
d/d 1152921500312400548:        opt
d/d 1152921500312400549:        dev
d/d 1152921500312400550:        Volumes
l/l 1152921500312400551:        tmp
d/d 1152921500312400553:        cores
mattgenious commented 4 years ago

just tried the same tsk_loaddb command as before but with an ntfs image that I know loads correctly and it worked as intended. I'm guessing next step for me is to run tsk_loaddb.exe in debug environment and see if I can find uncaught exceptions?

APriestman commented 4 years ago

I'm guessing next step for me is to run tsk_loaddb.exe in debug environment and see if I can find uncaught exceptions?

Yes I think that's the right next step.

mattgenious commented 4 years ago

Hi I have built the libraries, though libewf64 was an adventure haha. But I can't seem to get the environment variables to work, do I just set them in the project properties for the tsk_loaddb project and if so is

LIBEWF_HOME=C:\Work\libewf_64bit\msvscpp\libewf_dll\Release

not correct? I am still getting

Severity    Code    Description Project File    Line    Suppression State
Error   C1083   Cannot open include file: 'libewf.h': No such file or directory libtsk  C:\Work\sleuthkit\tsk\img\ewf.h 21

I am at a loss haha

markmckinnon commented 4 years ago

Try LIBEWF_HOME=C:\Work\libewf_64bit instead

mattgenious commented 4 years ago

thanks @markmckinnon still giving me the same error, my environment variables are 

LIBEWF_HOME=C:\Work\libewf_64bit
LIBVHDI_HOME=C:\Work\libvhdi_64bit
LIBVMDK_HOME=C:\Work\libvmdk_64bit\libvmdk

the linker addition library directories are:

$(LIBVMDK_HOME)\msvscpp\x64\release
$(LIBVHDI_HOME)\msvscpp\x64\release
$(LIBEWF_HOME)\msvscpp\x64\release

and I am still getting the following errors:

Severity    Code    Description Project File    Line    Suppression State
Error   LNK1181 cannot open input file 'libvhdi.lib'    tsk_loaddb  C:\Work\sleuthkit\win32\tsk_loaddb\LINK 1   
Error (active)  E1696   cannot open source file "libewf.h"  libtsk  C:\Work\sleuthkit\tsk\img\ewf.h 21  
Error (active)  E0020   identifier "libewf_handle_t" is undefined   libtsk  C:\Work\sleuthkit\tsk\img\ewf.h 39  
Error   C1083   Cannot open include file: 'libewf.h': No such file or directory libtsk  C:\Work\sleuthkit\tsk\img\ewf.h 21  
Error   C1083   Cannot open include file: 'libewf.h': No such file or directory libtsk  C:\Work\sleuthkit\tsk\img\ewf.h 21  
Error   C1083   Cannot open include file: 'libewf.h': No such file or directory libtsk  C:\Work\sleuthkit\tsk\img\ewf.h 21

I am trying to build Release -> x64

mattgenious commented 4 years ago

I have succesfully built and debugged x64. I hit an exception when running tsk_loaddb in debug mode, this is the info I collected, if you need anything else or more data I will be happy to deliver that.

Exception in line 697 of apfs.cpp.

Unhandled exception at 0x00007FF805A4A799 in tsk_loaddb.exe: Microsoft C++ exception: std::runtime_error at memory location 0x00000014CCCFD8F0.

        APFS_OBJ_TYPE_CHECKPOINT_DESC   APFS_OBJ_TYPE_CHECKPOINT_DESC (12)  APFS_OBJ_TYPE_ENUM
        block_num   0   const unsigned __int64
-       pool    {...}   const APFSPool & {APFSPoolCompat}
-       [APFSPoolCompat]    {...}   APFSPoolCompat
+       TSKPoolCompat<APFSPool,void>    {_info={tag=1347374156 ctype=TSK_POOL_TYPE_APFS (1) block_size=4096 ...} }  TSKPoolCompat<APFSPool,void>
-       TSKPool {_members={ size=1 } _uuid={_bytes={ size=16 } } _num_blocks=29571062 ...}  TSKPool
+       __vfptr 0x00007ff75cdb9cb0 {tsk_loaddb.exe!void(* APFSPoolCompat::`vftable'[6])()} {0x00007ff75cab327e {tsk_loaddb.exe!APFSPoolCompat::`vector deleting destructor'(unsigned int)}, ...}    void * *
+       _members    { size=1 }  std::vector<std::pair<TSK_IMG_INFO * const,__int64 const>,std::allocator<std::pair<TSK_IMG_INFO * const,__int64 const>>>
+       _uuid   {_bytes={ size=16 } }   Guid
        _num_blocks 29571062    unsigned __int64
        _num_vols   5   int
        _block_size 4096    unsigned int
        _dev_block_size 512 unsigned int
-       _img    0x00000150bbd8b070 {tag=958415409 itype=TSK_IMG_TYPE_RAW (1) size=121332826112 ...} TSK_IMG_INFO *
        tag 958415409   unsigned int
        itype   TSK_IMG_TYPE_RAW (1)    TSK_IMG_TYPE_ENUM
        size    121332826112    __int64
        num_img 1   int
        sector_size 512 unsigned int
        page_size   0   unsigned int
        spare_size  0   unsigned int
+       images  0x00000150ba344bf0 {0x00000150ba342c00 L"D:\\temp\\new\\new.img"}   wchar_t * *
+       cache_lock  {critical_section={DebugInfo=0xffffffffffffffff {Type=??? CreatorBackTraceIndex=??? CriticalSection=...} ...} } tsk_lock_t
+       cache   0x00000150bbd8b0c0 {0x00000150bbd8b0c0 "", 0x00000150bbd9b0c0 "L\x1fÑ—\x1cè:‡L÷E\x1", 0x00000150bbdab0c0 "ø¥8ãÈw\x14É\x1", ...} char[32][65536]
+       cache_off   0x00000150bbf8b0c0 {1593344, 87710568448, 209735680, 87711027200, 1527808, 87710371840, 1462272, 87710765056, ...}  __int64[32]
+       cache_age   0x00000150bbf8b1c0 {718, 847, 1000, 959, 704, 799, 688, 895, 624, 672, 544, 999, 592, 831, 943, 560, ...}   int[32]
+       cache_len   0x00000150bbf8b240 {65536, 65536, 65536, 65536, 65536, 65536, 65536, 65536, 65536, 65536, 65536, 65536, ...}    unsigned __int64[32]
        read    0x00007ff75cb2e2a0 {tsk_loaddb.exe!raw_read(TSK_IMG_INFO *, __int64, char *, unsigned __int64)} __int64(*)(TSK_IMG_INFO *, __int64, char *, unsigned __int64)
        close   0x00007ff75cb2e820 {tsk_loaddb.exe!raw_close(TSK_IMG_INFO *)}   void(*)(TSK_IMG_INFO *)
        imgstat 0x00007ff75cb2e690 {tsk_loaddb.exe!raw_imgstat(TSK_IMG_INFO *, _iobuf *)}   void(*)(TSK_IMG_INFO *, _iobuf *)
        _offset 209735680   __int64
        _nx_block_num   21362651    unsigned __int64
+       _vol_blocks { size=5 }  std::vector<unsigned __int64,std::allocator<unsigned __int64>>
+       _block_cache    { size=3574 }   std::unordered_map<unsigned __int64,lw_shared_ptr<APFSBlock>,std::hash<unsigned __int64>,std::equal_to<unsigned __int64>,std::allocator<std::pair<unsigned __int64 const ,lw_shared_ptr<APFSBlock>>>>
        _hw_crypto  false   bool
-       this    0x00000014cccfd970 {...}    APFSCheckpointMap *
-       APFSObject  {...}   APFSObject
-       APFSBlock   {_storage={ size=4096 } _pool={...} _block_num=0 }  APFSBlock
+       __vfptr 0x00007ff75ce0ddd0 {tsk_loaddb.exe!void(* APFSCheckpointMap::`vftable'[2])()} {0x00007ff75cab2b3f {tsk_loaddb.exe!APFSCheckpointMap::`vector deleting destructor'(unsigned int)}}   void * *
+       _storage    { size=4096 }   std::array<char,4096>
+       _pool   {...}   const APFSPool & {APFSPoolCompat}
        _block_num  0   const unsigned __int64
APriestman commented 4 years ago

Ok I'll take a look when I have some time. What sleuthkit branch were you using?

mattgenious commented 4 years ago

I am using the "develop" branch, should I use a different one? I also continued the debug and got a seemingly infinite loop of the following two exceptions occurring on the same line:

apfs.cpp at line 708.
Exception thrown: read access violation.
entry was 0x14CCCFFFE8.

and

Unhandled exception thrown: read access violation.
entry was 0x14CCCFFFE8.
APriestman commented 4 years ago

Oh no develop's fine - I just want to make sure I'm looking at the same code.