Closed Banaanhangwagen closed 4 years ago
Seems like there might be slight corruption in one of the records, and I don't handle the corruption. I will update the code shortly to fix this.
Also, I am curious as why this took 1 hour to read the apfs volumes. Are you on a slow usb2 external disk? This should normally only take 5-10 minutes.
I just updated the code. Can you run from code or do you want me to create an EXE to test this?
It would be nice if you would make a ready-to-use-EXE.
OK, try the new release here: https://github.com/ydkhatri/mac_apt/releases/tag/v0.7.dev
This version should bypass the bad data. Also, send me the debug log lines which should start with:
DEBUG values of row = ...
That should help in further debugging.
Thank you for your quick response! The image is on a external hdd connected with usb3. As asked I ran the v0.7dev; now there are some other errors
2020-06-03 08:20:09|MAIN|INFO|Started macOS Artifact Parsing Tool, version 0.7.dev
2020-06-03 08:20:09|MAIN|INFO|Dates and times are in UTC unless the specific artifact being parsed saves it as local time!
2020-06-03 08:20:09|MAIN|DEBUG|mac_apt.x64.exe -o OUT -x -l DEBUG E01 IMAGE.E01 ALL
2020-06-03 08:20:09|MAIN|INFO|Pytsk version = 20170801
2020-06-03 08:20:09|MAIN|INFO|Pyewf version = 20190317
2020-06-03 08:20:09|MAIN|INFO|Pyvmdk version = 20190316
2020-06-03 08:20:09|MAIN|INFO|PyAFF4 version = 0.31
2020-06-03 08:20:16|MAIN|INFO|Opened image IMAGE.E01
2020-06-03 08:20:16|MAIN|DEBUG|Skipping EFI System Partition @ offset 20480
2020-06-03 08:20:16|MAIN|INFO|Looking at FS with volume label 'Customer' @ offset 209735680
2020-06-03 08:20:16|MAIN|INFO|Found an APFS container with uuid: 58E18E64-2D21-442D-BA0C-21EA4F6D60BE
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|self.is_sw_encrypted = False
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|There are 4 volumes in this container
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|Volume Block IDs: [1027, 1030, 154008, 195483], Mapping-omap: 2681122
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|Volume Blocks:{1027: 1407544, 1030: 1646153, 154008: 1476348, 195483: 1382481}
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| -- Volume information:
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol name = Macintosh HD
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num files = 1327877
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num dirs = 294609
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol used = 821.10 GiB
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| incompatible_features=0x3, fs_flags=0x1
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|root_block_num = 2671243
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| -- Volume information:
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol name = Preboot
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num files = 60
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num dirs = 17
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol used = 20.87 MiB
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| incompatible_features=0x1, fs_flags=0x1
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|root_block_num = 2662063
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| -- Volume information:
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol name = Recovery
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num files = 17
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num dirs = 2
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol used = 490.60 MiB
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| incompatible_features=0x1, fs_flags=0x1
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|root_block_num = 2668965
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| -- Volume information:
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol name = VM
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num files = 2
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Num dirs = 0
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| Vol used = 5.00 GiB
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG| incompatible_features=0x1, fs_flags=0x1
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|root_block_num = 2669615
2020-06-03 08:20:16|MAIN|INFO|Reading APFS volumes from container, this may take a few minutes ...
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|INFO|Vol_2_Preboot Type=inode Count=79
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|INFO|Vol_2_Preboot Type=xattr Count=7
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|INFO|Vol_2_Preboot Type=dstream_id Count=60
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|INFO|Vol_2_Preboot Type=file_extent Count=64
2020-06-03 08:20:16|MAIN.HELPERS.APFS_READER|INFO|Vol_2_Preboot Type=dir_rec Count=79
2020-06-03 08:20:18|MAIN.HELPERS.APFS_READER|WARNING|Block values are deleted? ,block=2092400
<snip>
2020-06-03 09:04:29|MAIN.HELPERS.APFS_READER|WARNING|Block values are deleted? ,block=1380069
2020-06-03 09:05:46|MAIN.HELPERS.APFS_READER|ERROR|Perhaps a corrupted record in APFS volume, skipping it.From populate_compressed_files_table(). Got NULL for block number
2020-06-03 09:05:46|MAIN.HELPERS.APFS_READER|ERROR|DEBUG values of row = (137028, 12886162046, 12886176873, 3203, None, 12886162077, None, 0)
2020-06-03 09:06:30|MAIN.HELPERS.APFS_READER|DEBUG|244165 rows deleted
2020-06-03 09:06:30|MAIN.HELPERS.APFS_READER|DEBUG|4518 rows deleted
2020-06-03 09:06:32|MAIN.HELPERS.APFS_READER|DEBUG|731642 rows deleted
2020-06-03 09:06:35|MAIN.HELPERS.APFS_READER|DEBUG|453861 rows deleted
2020-06-03 09:06:37|MAIN.HELPERS.APFS_READER|DEBUG|622701 rows deleted
2020-06-03 09:06:37|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:06:38|MAIN.HELPERS.APFS_READER|DEBUG|386529 rows deleted
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=inode Count=2140788
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=xattr Count=1112356
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=sibling_link Count=43645
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=dstream_id Count=1148606
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=file_extent Count=2641676
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=dir_rec Count=2309793
2020-06-03 09:07:12|MAIN.HELPERS.APFS_READER|INFO|Vol_1_Macintosh_HD Type=sibling_map Count=43959
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|1 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|1 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=inode Count=21
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=xattr Count=2
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=sibling_link Count=2
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=dstream_id Count=17
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=file_extent Count=74
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=dir_rec Count=22
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_3_Recovery Type=sibling_map Count=2
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|0 rows deleted
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_4_VM Type=inode Count=4
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_4_VM Type=dstream_id Count=2
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_4_VM Type=file_extent Count=321
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|INFO|Vol_4_VM Type=dir_rec Count=4
2020-06-03 09:07:13|MAIN|INFO|Found valid OSX/macOS kernel
2020-06-03 09:07:13|MAIN.HELPERS.MACINFO|DEBUG|Trying to get system version from /System/Library/CoreServices/SystemVersion.plist
2020-06-03 09:07:13|MAIN.HELPERS.APFS_READER|DEBUG|Trying to open file : /System/Library/CoreServices/SystemVersion.plist
2020-06-03 09:07:13|MAIN.HELPERS.MACINFO|INFO|macOS version detected is: Mojave (10.14.1) Build=18B75
2020-06-03 09:07:14|MAIN.HELPERS.APFS_READER|DEBUG|Trying to open file : /private/var/db/dslocal/nodes/Default/users/_krbfast.plist
2020-06-03 09:07:14|MAIN.HELPERS.APFS_READER|DEBUG|Trying to copy out /private/var/db/dslocal/nodes/Default/users/_krbfast.plist
2020-06-03 09:07:15|MAIN.HELPERS.APFS_READER|DEBUG|Trying to open file : /private/var/db/dslocal/nodes/Default/users/_krbtgt.plist
2020-06-03 09:07:15|MAIN.HELPERS.APFS_READER|DEBUG|Trying to copy out /private/var/db/dslocal/nodes/Default/users/_krbtgt.plist
2020-06-03 09:07:15|MAIN.HELPERS.APFS_READER|DEBUG|Trying to open file : /private/var/db/dslocal/nodes/Default/users/_serialnumberd.plist
2020-06-03 09:07:15|MAIN.HELPERS.APFS_READER|DEBUG|Trying to copy out /private/var/db/dslocal/nodes/Default/users/_serialnumberd.plist
2020-06-03 09:07:15|MAIN|INFO|Sqlite db could not be created at : OUT\APFS_Volumes_58E18E64-2D21-442D-BA0C-21EA4F6D60BE.db
2020-06-03 09:07:15|MAIN|ERROR|Exception occurred when trying to create APFS_Volumes Sqlite db
Traceback (most recent call last):
File "site-packages\biplist\__init__.py", line 126, in readPlist
File "site-packages\biplist\__init__.py", line 234, in parse
File "site-packages\biplist\__init__.py", line 248, in readRoot
biplist.NotBinaryPlistException
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages\biplist\__init__.py", line 138, in readPlist
File "plistlib.py", line 959, in loads
File "plistlib.py", line 944, in load
plistlib.InvalidFileException: Invalid file
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "plugins\helpers\macinfo.py", line 908, in _GetUserInfo
File "site-packages\biplist\__init__.py", line 143, in readPlist
biplist.InvalidPlistException: Invalid file
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mac_apt_compiled.py", line 382, in FindMacOsPartitionInApfsContainer
File "mac_apt_compiled.py", line 257, in FindMacOsFiles
File "plugins\helpers\macinfo.py", line 938, in _GetUserInfo
NameError: name 'InvalidPlistException' is not defined
2020-06-03 09:07:15|MAIN.DISK_REPORT|INFO|Disk info
2020-06-03 09:07:15|MAIN.DISK_REPORT|INFO|Disk Size = 931.51 GB (1000204886016 bytes)
2020-06-03 09:07:15|MAIN.DISK_REPORT|INFO|Part Scheme = GPT
2020-06-03 09:07:15|MAIN.DISK_REPORT|INFO|Block size = 512 bytes
2020-06-03 09:07:15|MAIN.DISK_REPORT|INFO|Num Sectors = 1953525168.0
2020-06-03 09:07:15|MAIN.HELPERS.WRITER|DEBUG|Trying to write out disk, partition & volume information
2020-06-03 09:07:15|MAIN|WARNING|:( Could not find a partition having a macOS installation on it
2020-06-03 09:07:15|MAIN|INFO|--------------------------------------------------
2020-06-03 09:07:15|MAIN|INFO|Finished in time = 00:47:05
2020-06-03 09:07:15|MAIN|INFO|Review the Log file and report any ERRORs or EXCEPTIONS to the developers
It seems some files might be corrupted, but I can debug better if I have the APFS database. Any chance you can share the database file APFS_Volumes_58E18E64-2D21-442D-BA0C-21EA4F6D60BE.db
with me? Don't post it here, you can send privately to me at yogesh@swiftforensics.com
Thanks. You've got mail !
The database looks fine. It is likely that there is disk corruption and the specific file /private/var/db/dslocal/nodes/Default/users/_serialnumberd.plist
either had an invalid plist or all zeroes returned from the image. Another possibility is that libewf (dependancy) may be returning bad data for certain sectors. Can you confirm that XWF can read this file?
I have added some more exception handling to skip this error and move along. The files under release have been updated. Please take a look. Put the APFS db in the output folder and the script will not need to re-create it saving you a lot of time.
I think it's working now, mac_apt created successfully a XLSX with results in it. Thank you.
First let me show how XWF sees the _serialnumberd.plist
There are a lot of errors on the other hand.
A quick view learns me that it is mostly could not read plist
Are you interested in the logfile ?
Glad it works now, and thanks for helping make open source software better. Yes please send the log file. Also can you send me this plist too? I'd like to confirm whether the plist is the problem or libewf?
OK, since the plist exported by mac_apt is all zeroes, this is a libewf problem. I will see if I can make an alternate version of mac_apt with a different version of libewf.
If you can test this on Linux or Windows Subsystem for Linux, try that. I have detailed installation instructions here. The default installation on linux/WSL uses a different libewf verison than what is packaged in the windows exe.
Finally could do some testing on a Linux.
The proposed libewf-20140808.tar.gz
is a no-go. It can't even open the image.
Tried also the latest dev-version of libewf (via git clone), but the problem 'cannot correctly read plist' persists.
Well, since there is only one open source ewf library, this is beyond us at this point.. Perhaps a reacquire would fix it, but not worth the time and effort! This seems to be a known documented issue here https://github.com/libyal/libewf/issues/93
Since you have already installed the latest libewf, one thing you could try on linux is to use ewfmount to mount the raw image. Then point mac_apt to the raw dd image provided by ewfmount. Run everything as sudo.
Same error reading plist.
Taking a new image isn't possible any more. I'll leave it for what it is. Thank you for your time!
One last thing I would suggest. Since this is E01, you can use encase's "Mount as Network Share" option to access the files and folders via a logical drive letter. Once you have it setup, you can point mac_apt to it, using the MOUNTED option. You will have to use mac_apt_mounted_sys_data for that.
To access this, you need to right-click on the container in Encase (left on tree view), select DEVICE -> SHARE->MOUNT AS NETWORK SHARE.
I believe Xways has a similar option too, "Mount as drive letter". But it requires some external components to be installed separately (Dokan..)
I don't have acces to Encase, so I tried xwf. I succesfully mounted it (all files, even selected hidden/system) There was an error:
mac_apt.x64.exe -o "OUT" -x MOUNTED "X:\Macintosh HD" WIFI
Output path was : OUT
MAIN-INFO-Started macOS Artifact Parsing Tool, version 0.7.dev
MAIN-INFO-Dates and times are in UTC unless the specific artifact being parsed saves it as local time!
MAIN-INFO-Pytsk version = 20170801
MAIN-INFO-Pyewf version = 20190317
MAIN-INFO-Pyvmdk version = 20190316
MAIN-INFO-PyAFF4 version = 0.31
MAIN-INFO-Found valid OSX/macOS kernel
MAIN.HELPERS.MACINFO-ERROR-Could not get ProductVersion from plist. Is it a valid xml plist? Error=[Errno 22] Invalid argument
MAIN-ERROR-Failed to load image. Error Details are: 'MountedMacInfo' object has no attribute '_GetDarwinFoldersInfos'
You need to point the script to the root folder. Try
mac_apt.x64.exe -o "OUT" -x MOUNTED "X:\Macintosh HD\root" WIFI
Edit - Above does not apply to xways, as is does not present a root folder.
OK wait, I think there is also a bug. Hold on, let me investigate..
OK, all fixed now. MOUNTED mode issues have been fixed. New build available under releases.
Edit - This works with encase mounted. But Xways mounted volume cannot be accessed in python. Probably a dokan thing, needs more investigation..
Cool, thank you. It continued now. But at the end, there was still the error "could not open plist".
I sent you the debug log by mail, just fyi.
It is now an xways problem.. Files mounted by xways are inaccessible in python..
OK, I've added some code to work around the python XWF problem. This one should work now with XWF mounted files. https://github.com/ydkhatri/mac_apt/releases/tag/v0.7.dev.20200625
Allright!
I believe it might have worked now with MOUNTED
. I ran FAST and there were no more errors; also the Excel-output is much more complete (as expected)
Thank you for your work and time!
To be complete: only one plugin had some difficulties with plist, namely AUTOSTART Example:
2020-06-26 23:03:33|MAIN.HELPERS.MACINFO|DEBUG|Trying to open file : X:Macintosh HD\System\Library\LaunchDaemons\bootps.plist
2020-06-26 23:03:33|MAIN.HELPERS.MACINFO|DEBUG|Trying to open plist file : /System/Library/LaunchDaemons/bootps.plist
2020-06-26 23:03:33|MAIN.HELPERS.MACINFO|DEBUG|Trying to open file : X:Macintosh HD\System\Library\LaunchDaemons\bootps.plist
2020-06-26 23:03:33|MAIN.HELPERS.MACINFO|DEBUG|Trying to read plist file : /System/Library/LaunchDaemons/bootps.plist
2020-06-26 23:03:33|MAIN.AUTOSTART|ERROR|Problem reading plist - Could not read plist: /System/Library/LaunchDaemons/bootps.plist Error was : Invalid file
To be complete: only one plugin had some difficulties with plist, namely AUTOSTART
I had the same errors during testing. This is due to zero file size for those files. XWF's apfs parsing is flawed, these files do have file size, but you can look them up in the xwf gui, and it will show 0 bytes.
Appreciate you help in testing this code. 😄
I succesfully took an image with MacQuisition of a hdd 1TB (disk0) APFS is configured but not locked/encrypted.
Importing the image in X-Ways goes as expected.
However, when executing
mac_apt
it says that the APFS volume could not be found. Here's the output with log on DEBUG.