Open pwmoore opened 1 year ago
DecompressFile 10 => 21cd8000, algo = e (Unknown) Unsupported decompression algorithm.
This compression type 14 turns out to be the LZBITMAP type, which is part of `libcompression` on macOS. I added a switch case for 14 to the `IsDecompAlgoSupported()` and `IsDecompAlgoInRsrc()` functions in `Decmpfs.cpp`, and `apfs-fuse` reports the correct size, but obviously it still needs the algorithm. I am not sure if `apfs-fuse` can support this algorithm, because it is closed source. I did take a look at the `lzbitmap_decode_buffer()` function in `libcompression` on macOS and it's a lot of architecture specific vectorized instructions. I certainly understand if this is not something that can be added, either because of legal concerns with proprietary algorithms, or just lack of time to re-implement something like this. Either way, I just wanted to bring it to your attention.
I'm facing the same issue.. How do you know that the missing algorithm is LZBITMAP ?
So when I tried to mount with debug messages turned on I got an error stating "unknown compression type 14". I found an earlier issue #145 talking about compression type 9. They found some information in the Apple opensource copyfile, so I went and looked at the most recent version, and in copyfile.c we see:
switch (OSSwapLittleToHostInt32(hdr->compression_type)) {
// ... SNIP ...
case 13: /* LZBITMAP-compressed data in xattr */
case 14: /* 64k chunked LZBITMAP-compressed data in resource fork */
/* valid compression type, we want to copy. */
break;
case 5: /* specifies de-dup within the generation store. Don't copy decmpfs xattr. */
copyfile_debug(3, "compression_type <5> on attribute com.apple.decmpfs for src file %s is not copied.",
s->src ? s->src : "(null string)");
continue;
case 6: /* unused */
case 0x80000001: /* faulting files are deprecated, don't copy decmpfs xattr */
default:
copyfile_warn("Invalid compression_type <%d> on attribute %s for src file %s",
OSSwapLittleToHostInt32(hdr->compression_type), name, s->src ? s->src : "(null string)");
continue;
}
So that gave me enough hints to get started.
Did you started the RE ? are you working on it ? Or you are waiting for apfs-fuse to develop it ? @pwmoore
I'm working on it myself. I did most of the work already, the problem is that I'm not sure what _lzbitmap_decode should get as parameters ? Where I get the 64k chunks to decompress ? @pwmoore
I'm working on it myself. I did most of the work already, the problem is that I'm not sure what _lzbitmap_decode should get as parameters ? Where I get the 64k chunks to decompress ? @pwmoore
I haven't had an opportunity to work on it much further. If you look at the function DecompressFile()
in ApfsLib/Decmpfs.cpp you'll see that algorithm type 4 is a resource fork algorithm that decompresses 64K chunks, calling DecompressLZVN()
on each chunk. So I imagine you'd want to start with a DecompressLZBITMAP()
function with the same parameters. The chunking probably wouldn't be exactly the same... you might have to fiddle with it some, but it's probably a good example on how to get started.
However LZVN decompression works a little differently under the hood than LZBITMAP. On macOS, you get to lzbitmap_decode()
through the following callchain:
compression_decode_buffer()
lzbitmap_decode_buffer()
lzbitmap_decode()
And lzbitmap_decode_buffer()
just passes its arguments to lzbitmap_decode()
. You can take a look at DecompressLZFSE()
in ApfsLib/Util.cpp for a good example of how this works... LZFSE is an open source compression library developed by Apple that's used in libcompression
. They have a similar lzfse_decode_buffer()
function that compression_decode_buffer()
calls for LZFSE compression.
That's basically about as far as I was able to get... I work full time plus I'm in grad school with a family, so taking on another project just isn't on my plate right now :(
I ran ipsw extractor
on my mac, trying to listen to libcompression
calls, didn't found any call to compression_decode_buffer
.
Found only calls to compression_stream_process
.
Maybe the algo isn't LZBITMAP ? @pwmoore
So the apfs decompression is done in the kernel on the Mac. If you look at the copyfile.c
link I gave above, there's a comment that reads:
/*
* From AppleFSCompression documentation:
* "It is incumbent on the aware copy engine to identify
* the type of compression being used, and to perform an
* unaware copy of any file it does not recognize."
*
* Compression Types are defined in:
* "AppleFSCompression/Common/compressorCommon.h"
*
* Unfortunately, they don't provide a way to dynamically
* determine what possible compression_type values exist,
* so we have to update this every time a new compression_type
* is added. Types 7->10 were added in 10.10, Types 11 & 12
* were added in 10.11.
*
* Ubiquity faulting file compression type 0x80000001 are
* deprecated as of Yosemite, per rdar://17714998 don't copy the
* decmpfs xattr on these files, zero byte files are safer
* than a fault nobody knows how to handle.
*/
Further, the "Decmpfs" refers to the XNU kernel's decmpfs subsystem for file system compression.
If you do an nm
on /System/Library/Extensions/AppleFSCompressionTypeZLib.kext/MacOS/AppleFSCompressionTypeZlib
, you can see that it has libcompression
built in, and although the symbol isn't there, it also has LZBITMAP
. I forget the order of how it works, but basically the APFS kext and this kext provide transparent filesystem compression via registering decmpfs handlers.
I didn't do any kernel debugging to see if I hit any of this code, but I'm pretty sure that it is LZBITMAP
.
I just finished a library for LZBITMAP decompression if you want to use it: https://github.com/eafer/libzbitmap. I'll be adding support for cryptex to my driver in a few days.
@eafer You are a hero to us all!
I just pushed a version with experimental support for lzbitmap. Sorry didn't notice others have already done the work ...
Hi,
I am trying to use
apfs-fuse
to mount the Cryptex1,SystemOS DMG from the latest iOS 16 IPSWs on Linux. It appears that Apple has started compressing some files with a new type of compression thatapfs-fuse
does not support, called LZBITMAP.To reproduce this issue, download the iPhone 14 Plus 16.0 IPSW from here and extract the file
098-09358-001.dmg
and mount it withapfs-fuse.
, then browse to$MOUNT_POINT/root/System/Library/Caches/com.apple.dyld
. If you do anls -l
, you will see three files with a weird size and timestamp:Further, if you try to read one of these files, you will see the following:
The
apfs-fuse
logs will show something like this.This compression type 14 turns out to be the LZBITMAP type, which is part of
libcompression
on macOS. I added a switch case for 14 to theIsDecompAlgoSupported()
andIsDecompAlgoInRsrc()
functions inDecmpfs.cpp
, andapfs-fuse
reports the correct size, but obviously it still needs the algorithm.I am not sure if
apfs-fuse
can support this algorithm, because it is closed source. I did take a look at thelzbitmap_decode_buffer()
function inlibcompression
on macOS and it's a lot of architecture specific vectorized instructions. I certainly understand if this is not something that can be added, either because of legal concerns with proprietary algorithms, or just lack of time to re-implement something like this. Either way, I just wanted to bring it to your attention.Thanks!