Compression and APFS

[szhorvat]

Will this tool work on High Sierra with APFS volumes?

Are there any known problems with converting a HFS+ volume containing compressed files (compressed using afsctool) tp APFS? Is it safe to upgrade to High Sierra if we have HFS+ compressed file?

[RJVB]

On Tuesday September 26 2017 08:05:20 Szabolcs Horvát wrote:

Will this tool work on High Sierra with APFS volumes?

If it works with APFS it ought to work wherever you take your computer, even high up in the Himalaya ;) Seriously, check the other issues and pull requests as well as the commit history. IIRC someone tested it with APFS and got it to work - it was only a question of detecting APFS as a compatible filesystem. I cannot recall having made any changes to the code that does the actual compression and rewriting.

Are there any known problems with converting a HFS+ volume containing compressed files (compressed using afsctool) tp APFS? Is it safe to upgrade to High Sierra if we have HFS+ compressed file?

All I can say is that the original author seems to have done his homework; files compressed with afsctool behave exactly as files compressed by Apple ... as far as I've been able to tell in the few years that I'm using this now. As to the migration to APFS: I think it should not be significantly less safe with than without compression. Apple use compression too (and well they should!). Try first on a thumb drive, and whatever you do, make a full backup first (in addition to any TimeMachine backups you may have).

The best way I found to upgrade between major Mac OS upgrades is this:

• clone your current (internal) boot volume to an external (using Carbon Copy Cloner)
• boot once from the external to make sure it works, boot back into your internal disk (this also gives you a clean session)
• run the upgrade application and point it to the external
• test-drive the result from the external

When pleased, everything works etc etc, and while booted from the external:

• format your internal (just the partition holding the OS if you have more than 1 partition)
• clone back the external onto your internal disk
• don't forget to clone the recovery partition too (there are other tools for that, though).

[szhorvat]

Reporting back after High Sierra upgrade:

afsctool appears to work fine with APFS

Compressed files were maintained as such during HFS+ -> APFS conversion.

[RJVB]

Great to know, thanks!

[DanielSmedegaardBuus]

Wait, what? I was under the impression that compression hasn't yet been implemented in APFS? Are you saying that you can use afcstool on APFS and it actually compresses files? As in, it doesn't just say it does, it actually does?

[szhorvat]

The Finder reports smaller sizes. Also, compression is preserved during conversion to APFS>

But there are issues. I have some apps compressed and when running them from a terminal, I get messages like this (but everything appears to work fine):

/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Common/ChunkCompression.cpp:50: Error: unsupported compressor 8
/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Libraries/CompressData/CompressData.c:353: Error: Unknown compression scheme encountered for file '/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist'
/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Common/ChunkCompression.cpp:50: Error: unsupported compressor 8
/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.30.2/Libraries/CompressData/CompressData.c:353: Error: Unknown compression scheme encountered for file '/System/Library/CoreServices/CoreTypes.bundle/Contents/Library/AppExceptions.bundle/Exceptions.plist'

@RJVB I haven't forgotten about this but didn't yet find the time to look at it.

[RJVB]

I've seen some mention of this on another issue, but understood it as an indication that APFS supports additional compression types.

So do you get systematic warnings about the compression type after compressing with afsctool, but you can still read the files without problem? What happens when you transfer compressed files preserving their compression, or when you convert a HFS volume with compressed files? Is there a difference when the files concerned are compressed by afsctool or came from Apple (for instance, the iTunes installer creates a compressed app bundle, AFAIK)?

I don't have much time to work on this right now (nor a system that supports APFS) but I can try to figure out if there's a way to print the compression types, and if there are symbolic representations (constants) somewhere in the header files.

[szhorvat]

@RJVB Don't bother with this now. I still have to investigate when the warnings appear, I just haven't had the time yet. I mentioned it in response to Daniel's comment.

I mainly use afsctool to compress Mathematica. It's 6-8 GB and I need several versions installed. When I start the GUI part of Mathematica from a terminal, I sometimes see these messages. When I start its "kernel" part (the command line program that actually does computations) I never see them. It might have something to do with Mathematica specifically.

I use the program daily and I have not yet noticed any tangible problems.

I'll update you in a couple of weeks.

[RJVB]

Okidoki, but getting a handle on valid and unsupported compression types could be useful anyway.

I was going to suggest you figure out what files cause the errors, but from the looks of it the ones you copied here concern a system file (Exceptions.plist). It would be very weird if that error goes away when you run an uncompressed copy of Mathematica (or the message reports the wrong file).

That file is compressed for me, but contains entries like AppNapOverrides for applications that I doubt were prerecorded there by Apple (Adium, but also Handbrake...). Maybe Mathematica causes an exception to be recorded in there which somehow leads to the error, independently of the fact you used afsctool?

[RJVB]

I'll update you in a couple of weeks.

I just pushed a minor upgrade which makes the code a bit more legible (cf. comparable to libarchive, by borrowing some of their constants). Doing

> afsctool -v /path/to/file

will now also print out the compression type (and claimed uncompressed file size) if those data are available from the compression header.

For the file giving you errors:

> afsctool -v /System/Library/CoreServices/CoreTypes.bundle/Contents/Library/AppExceptions.bundle/Exceptions.plist
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library/AppExceptions.bundle/Exceptions.plist:
File is HFS+/APFS compressed.
Compression type: decmpfs xattr (3)
File content type: com.apple.property-list
File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): 5285 bytes / 5 KB (kilobytes, base-10)
File size (compressed data fork): 3068 bytes / 3 KiB
Compression savings: 41.9%
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Approximate overhead of extended attributes: 268 bytes
Approximate total file size (compressed data fork + EA + EA overhead + file overhead): 3584 bytes / 4 KiB
Uncompressed file size reported in compressed header: 5285 bytes

The take-home message here is that afsctool (and libarchive) only implement 2 different compression types. Type 3 is for small files that compress to less than 3082 bytes and take up only a single disk block: here the compressed data is stored directly in the extended attribute (xattr). All other files have type 4, which means the compressed data is stored in the resource fork.

Afsctool does a lot of checking, and should not leave "compressed" files around that end up having another compression type than those 2.

[Henrietta1989]

Anybody tested in Mojave?

[maximumsomething]

@Henrietta1989 it works just as expected in Mojave.

[lpar]

Confirmed, seems to work on test files. Go binaries show up as already compressed, which I'm not entirely convinced by, but maybe the linker is that smart?

[RJVB]

Go binaries show up as already compressed, which I'm not entirely convinced by, but maybe the linker is that smart?

What kind of binaries, created how?

The (go) linker could indeed apply compression by default (under certain conditions).

[lpar]

A Mac executable built using go build. Looks like a failure to compress is being reported incorrectly:

meta@updog % afsctool -v mage 
/Users/meta/Programming/Go/bin/mage:
File is not HFS+/APFS compressed.
File content type: public.unix-executable
File data fork size (reported size by Mac OS X Finder): 4298904 bytes / 4.3 MB (megabytes, base-10)
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 4300800 bytes / 4.1 MiB
meta@updog % afsctool -c mage 
Unable to compress file (already compressed).
meta@updog % 

[RJVB]

A Mac executable built using go build. Looks like a failure to compress is being reported incorrectly:

Can you point me to an example of such a file, or attach one here?

[lpar]

Well, I had to zip it, but here you are. mage.zip

[gingerbeardman]

This compresses for me, with LZVN and the latest afsctool (it seems you're using that)

$ afsctool -cvvv -T LZVN mage
/Users/matt/Downloads/2019-06-13/mage:
Compression type: LZVN in resource fork (8)
File content type: public.unix-executable
File resource fork size: 2346302 bytes
File DECMPFS attribute size: 16 bytes
File size (uncompressed; reported size by Mac OS 10.6+ Finder): 4298904 bytes / 4.3 MB (megabytes, base-10)
File size (compressed): 2347008 bytes / 2.2 MiB
Compression savings: 45.4%
Number of extended attributes: 1
Total size of extended attribute data: 19 bytes
Uncompressed file size reported in compressed header: 4298904 bytes

With ZLIB I get this:

$ afsctool -cvvv -T ZLIB mage
/Users/matt/Downloads/2019-06-13/mage: file has a compressed chunk that's larger than the original chunk; -L to compress
/Users/matt/Downloads/2019-06-13/mage:
Unable to compress file.
File content type: public.unix-executable
File data fork size (reported size by Mac OS X Finder): 4298904 bytes / 4.3 MB (megabytes, base-10)
Number of extended attributes: 1
Total size of extended attribute data: 19 bytes
Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 4300819 bytes / 4.1 MiB

I'm on 10.13.6 with an APFS filesystem.

How about you @lpar ?

[lpar]

I unzipped the zip file and tried compressing the unpacked copy and it still wouldn't compress:

% afsctool -c mage
Unable to compress file (already compressed).
% afsctool --help
afsctool 1.6.9. (1.6.8.6-132-gc437635)

OS is 10.14.5 with an APFS filesystem.

Trying to use LZVN I get Sorry, LZVN compression has not been enabled in this build. I just used the README instructions to build it.

[lpar]

With ZLIB, same results as you, so maybe the root problem is that the build doesn't have LZVN enabled.

[RJVB]

No, you're both right. The crux is in Matt's earlier report:

$ afsctool -cvvv -T ZLIB mage
/Users/matt/Downloads/2019-06-13/mage: file has a compressed chunk that's larger than the original chunk; -L to compress
/Users/matt/Downloads/2019-06-13/mage:
Unable to compress file.

I didn't add such a test to the LZVN compression algorithm (simply because I don't know if it's needed), so the file will indeed compress fine with LZVN. BTW: HFS compression works on a chunk basis, which keeps memory requirements for things like searching low. The default ZLIB compressor will refuse to compress an entire file if one of those chunks becomes larger after compression. There is no indication anywhere that this has to be the case but I'm keeping it as default for backwards compatibility.

If you remove the verbose options from the command above you will get the "already compressed" error message. Which is in error.

I'll be pushing a fix.

[gingerbeardman]

Thanks @RJVB

@lpar let us know if you have a problem building with LZVN support which requires cmake+make+install of https://github.com/RJVB/LZVN before cmake+make+install of this afsctool.

[lpar]

Everything built and installed with no problems, thanks.

[ylluminate]

Do we know yet if compression is readily available with APFS on macOS 14 Sonoma yet?

[RJVB]

I have no idea but why would they disable compression in a specific OS version?

What they could do (and probably will, one day) is make it verboten to compress your own files unless the application that creates them offers it as a possibility...

[ylluminate]

With the way things are going generally I have a lot of apprehensions "upgrading" anymore.