RJVB / afsctool

This is a version of "brkirch"'s afsctool utility that allows end-users to leverage HFS+ compression.
https://brkirch.wordpress.com/afsctool
GNU General Public License v3.0
187 stars 18 forks source link

-T LZVN doesn't work #29

Open lucianmarin opened 5 years ago

lucianmarin commented 5 years ago
afsctool -T LZVN -c /path/to/random/file
/path/to/random/file: unsupported compression type 2 bytes
Unable to compress file (already compressed).

I think LZVN support is enabled (cmake LZVN repo then cmake afsctool), but it doesn't seem to work on High Sierra.

RJVB commented 5 years ago

afsctool -T LZVN -c /path/to/random/file

Two things seem to be going on here:

/path/to/random/file: unsupported compression type 2 bytes

This message indicates that LZVN support has in fact not been enabled.

Unable to compress file (already compressed).

And this suggests that your file was already compressed. But it can be an artefact because of LZVN support not having been enabled. What does /usr/bin/ls -lO /path/to/random/file print?

I think LZVN support is enabled (cmake LZVN repo then cmake afsctool), but it doesn't seem to work on High Sierra.

Did you install LZVN? Just building it isn't enough, the headers and library must be available in a location where cmake can find them. Cmake should report whether LZVN support is being enabled.

I've made a few small changes to afsctool so that it gives clearer error messages when LZVN support has not been enabled. Please pull afsctool, rebuild and try to compress your random file again. Then, install LZVN and re-cmake and re-build afsctool .

lucianmarin commented 5 years ago

make install LZVN

Install the project...
/usr/local/Cellar/cmake/3.13.2/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Up-to-date: /usr/local/lib/libFastCompression.1.0.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.1.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: /usr/local/lib/libFastCompression.a(liblzvn.c.o) has no symbols
-- Up-to-date: /usr/local/bin/lzvn
-- Up-to-date: /usr/local/include/lzvn.h
-- Up-to-date: /usr/local/include/FastCompression.h

cmake -Wno-dev .. afsctool (after last git pull)

-- LZVN_HEADER,LIB: /usr/local/include/FastCompression.h /usr/local/lib/libFastCompression.dylib
-- Enabling LZVN support
-- Setting afsctool version to: 1.6.9. (1.6.8.6-127-g731aeaa)

Later edit: seems to work when I do cmake .. instead of cmake -Wno-dev .. and make install instead of make install/fast for both projects.

afsctool -v afsctool
File is HFS+/APFS compressed.
Compression type: LZVN in resource fork (8)
Compression savings: 71.0%
lucianmarin commented 5 years ago

Can you make LZVN the default compression format if support is enabled?

afsctool -c afsctool
afsctool -v afsctool
File is HFS+/APFS compressed.
Compression type: ZLIB in resource fork (4)
Compression savings: 73.6%

I also get lzvn compression failed on chunk #0 on some files if they were previously ZLIB compressed then decompressed then compressed with LZVN. ZLIB works on those files, but LZVN doesn't. They are too tiny for LZVN compression. lzvn infile outfile produces a 0 bytes outfile.

RJVB commented 5 years ago

I could but I won’t.

gingerbeardman commented 5 years ago

LZVN works for me on High Sierra, so you'll need to check that you are compiling afsctool correctly.

I use:

afsctool -c -T LZVN "$@"

I'm using afsctool 1.6.9. (1.6.8.6-125-ge269dbf) as I've not built a new version in a while.

RJVB commented 5 years ago
I also get lzvn compression failed on chunk #0 on some files if they were previously ZLIB compressed then decompressed then compressed with LZVN. ZLIB works on those files, but LZVN doesn't. They are too tiny for LZVN compression. lzvn infile outfile produces a 0 bytes outfile.

I didn't notice this part. I'd need to have one of those files to do some testing, but if lzvn fails itself there probably isn't much I can do about that.

I don't really get why you'd want to recompress files with LZVN though. In every test I did it compresses less well than (higher levels of) ZLIB. It may decompress faster, but do you really notice that IRL? OS X has a rather agressive file cache, and to the best of my knowledge this comes after the file system (i.e. it caches data after "HFS decompression").

gingerbeardman commented 5 years ago

FWIW I use LZVN because macOS App Store uses LZVN, and I mainly compress apps so I want to keep things the same as those apps.

RJVB commented 5 years ago

Whatever floats your boat :)

I tend to recompress apps to make them smaller, not to do things the way Apple do them. Apple also makes money from selling disk space so they have a vested interest in not minimising the space occupied by the software they sell ;)

gingerbeardman commented 5 years ago

I read that here the other day, and I countered by saying that Apple ship their OS and apps from the Mac App Store with HFS+ compression turned on, so I don't really see the reasoning?

My reason is not to "do it like Apple" but so that my scripts and workflow do not undo what Apple have done and then redo with the same. Also, LZVN is more efficient in terms of speed and CPU usage, so I don't notice when it's working in the background (compared to ZLIB that makes my fans go crazy). But, yes, whatever works for each of us.

Also, try FSNotes for an app that gives this:

/Applications/FSNotes.app/Contents/Resources/de.lproj/InfoPlist.strings: lzvn compression failed on chunk #0
lucianmarin commented 5 years ago

Also, try FSNotes for an app that gives this:

/Applications/FSNotes.app/Contents/Resources/de.lproj/InfoPlist.strings: lzvn compression failed on chunk #0

I can confirm this on multiple apps. LZVN utility produces 0 bytes output on the same files.

RJVB commented 5 years ago

Apple ship their OS and apps from the Mac App Store with HFS+ compression turned on

But not with the most efficient kind of compression.

lzvn compression failed on chunk #0

I can reproduce that. It's not overly surprising given that the input file is only 2 bytes large... It looks like there is an undocumented 8 byte lower limit to LZVN compression, under which it simply fails.

I pushed updates to lzvn and afsctool in order to ignore files < 8 bytes (and to improve verbose output a bit, the error I mentioned elsewhere that I wanted to fix).

gingerbeardman commented 5 years ago

Thanks René!

gingerbeardman commented 5 years ago

define for LZVN_MINIMUM_COMPRESSABLE_SIZE is missing in latest commit make sure you pull, build and install the latest LZVN

malthe commented 3 years ago

Now we just need ARM-support for LZVN :-)

RJVB commented 3 years ago

Now we just need ARM-support for LZVN :-)

In the library I presume, or is there (also) an endianness issue in my own code?

malthe commented 3 years ago

In the library yeah – there's lots of assembly code in there.

RJVB commented 3 years ago

I'm going to compare that implementation with the generic-C implementation in liblzfse. Turns out there's one in there... If performance is close enough to the one from the assembly code I'm just going to use a single dependency.

RJVB commented 3 years ago

@malthe - Can you check the lzfse branch please (be sure to initialise the submodule!)? A priori it should provide LZVN and LZFSE support on both Apple's architectures now.

malthe commented 3 years ago

@RJVB works perfectly.

RJVB commented 3 years ago

@RJVB works perfectly.

That was fast! :) Both LZVN and LZFSE compression?

malthe commented 3 years ago

It takes quite a while, but this is an 8GB run:

Number of HFS+/APFS compressed files: 140960
Total number of files: 145251
Total number of folders: 20516
Total number of items (number of files + number of folders): 165767
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 7840604172 bytes / 8.2 GB (gigabytes, base-10)
Folder size (compressed): 3696919434 bytes / 3.45 GiB
Compression savings: 52.8% over 140960 of 145251 files
Approximate total folder size (files + file overhead + folder overhead): 3704770889 bytes / 3.45 GiB

(Both LZVN and LZFSE tested.)

RJVB commented 3 years ago

You could have tested with a single file ;)

For reference, LZVN compression of a 128Mb file takes about 3.5s on my 2nd gen. i7 using the lzfse library. That's maybe 10% slower than with the assembly implementation. That surprises me a little bit, the C implementation almost looks like assembly, with lots of cases and gotos so it wouldn't surprise me if the assembly implementation actually came from an older version of that same code. We'll see if I get flamed for a performance regression from users on Intel hardware!

gingerbeardman commented 3 years ago

@malthe off-topic, slightly, but regarding your (old) blog post: https://maltheborch.com/#using-built-in-transparent-compression-on-macos ditto is in-built tool that can do this (HFS+ in-place compression) to a degree, and Apple do it for all system files in an install and all apps installed/updated using the Mac App Store. Just an FYI

malthe commented 3 years ago

@gingerbeardman thanks – I was planning to update that post once this is all merged in. Will try and see if I can come up with a command-line for ditto.

gingerbeardman commented 3 years ago

@malthe from man it's ditto --hfsCompression src dest

malthe commented 3 years ago

@gingerbeardman doesn't work if src is the same as dest though:

<src> and <dest> are identical (not copied).
gingerbeardman commented 3 years ago

ah yes, you have to do a dance through a temporary file

RJVB commented 3 years ago

FWIW, it's always been a pet theory of mine that Apple don't provide a tool for in-place compression (and even less so, transparent automatic FS-level compression) because they also sell storage space.

You may remember that it was long possible (and somewhat common usage) to clean off all the useless translation data (which often comes with copies of image resources) and indeed even the OS X installer allowed you to select to install only the languages you had use for (which would then be "completed" during subsequent updates which didn't have that option). App bundle signing has made that impossible.

gingerbeardman commented 3 years ago

I've heard that before, but my counter is that they apply this to the system and Mac App Store so they're doing more than nothing.

I do think they should enable it by default but there are so many issues that would raise.

Apple storage is pretty reasonable price these days.

gingerbeardman commented 3 years ago

Oh, and I use CleanMyMac to trim language files which still works well.

Screen shot 2021-01-21 at 11 21 27
RJVB commented 3 years ago

My counter would be that they needed compression to bring down the OS and size of most apps to something reasonable enough to be able to sell their entry-level computers with fixed storage back when those barely had enough space. Once you start deploying compression it probably becomes cheaper just to apply it unconditionally to everything.

Apple storage is pretty reasonable price these days.

Yeah right, as in just buy a new computer because you can't increase your internal storage? ;)

I presume "CleanMyMac" only removes language resources that are not inside app bundles, and possibly from apps that aren't signed (or are known not to be crippled when they lose their signing)?

gingerbeardman commented 3 years ago

I presume "CleanMyMac" only removes language resources that are not inside app bundles, and possibly from apps that aren't signed (or are known not to be crippled when they lose their signing)?

I'd say: you presume too much :)

Well, I just installed OmniGraffle from the Mac App Store and it stripped the localisation out of that with no effect on the running of the app.

Screen shot 2021-01-21 at 12 47 45

before

Screen shot 2021-01-21 at 12 48 06

after

Screen shot 2021-01-21 at 12 48 38
RJVB commented 3 years ago

Good to know but I wonder how they pull it off. Maybe Apple changed something in the way apps are signed, taking into account the fact that removing a translation resource cannot possibly be the cause of a security issue?

Dr-Emann commented 1 year ago

BTW, According to libfsapfs (scroll past the table):

If the LZVN compressed data starts with 0x06 (end of stream oppcode) the data is stored uncompressed after the first compressed data byte.

Similar to how an 0xFF prefix can be used for zlib compression to store uncompressed data with only 1 byte overhead.

I think this could be used to handle cases where the last block is too small for LZVN (or if LZVN compression would ever increase the size of the block)

RJVB commented 1 year ago

I think this could be used to handle cases where the last block is too small for LZVN

If memory serves me well that case is already handled (and not by rejecting the entire file).

(or if LZVN compression would ever increase the size of the block)

Both options would need to be tested to see if the kernel supports them!

Dr-Emann commented 1 year ago

It looks like it's currently handled by rejecting the whole file:

https://github.com/RJVB/afsctool/blob/cadc2a3e1500a133d217aeb5a1bf14ddde6a1e94/src/afsctool.c#L239-L246

$ truncate -s $((0x10000 + 7)) /tmp/bad_block

$ ./afsctool -c -vvvv -T LZVN /tmp/bad_block
"/tmp/bad_block": file too small or will contain a too small compression chunk (try ZLIB compression)
"/tmp/bad_block": file too small or will contain a too small compression chunk (try ZLIB compression)
/tmp/bad_block:
Unable to compress file.
File content type: public.data
File data fork size (reported size by Mac OS X Finder): 65543 bytes / 70 KB (kilobytes, base-10)
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Approximate total file size (data fork + resource fork + EA + EA overhead + file overhead): 69632 bytes / 68 KiB
RJVB commented 1 year ago

It looks like it's currently handled by rejecting the whole file:

Right, I remember now being p*d to have to reject a file for a single chunk. Would you like to have a go at implementing your idea?

RJVB commented 1 year ago

I have a hunch though that the original author might have thought of (and tested) the ZLIB equivalent for increasing chunks - instead he implemented an option to allow compressing them.