0 byte files after running afsctool (1.7/1.6.9 with LZVN & zlib support)

[subzerofun]

Hello, first – thanks for the updated tool, was always hoping someone could improve the archived version!

System Info: macOS 10.12.6 with both HFS and APFS disks. Boot volume is of course HFS. Two internal SSDs with Mojave in APFS. cmake: stable 3.11.4 GNU Make 3.81 zlib: stable 1.2.11 (PKG_CONFIG_PATH: /usr/local/opt/zlib/lib/pkgconfig) LZVN from here: https://github.com/RJVB/LZVN XCode 9.2 and Command Line Tools for 9.2

I've compiled the latest version (1.7/1.6.9) with ZLIB and LZVN support and have run several tests with different settings, but always encounter the same problem: When i try to compress an uncompressed (or previously compressed [w afsctool from brew]) folder a lot of files end up showing only 0 bytes in Finder (and Terminal). If if try to open that "empty" file the content is not accessible – the default App/software for that file extension throws an error message. It's also impossible to decompress the 0 byte files – after decompression they remain at 0 bytes. This occurs on both HFS and APFS volumes, i've tested it with several different folder and subfolder structures containing various file types. It is not restricted to certain file types or files that have anything in common. I just wanted to pin the problem down to certain file names / too many subfolders – but i could not find a pattern. The only thing that is constant: It always happens to the same files within that folder – regardless of the file name, extension or my input options.

Have also tried older versions (from the "Tags" menu), but couldn't find one that does not cause this issue. Tried to find an error while building from source, but everything looks fine – the source files seem to be OK.

I've tried running it with the ZLIB and LZVN option, with just one or 4-8 worker(s) and several other different input options, but the result is always the same. When i try to compress a large folder with several subfolders a lot of files have 0 bytes afterwards and end up being corrupted.

For now i've reverted to the brew build (afsctool 1.6.4 build 34) and have tested compressing the same folders with different options and don't run into this issue. Would really love to use your updated version though – without destroying the content of my files :-)

Are there any arguments i should add that could prevent this? Have other users also experienced this issue?

[RJVB]

I've heard about this issue but only really in a recent report here where in the end the problem was not caused by my afsctool version.

From what you're reporting here it would seem that there is indeed an issue. Do you get it with both LZVN and (all levels of) ZIP compression? Does it also affect files that were already compressed (it shouldn't so if it does there's another bug)?

You are not getting any messages about failed operations, not even with -vvv, do you? Since the original afsctool (built from source I presume?) is unaffected the only explanation I can see is that my code somehow fails to set the compression attributes. If that happens you indeed end up with an empty file, exactly as when you mount a disk with compressed files on a Mac OS X < 10.6 version.

Can you please create a dmg from a folder that contains a few files that compress OK and a few that lead to empty files, taking care to preserve all attributes these files may have? The surest way would be to create a writable dmg, mount it, and then copy the sample folder with ditto --rsrc $folder /Volumes/afsctoolSample/, unmount the dmg and upload it here.

That way I can try to reproduce your issue and hopefully understand what happens. The other thing you could do is to checkout the very first commit from my repo which should be the original v34 version with just a new build system and see if it's issue free. If so you can work your way back up the commit log, a priori with the help of git bisect.

[subzerofun]

Thanks for the quick response! I've created four test folders and copied them via ditto to an dmg. You can download it here: https://we.tl/t-YZQlmMfcGr

I get the empty files with both the LZVN and ZIP option – the compression level does not seem to change the outcome (tried 5 & 9).

In the PNG and JPEG folders (ZLIB & LZVN) there are a few 0 byte files (Samsung*.png). When using the brew afsctool these files compress without losing their content (folder test-afsctool164).

I've also copied all terminal outputs from building the LZVN libraries and afsctool files as well as the output (with 'vvv') of afsctool as txt files (plus some screenshots). Hope you can solve this issue – i have no idea what goes wrong during compression.

Have compiled the first version (v34) and when i try to compress the test folder i get the error "No compressable files in folder"... Strange. I've doublechecked the files, they are all uncompressed. File permissons are OK. Tried running it with "sudo", but get the same error. Will go up the ladder and see if later versions work.

[RJVB]

Thanks! I'll have a look over the weekend.

Have compiled the first version (v34) and when i try to compress the test folder i get the error "No compressable files in folder"... Strange.

Without having looked at the folder contents I can only say that this may not be strange; afsctool never tried to compress files that can reasonably be expected not to compress, by looking at file types. Png files will most likely be skipped, for instance. So "not compressable" is really to be taken in the broad sense of "I (probably) cannot make this any smaller".

PS: yikes, yes, there's a problem with compressing png files! Judging from an older MacPorts version I still had installed commit #28ec92ecb65660da0d591e036c44592db9921e23 should still be safe.

[subzerofun]

With v34 i get this error message for every folder i select – i wonder if something simply did go wrong during the build process. But it doesn't matter – i am right now at commit 331893d1 (oct 2018) and both ZIP and LZVN compression work without producing empty files! Will try out the next few versions to narrow down the breaking change.

This issue is not only limited to png and jpg files, it also happened with a >20 MB PSD file. And i was so smart to compress two crucial folders on my system: /usr/local/Cellar (all the homebrew files) and the "site-packages" folders of my python2.7 and 3.6 installs. And after that i had to clean up hundreds of 0 byte .py, .rb and other source files... At least now i have some commands ready to reinstall all my brew and pip packages for the next catastrophe :-).

[RJVB]

Will try out the next few versions to narrow down the breaking change.

Thanks! Remember git bisect. Never used it myself but it's designed to pinpoint the commit that introduced an error. Fortunately afsctool is small and should build quickly (invoking make in the build dir. after a checkout should be enough).

This issue is not only limited to png and jpg files, it also happened with a >20 MB PSD file.

PSD is PhotoShop? My hunch would be that it happens to any file judge to be incompressable, I must have overlooked an error path (or introduced one) that leaves the files empty.

And i was so smart to compress two crucial folders on my system: /usr/local/Cellar (all the homebrew files) and the "site-packages" folders of my

Thing is, I may have done similar things without yet having discovered it. I've been known to run afsctool on /Applications ... So thanks for catching this!! The hunt will be on first thing after I finish breakfast :)

[gingerbeardman]

Are the files really zero? What happens if you decompress them? What happens if you load/run them?

The only reason I ask is that in Clusters case the data was still there?

[RJVB]

No, they appear to be really empty, and lack the special xattrs too (so the data isn't even in the resource fork either).

I'm really confused that I could have missed this :-/

[gingerbeardman]

I've not seen it, or found any compressed apps that won't work because of it. Strange.

[RJVB]

I think I have found the issue. While re-organising code recently I moved the file truncation before a certain number of condition tests that assumed the original file was still intact. And the problem files here were probably all examples where compression INCREASED the size, something afsctool refuses to do. Stupidly there was no warning, nor an attempt to restore the file.

I'm finishing up a few changes that should prevent this from happening in the future, I think there were a few other (even more) corner cases where failure could lead to an empty file.

[RJVB]

Let's keep the issue open until we have confirmation this was all...

[gingerbeardman]

What commit did things go bad in? I guess I could be using my own build from before that date?

[gingerbeardman]

On closer inspection a couple of apps I compressed did suffer from this (missing icons, so easy to spot!)

[gingerbeardman]

For those needing to repair/restore from backup, you can find zero-byte files easily within a directory:

find ./ -type f -size 0

[RJVB]

Also, please, if you find such files please check if they are image files or any kind of content that is already compressed. If they are, they probably ended up truncated because of the bug I just fixed (hopefully), because most likely the HFS compression INCREASED their size.

If they are not that kind of file, please verify if they are now compressed OK (on a copy of course!)

Thanks!

[gingerbeardman]

I restored Acorn.app and Transmit.app by downloading fresh apps, and compressing now they are fine (icon does not become 0-bytes)

I only really compress apps, which can be easily recovered or restored, so not much harm was done here for me.

Thanks for fixing quickly.

[RJVB]

And I think the culprit is simply this commit: #bbd7c341 . Before that the file was truncated after writing the extended attributes (= the compressed resource fork and/or the decmpfs xattr).

The goal was to avoid free space fragmentation when working with large files ... that overshot its mark a bit for some users...

[ahelpingchip]

Thank you for this! I've been running into the same issue and was just about to put together a sample test case, and arrived at the issues page to find it already resolved 🎉

I can confirm that in my case (mostly PNGs and assets used in apps), the problematic images are no longer truncated to zero-byte files.

[RJVB]

Thanks for the heads-up!!

[RJVB]

Seems safe to close this now.