Kernel panic on huge files?

[biziclop]

Just for reference, if it happens to other people:

I ran afsctool on directories having some few GB files using the following command (if I understand, it works in single-threaded mode):
afsctool -cvvv -9 -L …/folders/at/wherever

When processing the huge files, memory compression and swapfile usage went up quite extremely, but of course it's expected. The unexpected thing is the kernel panic. It happened twice so far. I have no clue if the error happened in 's memory compression or the file compression part.

Panic happens here (just for fun):
https://opensource.apple.com/source/xnu/xnu-4570.71.2/osfmk/vm/vm_fault.c.auto.html

if ((map == kernel_map) && (caller_prot & VM_PROT_WRITE)) {
    if (compressor_map) {
        if ((vaddr >= vm_map_min(compressor_map)) && (vaddr < vm_map_max(compressor_map))) {
            panic("Write fault on compressor map, va: %p type: %u bounds: %p->%p", (void *) vaddr, caller_prot, (void *) vm_map_min(compressor_map), (void *) vm_map_max(compressor_map));

Shortened panic report:

Panic Report panic(cpu 0 caller 0xffffff80274ed17b): "Write fault on compressor map, va: 0xffffff9128ce8000 type: 3 bounds: 0xffffff821247d000->0xffffff92126bd000"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.71.45/osfmk/vm/vm_fault.c:3523 Backtrace (CPU 0), Frame : Return Address 0xffffff803aa03170 : 0xffffff802746bf96 … 0xffffff803aa03fa0 : 0xffffff802741dec2 Kernel Extensions in backtrace: com.apple.filesystems.hfs.kext(407.50.6)[…]@0xffffff7fa9b3a000->0xffffff7fa9ba0fff dependency: com.apple.filesystems.hfs.encodings.kext(1)[…]@0xffffff7fa9b32000 com.apple.AppleFSCompression.AppleFSCompressionTypeZlib(1.0)[…]@0xffffff7faa77c000->0xffffff7faa788fff

BSD process name corresponding to current thread: afsctool

Mac OS version: 17G7024 /// == High Sierra 10.13.6 Kernel version: Darwin Kernel Version 17.7.0: Wed Apr 24 21:17:24 PDT 2019; root:xnu-4570.71.45~1/RELEASE_X86_64

Model: Macmini6,2, BootROM 278.0.0.0.0, 4 processors, Intel Core i7, 2.3 GHz, 16 GB, SMC 2.8f1

[gingerbeardman]

this seems to be virtual memory. is your hard drive definitely ok? could it be failing?

how much VM was being used?

[RJVB]

how much VM was being used?

And (more) importantly, how much free space do you have on the drive? If you're short on space it can be a good idea to deactivate the verify/backup feature (-n option); this will probably reduce memory requirements too.

[biziclop]

I don't remember where did I experiment first time, but this time GSmartControl warned about soft read errors, so I'll have to experiment more with another disk. Both the external drive hosting the large file and the startup disk have plenty of space (30+GB).

Anyway, I ran an experiment with just one file (a copy of an OSX installer image around 6-7GB). It took quite long, it ate all RAM, etc., but it apparently compressed the file, released all the memory. But at the verification phase (I think) it kernelpanicked again. I compared the compressed copy with the original with another tool, and … panic again. Copied the file to another drive, tried to open it, zip it, but they complained about "invalid argument" or such. afsctool -d image.dmg said that the (otherwise almost incompressible) file occupies 4GB on disk. Finder's Get Info said 6GB (compressed) 2GB on disk.

So it's funny. But as I said I have to test this more. Anyway, thanks for replying!

[RJVB]

I thought I disabled the support for files > 2Gb?!

Such files could be stored in a compressed disk image, which also means the much more capable bzip2 compressor can be used.

[biziclop]

I actually made a service to Finder to (re)compress disk images, but this time I just sent afsctool to every old & forgotten file in a folder, because why not. This sh*t is somehow addictive :)

[RJVB]

Can you please check with the latest commit?

[biziclop]

Can you please check with the latest commit?

So far it works and when file is too big, it properly emits with unsupportable size.
Thanks! :)

BTW, maybe this useful tool could be promoted among developers who offer .dmg images with "drag this to the Applications folder to install". Gimp, LibreOffice, VirtualBox are compressed to less than the half of their original sizes. My top (by size) 3rd party apps are 29GB uncompressed, but became 14GB after compression. On small  SSDs, this can be quite a huge saving.

[gingerbeardman]

BTW, maybe this useful tool could be promoted among developers who offer .dmg images with "drag this to the Applications folder to install".

I agree, perhaps @biziclop can ask the most popular DMG creation apps to add support?

• DropDMG
• Create DMG
• etc

[RJVB]

I'm inclined to leave that to users of said tools, which I'm not. After all, why would the devs add support for something for which there's no user demand?

[gingerbeardman]

Yes, I meant @biziclop who raised the point.

I also think that Sparkle Updater framework could be educated in the same way about HFS+ compression.