Archive created by tar2sqfs takes an estimated 16 hours to open by 7zip.

[Error1000]

So recently i created a squash fs archive of a ~250gb uncompressed tar with: tar2sqfs -b 1M -X extreme, the resulting squashfs archive (~213GB in size) mounts instantly in linux, but on windows using 7 zip it says it would take ~16 hours to open ( i gave up after ~10 minutes ), this isn't just an issue with 7zip being bad at big archives either as another even bigger squashfs archive that i created using the normal squashfs-tools, with mksquashfs -comp xz -b 1M opens in seconds in 7zip, so it must be a quirk of squashfs-tools-ng. NOTE: So far i haven't yet managed to reproduce the bug on a smaller tar-ball.

Versions: 7-zip: 22.01 squashfs-tools-ng: 1.1.4

[AgentD]

Hi @Error1000 ,

So recently i created a squash fs archive of a ~250gb uncompressed tar with: sqfs2tar -b 1M -X extreme, the resulting squashfs archive (~213GB in size)

I take it you meant tar2sqfs to create the SquashFS image? sqfs2tar does the reverse, i.e. turn the SquashFS image into a tarball. Also, are you sure that the 250G tarball is really uncompressed? sqfs2tar in version 1.1.x supports compressed tarballs transparently.

If that is correct, the compression ratio seems suspiciously low, unless the input is already mostly compressed (e.g. typical for media files). In that case, a lot of the data is probably stored uncompressed anyway and should be quick to unpack.

mounts instantly in linux, but on windows using 7 zip it says it would take ~16 hours to open ( i gave up after ~10 minutes ), this isn't just an issue with 7zip being bad at big archives either as another even bigger squashfs archive that i created using the normal squashfs-tools, with "-comp xz -b 1M" opens in seconds in 7zip, so it must be a quirk of squashfs-tools-ng.

When you created the image with squashfs-tools, did you use sqfstar, or run mksquashfs on the unpacked data?

From the top of my head, I can think of two differences between the tools:

• You specified the extreme compression mode for XZ, regular squashfs-tools don't do that. Given the low compression ratio, that shouldn't have too much of an impact, but there could still be some quirk here on the decompression side.
• Regular squashfs-tools by default create an export table (for NSF support). This maps inode numbers to an exact location, allowing O(1) lookup based on the inode number. tar2sqfs does not do that by default (can be enabled using -e). Perhaps 7zip relies on that and does an inefficient tree-walk-per-file as a fallback? Not sure how 7zip comes up with that time estimate tough.

Also, just to make sure there isn't some severe bug lurking in squashfs-tools-ng: when you mount the image from tar2sqfs, are the files are all there? Is their content what it's supposed to be?

[Error1000]

Hi,

I take it you meant tar2sqfs to create the SquashFS image

Yes you are right, i made a typo. ( the original comment is now fixed )

the input is already mostly compressed

Again yes, i know that it doesn't make sense to compress already compressed data, but i had compute and time to spare, and wanted to test out squashfs-tools-ng to see how it would fare against a big dataset.

When you created the image with squashfs-tools, did you use sqfstar

I used mksquashfs.

Also, just to make sure there isn't some severe bug lurking in squashfs-tools-ng: when you mount the image from tar2sqfs, are the files are all there? Is their content what it's supposed to be?

I used sqfs2tar to get back the tar file for testing, the recovered tar file and original tar file had similar sizes and they both seemed to contain all files ( i didn't think to do a thorough check at the time tough ). Since then, due to space constraints on my side i deleted the archive created by squashfs-tools-ng that was "bugged" and the recovered tar, and am now in the process of trying to do: tar2sqfs -b 1M -X extreme -e on the original uncompressed tar, i will leave a comment once the squashfs with the export table is crated, however it may take a couple of days, as i am doing the tests on an older machine i have laying around.

[Error1000]

tar2sqfs -b 1M -X extreme -e generated a squashfs with the same problem. I am now trying to sqfstar -comp xz -b 1M -xatttrs the original tar to see if that helps.

[Error1000]

The archive created by sqfstar -comp xz -b 1M -xatttrs opened instantly.