search

AgentD / squashfs-tools-ng

A new set of tools and libraries for working with SquashFS images
Other
194 stars 30 forks source link

Too many links error followed by double free in tcache 2 when converting tarball to sqfs #123

Closed kkew3 closed 6 months ago

kkew3 commented 6 months ago

I have a large wikipedia html dump tarball and I want to convert it to squashfs through tar2sqfs. However, it raises the following error:

en/a/r/t/Wikipedia~Articles_for_deletion_Narodnye_Naviny_Vitsebska_88aa.html: Too many links
free(): double free detected in tcache 2
zsh: IOT instruction (core dumped)

The tarball is too large to be uploaded, but the (gzipped) result of tar tf wiki-1.tar | gzip > wiki-1.txt.gz is attached below:

wiki-1.txt.gz

where there are about 98804 files under directory en/a/r/t.

The tar2sqfs --version is:

tar2sqfs (squashfs-tools-ng) 1.1.3
Copyright (c) 2019 David Oberhollenzer et al
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Is this behavior expected? Are there any workaround to make squashfs from such tarball? Thank you so much in advance!

AgentD commented 6 months ago

Hi @kkew3 ,

while processing the tarball, tar2sqfs creates an in-memory representation of the filesystem tree that it serializes, once the end of the tarball is reached. The link count error is produced by that code.

For directories, the link count is basically the number of files plus 2. Versions 1.1.3 that you are using used a 16 bit counter and your en/a/r/t directory easily overflows that.

This (and the double free in the error path) should be fixed in version 1.2.0.

kkew3 commented 6 months ago

I compiled the version 1.2.0 and it worked. Thank yo so much!