search

ckolivas / lrzip

Long Range Zip
http://lrzip.kolivas.org
GNU General Public License v2.0
618 stars 76 forks source link

lrzip 0.631 occasionally hangs forever #64

Closed doctaweeks closed 2 years ago

doctaweeks commented 7 years ago

I have not investigated what causes this but occasionally when running lrztar it will hang for unreasonably long period of time until it is killed. In some cases the process was in the background for nearly a month for a 27M directory!

haneefmubarak commented 7 years ago

I looked a touch into it and from what I can tell, lrzip spends a lot of time around munmap() with exceptionally big files, which is confusing, but might be a lead to work from...

LinuxMatt commented 7 years ago

With version 0.631, I confirm that decompression can hang forever as well on huge files (size > 40 Gb uncompressed).

doctaweeks commented 7 years ago

To be clear, in this case I'm not talking about large files or large directories. This is on single directories less than 30M with files of ~23K.

JamesOlvertone commented 7 years ago

Hangs sometime with --gzip Option on compressing a tar. Lesser in newest version, almost always in earlier versions.

v0.631 crashes with --gzip --threads 1 on a 17GB tar, on the fist few GBs.

jay-to-the-dee commented 3 years ago

My lrzip version 0.631 from Ubuntu 20.04 works fine for the compression but nearly always segfaults on multi gigabyte files for decompression :(

-p 1 seems to help significantly though (disabling multi core decompresion). This is on a Haswell generation processor so don't know if Ubuntu compile options could be affecting this scenario sometimes on that ISA? :/

ckolivas commented 2 years ago

Unknown if this is still an issue with a more recent version.

ArrayBolt3 commented 1 year ago

Just reproduced this with lrzip 0.651. Trying to decompress a 28.97 GB file on Kubuntu 22.04. It was compressed with lrzip --zpaq ./file and I'm decompressing it with lrzcat ./file.lrz > newfile. It's hung here:

Decompressing...
 71%      20.75 /     28.97 GB  1:100%  2:100%  3:100%  4:100%  5:100%  6:100%  7:100%  8:100%  9:100%  10:100%

Worthy of note, lrzip is still keeping my CPU busy (currently keeping one core at 100% usage).

Will try a single-core decompress and see if it helps things.

ckolivas commented 1 year ago

I am too low on cycles to work on lrzip for now. Can I suggest you try lrunzip instead of lrzcat as the latter uses STDIO whereas the former uses the core decompression code.