search

ckolivas / lrzip

Long Range Zip
http://lrzip.kolivas.org
GNU General Public License v2.0
619 stars 76 forks source link

Streaming issue "No space left on device" in lrzip 0.651 #233

Open stevenshiau opened 2 years ago

stevenshiau commented 2 years ago

We encountered an issue when using lrzip to compress the stdin data and output to a file. It gave an error "No space left on device". However, we are very sure the disk space is enough. It's on Debian Sid live system. If we switched to lrzip 0.641, this issue was not reproduced. More information is provided in the following. If you need more info, please let me know. Thanks.

root@debian:~# dpkg -l lrzip Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-====================================================== ii lrzip 0.651-2 amd64 compression program with a very high compression ratio

root@debian:~# dd if=/dev/nvme0n1p3 | lrzip -v - > /home/partimag/nvme0n1p3.dd.lrz The following options are in effect for this COMPRESSION. Threading is ENABLED. Number of CPUs detected: 4 Detected 4119420928 bytes ram Compression level 7 Nice Value: 19 Show Progress Verbose Temporary Directory set as: ./ Compression mode is: LZMA. LZ4 Compressibility testing enabled Heuristically Computed Compression Window: 6 = 600MB Outputting to stdout. Minimising number of threads to 3 to limit memory usage Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Unable to compress entirely in ram, will use physical files Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Write of length 25336877 failed - No space left on device Failed to write_buf s_buf in compthread 0 No space left on device Fatal error - exiting

After downgraded to 0.641, no such issue. root@debian:~# dpkg -l lrzip Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-===============-============-====================================================== ii lrzip 0.641-1+deb11u1 amd64 compression program with a very high compression ratio

root@debian:~# dd if=/dev/nvme0n1p3 | lrzip -v - > /home/partimag/nvme0n1p3.dd.lrz The following options are in effect for this COMPRESSION. Threading is ENABLED. Number of CPUs detected: 4 Detected 4119420928 bytes ram Compression level 7 Nice Value: 19 Show Progress Verbose Temporary Directory set as: ./ Compression mode is: LZMA. LZ4 Compressibility testing enabled Heuristically Computed Compression Window: 6 = 600MB Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase Beginning rzip pre-processing phase 124370944+0 records in 124370944+0 records out 63677923328 bytes (64 GB, 59 GiB) copied, 3372.52 s, 18.9 MB/s Compression Ratio: 0.000. Average Compression Speed: 17.930MB/s. Total time: 00:56:26.90

root@debian:\~# free -h total used free shared buff/cache available Mem: 3.8Gi 242Mi 2.8Gi 287Mi 788Mi 3.1Gi Swap: 0B 0B 0B

root@debian:~# df -h Filesystem Size Used Avail Use% Mounted on udev 1.9G 0 1.9G 0% /dev tmpfs 393M 2.1M 391M 1% /run /dev/sr0 365M 365M 0 100% /run/live/medium /dev/loop0 305M 305M 0 100% /run/live/rootfs/filesystem.squashfs tmpfs 2.0G 286M 1.7G 15% /run/live/overlay overlay 2.0G 286M 1.7G 15% / tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 5.0M 8.0K 5.0M 1% /run/lock tmpfs 2.0G 48K 2.0G 1% /tmp tmpfs 393M 0 393M 0% /run/user/1000 /dev/nvme0n3p1 1000G 66G 935G 7% /home/partimag

pete4abw commented 2 years ago

This commit may be the culprit.