search

benkasminbullock / gzip-faster

Perl module for gzip and gunzip
https://metacpan.org/release/Gzip-Faster
8 stars 2 forks source link

Zlib did not finish processing the string: 6327772 bytes left at .../perl5/aix-thread-multi-64all/Gzip/Faster.pm line 76. #8

Open XSven opened 10 months ago

XSven commented 10 months ago

perl version: 5.14.2 Gzip::Faster version: 0.21 operating system: AIX 7.3.0.0 zlib version: zlib-1.2.13-1.ppc, zlib-devel-1.2.13-1.ppc

perl -MGzip::Faster=gunzip_file -we 'gunzip_file(shift);' file.gz
Zlib did not finish processing the string: 6327772 bytes left at .../perl5/aix-thread-multi-64all/Gzip/Faster.pm line 76.

The zlib status was Z_STREAM_END (1).

Character count

gzip -cd file.gz | wc -c
 452058742

What may cause this problem? How to investigate it? How to fix it?

Thx.

benkasminbullock commented 10 months ago

Your file is 500 MB large, and gunzip_file tries to read the entire file into memory before uncompressing it, so it is very likely to have been an error caused by insufficient memory.

Gzip::Faster is mostly meant for compressing web pages and other small things, but faster than the standard module of Perl.

I think the best I can do here is to try to improve the error message.

On Thu, 7 Dec 2023 at 19:52, XSven @.***> wrote:

perl version: 5.14.2 Gzip::Faster version: 0.21 operating system: AIX 7.3.0.0 zlib version: zlib-1.2.13-1.ppc, zlib-devel-1.2.13-1.ppc

perl -MGzip::Faster=gunzip_file -we 'gunzip_file(shift);' file.gz Zlib did not finish processing the string: 6327772 bytes left at .../perl5/aix-thread-multi-64all/Gzip/Faster.pm line 76.

Character count

gzip -cd file.gz | wc -c 452058742

What may cause this problem? How to investigate it? How to fix it?

Thx.

— Reply to this email directly, view it on GitHub https://github.com/benkasminbullock/gzip-faster/issues/8, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAY45PDW2PSJORJVY2MJEDYIGNWXAVCNFSM6AAAAABAK3FMPKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTANBXGAYTMMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

XSven commented 10 months ago

Thx for the quick feedback. Yes improving the error message would be good. In general I would suggest that "all" the error messages should contain the zlib status information if applicable. Otherwise everybody has to apply a temporary patch at the right place to get the zlib status information.

By the way what is the "standard module of Perl" that you are refering to. I have tried the following

perl -MIO::Uncompress::Gunzip -we 'my $gu = IO::Uncompress::Gunzip->new( shift, { Append => 1 } );my $buffer; 1 while $gu->read( $buffer ) > 0; print $buffer' file.gz > output

The output file is truncated. It contains exactly 64MB. No error ($GunzipError is not set) and/or warning was raised. The status of the read()method is the empty string "".