Open tlhackque opened 2 years ago
Also, FWIW: I did some code reading in Storable that may be helpful. Storable.pm calles the C code pretrieve
, which calls do_retrieve
, where, after some initialization, a call to magic_check is made.
magic_check
ends up doing a READ()
at line 6966 . READ()
is a C macro.
For a file handle, READ()
ends up at line 1183 where (PerlIO_read(cxt->fio, x, y) != (SSize_t)y)
is done.
PerlIO_read is documented as returning the bytes read.
So, this would seem to come down to whether the call to IO::Compress::Base::Common::createSelfTiedObject produces a PerlIO-compatible file handle.
And, if as it appears, it does not, what can be done about it. I have the vague sense that perhaps implementing the file handle as a PerlIO
layer rather than (or on the way to) a tied object might be a fruitful approach. See PerlIO::via But that's a guess. I'm not sure how you would hook up the handle you generate to provide the lowest IO layer.
Not being an XS
or PerlIO
person, this about exhausts my ability to diagnose what's going on. But hopefully you can take this to a helpful conclusion...
Couldn't leave well enough alone. Layers seems to be on the right track.
This proof of concept works, and could be used as the basis of a patch to get IO::Uncompress::Unzip
to produce a real file handle.
The toy example below adds a 'layer' command to the end of the previous reproducer. This provides a custom layer that returns the Storable hash. Storable::fd_retrieve
from the file handle succeeds. Obviously, a real patch - which I won't try to do - would hook up FILL
to the zip stream in the innards of IO::Uncompress
, and would fill in the other via
subroutines - SYSOPEN
, TELL
, SEEK
, CLOSE
come to mind, but I'm not exactly an expert. perldoc PerlIO::via
has the full list - very similar to tie
.
The second command inserts an intermediate file handle, with a similar layer, between IO::Uncompress::Unzip
and Storable
. This also works. So you have a path to a patch.
Here are the details:
To keep this short - replace the final die
and exit
in the previous code with this:
if( $cmd eq 'layer' ) {
my $bar;
open( my $file, '<:via(ME)', \$bar ) or die( "open $!\n" );
my $read = fd_retrieve( $file );
if( !$read || !exists $read->{key} ){
print "Failed\n";
} else {
print "OK\n";
}
exit;
}
if( $cmd eq 'layereduncompress' ) {
my $in = IO::Uncompress::Unzip->new( \*STDIN, { } ) or die( "IUU\n" );
binmode( $in );
open( my $via, '<:via(TOO)', $in) or die( "via: $!\n" );
my $read = fd_retrieve( $via );
if( !$read || !exists $read->{key} ){
print "Failed\n";
} else {
print "OK\n";
}
exit;
}
die( "Bad command $cmd\n" );
exit;
# Packages implementing PerlIO layers using PerlIO::via
package ME;
my $eof;
sub PUSHED { my($class, $mode, $fh) = @_; bless \*PUSHED, $class };
sub OPEN { my($self, $path, $mode, $fh ) = @_; return 1; };
sub BINMODE { return 0 };
sub FILL { return if $eof; $eof = 1; return pack 'C*', map { eval "0$_" } split( / /, '160 163 164 060 004 011 004 061 062 063 064 004 004 004 010 003 001 000 000 000 012 005 166 141 154 165 145 003 000 000 000 153 145 171'); };
1;
package TOO;
sub PUSHED { my($class, $mode, $fh) = @_; bless {}, $class };
sub OPEN { my($self, $path, $mode, $fh ) = @_; $self->{zip} = $path; return 1; };
sub BINMODE { return 0 };
sub FILL {
my $self = shift;
my $zip = $self->{zip};
return <$zip>;
}
1;
The package ME
in the first run returns the Storable
from a literal.
In the second, the package TOO
is open
ed with the file handle from IO::Uncompress::Unzip
. It reads from that file handle, and returns whatever it gets. Storable
likes reading from that too.
And in the third, we see that transparent mode thru IO::Uncompress::Unzip
(and via(TOO)
also works.
#./IUZ-Storable.pl layer
OK
# ./JUZ-Storable.pl layereduncompress <foo.zip
OK
#./JUZ-Storable.pl layereduncompress <foo
OK
Hi @tlhackque
my, you have been busy!
Not sure if the problem is with my code or with Storable
, but I see you have already opened a ticket against Storable
as well. I don't know the layers interface that well, so I will need to find some time to dig into this one a bit more to get a better understanding of the issue.
Also need to check if writing to a file with IO::Compress::*
+ Storable
works.
Thanks for the responses. Yes, it's been an adventure - a side trip from my real project.
I didn't investigate writes, as my project doesn't require them (yet, anyhow). Agree that they should also provide PerlIO
filehandles. The layers API would allow that.
The issue is that Storable
wants only a PerlIO
handle. The XS code there doesn't understand tie'd FILEHANDLES - it calls directly into the PerlIO
API to read.
This is a problem in both modules;
IO::Uncompress::Unzip
who also expect "real" file handles. Storable
really should take anything that looks and acts like a FILEHANDLE. So this really needs to be fixed in both places. Either would solve my immediate problem (I'm using the via()
hack for now), but a complete solution for everyone (for all time) requires both.
Layers (I pointed you to the documentation on my little adventure) are just a sneaky way of providing a real PerlIO
file handle instead of a tie'd object.
The short form of my little demo is that open()
using the PerlIO::via
mechanism produces a "real" PerlIO
file handle. By intercepting the open, the layer (ususally used for code translation, line ending adjustment, and even for timestamps), can provide data from any source. It's quite similar to tie
.
PUSHED
is inside the open, just before the OPEN
callback. $path
in the OPEN
can be anything - including your object. The demo blesses a small hash (a scalar would do) since it acts as an intermediary. BINMODE
returns 0 to prevent the caller from removing this layer. FILL
provides the data to the upper layers to buffer.I knew layers exist - but this was my first adventure inside one - so it doesn't take too long to learn - despite the poor documentation. Obviously, I'm no expert. But let me know if I can help explain further.
I noted another victim of Storable
's method of reading files in the issue I filed there.
So the good news is: this confirms that you're not alone. The bad news is that the the other victim reported the issue in 2006...and gave up in 2011.
Sigh.
I have a small reproducer demonstrating that IO::Uncompress::Unzip's file descriptor won't work with Storable. Presumably there are other cases...
If I had to guess, it would be that the FH, which is passed to Storable's XS code, may not be positioned (for XS) correctly after header detection...Clearly, it works for ordinary Perl consumers.
Here is the sequence:
\*STDIN
& verify that it's what we wrote. (passes)From this, we can see that the file is written correctly by Storable, and can be read after a trip thru the Zip archive. Further, Storable is happy reading from a file handle.
BUT, neither the raw file nor the Zip member can be read correctly by Storable if IUU is in the middle.
IO::Uncompress::Unzip 2.102
In real life, the work-around is to write all 300MB of uncompressed data to a tempfile, and pass that handle to Storable. Which is suboptimal...
Here is the test program:
And here are the commands (Yes, it fails with newer versions of Perl, and yes in real life there's error checking):
FWIW, strace doesn't show anything odd on the system side of the pipe.