Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.98k stars 559 forks source link

[BUG] local PerlIO::encoding::fallback settings + encoding causing infinite loop #15117

Open p5pRT opened 8 years ago

p5pRT commented 8 years ago

Migrated from rt.perl.org#127149 (status was 'open')

Searchable as RT127149$

p5pRT commented 8 years ago

From zhouzhen1@gmail.com

Created by zhouzhen1@gmail.com

I saw the issue when I use File​::Slurper + Data​::Table together in one script. I then simplified the demo case to be like below. Basically it's like\, you open a file in with some encoding\, and set $PerlIO​::encoding​::fallback in that "local"\, then out of that "local" scope you open another file with some encoding\, you get an infinite loop when reading the second file.

use 5.022;

my $file1 = '/tmp/foo'; my $file2 = '/tmp/bar'; system("echo bar > $file2");

{ # what's in File​::Slurper   my $layer = '​:raw​:encoding(utf-8)';   local $PerlIO​::encoding​::fallback = 1;   open my $fh1\, ">$layer"\, $file1 or die $!;   close $fh1 or die $!; }

{ # what's in Data​::Table   open(my $fh2\, $file2) or die $!;   binmode $fh2\, "​:encoding(UTF-8)";   while (my $line = \<$fh2>) {   warn $line;   } }

It can also be simplified to like below.

use 5.022;

my $file1 = '/tmp/foo'; my $file2 = '/tmp/bar'; system("echo foo > $file1"); system("echo bar > $file2");

{   local $PerlIO​::encoding​::fallback = 1;   open my $fh1\, "\<​:encoding(utf-8)"\, $file1 or die $!;   close $fh1 or die $!; }

{   open my $fh2\, '\<​:encoding(utf-8)'\, $file2 or die $!;   while (my $line = \<$fh2>) {   warn $line;   } }

Perl Info ``` Flags: category=library severity=medium module=PerlIO Site configuration information for perl 5.22.1: Configured by builduser at Tue Dec 15 10:04:11 CET 2015. Summary of my perl5 (revision 5 version 22 subversion 1) configuration: Platform: osname=linux, osvers=4.3.2-1-arch, archname=x86_64-linux-thread-multi uname='linux flo-64 4.3.2-1-arch #1 smp preempt fri dec 11 07:19:47 cet 2015 x86_64 gnulinux ' config_args='-des -Dusethreads -Duseshrplib -Doptimize=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -Dprefix=/usr -Dvendorprefix=/usr -Dprivlib=/usr/share/perl5/core_perl -Darchlib=/usr/lib/perl5/core_perl -Dsitelib=/usr/share/perl5/site_perl -Dsitearch=/usr/lib/perl5/site_perl -Dvendorlib=/usr/share/perl5/vendor_perl -Dvendorarch=/usr/lib/perl5/vendor_perl -Dscriptdir=/usr/bin/core_perl -Dsitescript=/usr/bin/site_perl -Dvendorscript=/usr/bin/vendor_perl -Dinc_version_list=none -Dman1ext=1perl -Dman3ext=3perl -Dcccdlflags='-fPIC' -Dlddlflags=-shared -Wl,-O1,--sort-common,--as-needed,-z,relro -Dldflags=-Wl,-O1,--sort-common,--as-needed,-z,relro' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong', cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='', gccversion='5.3.0', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags ='-Wl,-O1,--sort-common,--as-needed,-z,relro -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64 libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.22.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='2.22' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/core_perl/CORE' cccdlflags='-fPIC', lddlflags='-shared -Wl,-O1,--sort-common,--as-needed,-z,relro -L/usr/local/lib -fstack-protector-strong' @INC for perl 5.22.1: /home/zhou/perl5/lib/perl5/x86_64-linux-thread-multi /home/zhou/perl5/lib/perl5 /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl . Environment for perl 5.22.1: HOME=/home/zhou LANG=en_US.utf8 LANGUAGE (unset) LC_CTYPE=zh_CN.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/zhou/.rakudobrew/bin:/home/zhou/.rakudobrew/bin:/home/zhou/.rakudobrew/moar-nom/install/share/perl6/site/bin:/home/zhou/.rakudobrew/moar-nom/install/bin:/home/zhou/.rakudobrew/moar-nom/panda/bin:/home/zhou/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl PERL5LIB=/home/zhou/perl5/lib/perl5 PERL_BADLANG (unset) PERL_LOCAL_LIB_ROOT=/home/zhou/perl5 PERL_MB_OPT=--install_base "/home/zhou/perl5" PERL_MM_OPT=INSTALL_BASE=/home/zhou/perl5 SHELL=/bin/bash ```
p5pRT commented 8 years ago

From @tonycoz

On Mon Jan 04 06​:12​:19 2016\, zhouzhen1@​gmail.com wrote​:

I saw the issue when I use File​::Slurper + Data​::Table together in one script. I then simplified the demo case to be like below. Basically it's like\, you open a file in with some encoding\, and set $PerlIO​::encoding​::fallback in that "local"\, then out of that "local" scope you open another file with some encoding\, you get an infinite loop when reading the second file.

The problem is that since PerlIO​::encoding is loaded within the local's block\, $PerlIO​::encoding​::fallback is restored to it's original value at entry to the block - undef.

This can be prevented by explicitly loading PerlIO​::encoding before localizing $PerlIO​::encoding​::fallback

Also\, since PerlIO​::encoding is loaded *after* you initialized fallback\, it's being reset to the default set in PerlIO/encoding.pm before the value is used.

That said\, PerlIO​::encoding shouldn't go into an infinite loop\, the attached patch fixes that for this case by ensuring the STOP_AT_PARTIAL flag is set\, which is needed for decoding.

Tony

p5pRT commented 8 years ago

From @tonycoz

0001-perl-127149-ensure-the-CHECK-value-has-STOP_AT_PARTI.patch ```diff From 460df6ceb4e243f5e2b68e72c445fc4e1e883897 Mon Sep 17 00:00:00 2001 From: Tony Cook Date: Wed, 13 Jan 2016 12:11:45 +1100 Subject: [perl #127149] ensure the CHECK value has STOP_AT_PARTIAL This flag is required for PerlIO::encoding to do its job. Also document the race between localizing $PerlIO::encoding::fallback and loading PerlIO::encoding. --- ext/PerlIO-encoding/encoding.pm | 7 ++++++- ext/PerlIO-encoding/encoding.xs | 5 ++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/ext/PerlIO-encoding/encoding.pm b/ext/PerlIO-encoding/encoding.pm index 7bae036..df1ded6 100644 --- a/ext/PerlIO-encoding/encoding.pm +++ b/ext/PerlIO-encoding/encoding.pm @@ -1,7 +1,7 @@ package PerlIO::encoding; use strict; -our $VERSION = '0.23'; +our $VERSION = '0.24'; our $DEBUG = 0; $DEBUG and warn __PACKAGE__, " called by ", join(", ", caller), "\n"; @@ -46,6 +46,11 @@ When the layer is pushed, the current value of C<$PerlIO::encoding::fallback> is saved and used as the CHECK argument when calling the Encode methods encode() and decode(). +Note: localizing C<$PerlIO::encoding::fallback> without +PerlIO::encoding being loaded and then using the C<:encoding> layer +will leave C<$PerlIO::encoding::fallback> as C at the end of +the scope. + =head1 SEE ALSO L, L, L, L diff --git a/ext/PerlIO-encoding/encoding.xs b/ext/PerlIO-encoding/encoding.xs index 3e4645c..599728c 100644 --- a/ext/PerlIO-encoding/encoding.xs +++ b/ext/PerlIO-encoding/encoding.xs @@ -6,6 +6,9 @@ #define OUR_DEFAULT_FB "Encode::PERLQQ" +/* from Encode's encode.h */ +#define ENCODE_STOP_AT_PARTIAL 0x0800 /* stop at partial explicitly */ + #if defined(USE_PERLIO) /* Define an encoding "layer" in the perliol.h sense. @@ -163,7 +166,7 @@ PerlIOEncode_pushed(pTHX_ PerlIO * f, const char *mode, SV * arg, PerlIO_funcs * PerlIOBase(f)->flags |= PERLIO_F_UTF8; } - e->chk = newSVsv(get_sv("PerlIO::encoding::fallback", 0)); + e->chk = newSViv(SvIV(get_sv("PerlIO::encoding::fallback", 0)) | ENCODE_STOP_AT_PARTIAL); e->inEncodeCall = 0; FREETMPS; -- 2.1.4 ```
p5pRT commented 8 years ago

The RT System itself - Status changed from 'new' to 'open'