:mmap flag in open cause hang in eof function on files >=2G

[p5pRT]

Migrated from rt.perl.org#119475 (status was 'resolved')

Searchable as RT119475$

[p5pRT]

From perl-diddler@tlinx.org

Created by perl-diddler@tlinx.org

perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; use P; P "eof?"; if (eof $h) { P "yes"} else {P "no"} ' ----- "2G" is a 2G file (dd if={source} of=2G bs=1G count=2)

it works at 2GB-1\, but at 2GB and greater\, it hits the eof function and never comes back.

Without the mmap\, it works (at a 50% or greater speed penalty).

(FWIW\, the machine I am running on has​:

free -h   total used free shared buffers cached Mem​: 94G 92G 2.0G 0B 456K 76G -/+ buffers/cache​: 15G 78G Swap​: 8.0G 95M 7.9G ---- alot of "free" memory. It may be OS or driver related\, I don't know\, but since it works w/o mmap\, that is a fallback position -- though certainly not a desirable one.

I map pairs of files & compare them. For larger files\, I try a sparse compare first -- like only comparing maybe 4M out of every 64.

If all the segments compare equal\, then I have to do a full file compare\, but in doing the I/O\, there is alot of seeking as well as many files being mapped\, then released. If they are read into buffers first\, there is a large performance penalty.

Perl Info

``` Flags: category=core severity=high This perlbug was built using Perl 5.16.2 - Fri Feb 15 01:17:37 UTC 2013 It is being executed now by Perl 5.16.2 - Fri Feb 15 01:12:05 UTC 2013. Site configuration information for perl 5.16.2: Configured by abuild at Fri Feb 15 01:12:05 UTC 2013. Summary of my perl5 (revision 5 version 16 subversion 2) configuration: Platform: osname=linux, osvers=3.4.6-2.10-default, archname=x86_64-linux-thread-multi uname='linux build34 3.4.6-2.10-default #1 smp thu jul 26 09:36:26 utc 2012 (641c197) x86_64 x86_64 x86_64 gnulinux ' config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe', cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector' ccversion='', gccversion='4.7.2 20130108 [gcc-4_7-branch revision 195012]', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector' libpth=/lib64 /usr/lib64 /usr/local/lib64 libs=-lm -ldl -lcrypt -lpthread perllibs=-lm -ldl -lcrypt -lpthread libc=/lib64/libc-2.17.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='2.17' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.16.2/x86_64-linux-thread-multi/CORE' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector' Locally applied patches: @INC for perl 5.16.2: /home/law/bin/lib /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.16.2 /usr/lib/perl5/vendor_perl/5.16.2/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.16.2 /usr/lib/perl5/5.16.2/x86_64-linux-thread-multi /usr/lib/perl5/5.16.2 /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.16.2 /usr/lib/perl5/site_perl . Environment for perl 5.16.2: HOME=/home/law LANG=en_US.UTF-8 LANGUAGE (unset) LC_COLLATE=C LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/usr/lib64/mpi/gcc/openmpi/lib64 LOGDIR (unset) PATH=/home/law/bin/lib:/sbin:/usr/local/sbin:/usr/lib64/mpi/gcc/openmpi/bin:/home/law/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/usr/lib/qt3/bin:/opt/dell/srvadmin/bin:/usr/sbin:/etc/local/func_lib:/home/law/lib PERL5OPT=-Mutf8 -CSA -I/home/law/bin/lib PERL_BADLANG (unset) SHELL=/bin/bash ```

[p5pRT]

From @Leont

On Tue\, Aug 27\, 2013 at 1​:32 AM\, Linda Walsh \perlbug\-followup@&#8203;perl\.orgwrote​:

perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; use P; P "eof?"; if (eof $h) { P "yes"} else {P "no"} ' ----- "2G" is a 2G file (dd if={source} of=2G bs=1G count=2)

it works at 2GB-1\, but at 2GB and greater\, it hits the eof function and never comes back.

5.18 fixed a number of bugs related to >2GB strings\, could you check if a more recent version of perl shows this same behavior?

Leon

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From perl-diddler@tlinx.org

On Mon Aug 26 16​:44​:05 2013\, LeonT wrote​:

5.18 fixed a number of bugs related to >2GB strings\, could you check if a more recent version of perl shows this same behavior?

---

Would love to\, but 5.18 fails to build on my machine the same as 5.16 does. Fails in the database integration. AFAIK\, Suse isn't providing new versions of perl for their releases\, as they version lock much of their SW to the specific version of perl (including subreleases) and won't build new versions to include the previous version's lib dirs.

Their rpm /make file has sUsE specific changes to the db sections that are only geared toward building on a clean build machine. They don't believe in building or testing with a full devel install of their own product\, fresh from factory -- no eating their own dog food for them...

So the answer is ... not real quickly. Sorry. Just narrowing it down to the specific function has taken several months. This same program found an earlier problem when reading large files where some counters were off by 1 or something when going past a 4G mark. But not being able to build perl has been a problem for over a year.

[p5pRT]

From @tux

On Tue\, 27 Aug 2013 01​:43​:07 +0200\, Leon Timmermans \fawaka@&#8203;gmail\.com wrote​:

On Tue\, Aug 27\, 2013 at 1​:32 AM\, Linda Walsh \perlbug\-followup@&#8203;perl\.orgwrote​:

perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; use P; P "eof?"; if (eof $h) { P "yes"} else {P "no"} ' ----- "2G" is a 2G file (dd if={source} of=2G bs=1G count=2)

it works at 2GB-1\, but at 2GB and greater\, it hits the eof function and never comes back.

5.18 fixed a number of bugs related to >2GB strings\, could you check if a more recent version of perl shows this same behavior?

$ ll 2G 16252937 -rw-rw-rw- 1 merijn 10197045250 2013-08-27 08​:25​:31 2G $ perl -wE'open my$fh\,"\<​:raw​:mmap"\,"2G" or die;say"eof?"\,eof$fh?"yes"​:"no"' eof?no $ bleadperl -wE'open my$fh\,"\<​:raw​:mmap"\,"2G" or die;say"eof?"\,eof$fh?"yes"​:"no"' eof?no

Linux 3.4.47-2.38-desktop [openSUSE 12.2 (Mantis)] x86_64 Xeon(R) CPU E5-1650 0 @​ 3.20GHz/1300(12) x86_64 16009 Mb perl​: This is perl 5\, version 16\, subversion 2 (v5.16.2) built for x86_64-linux-ld bleadperl​: This is perl 5\, version 19\, subversion 4 (v5.19.4 (v5.19.3-138-g31d0736)) built for x86_64-linux

No hangs\, instant returns

Leon

-- H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX\, AIX\, and openSUSE http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/ http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

From @tux

On Mon\, 26 Aug 2013 16​:32​:48 -0700\, Linda Walsh (via RT) \perlbug\-followup@&#8203;perl\.org wrote​:

perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; use P; P "eof?"; if (eof $h) { P "yes"} else {P "no"} '

reporting "bug"s using unneeded and possibly confusing modules is to say at least confusing. Please do NOT use something as P when "say" will do

$ perl -wE'open my$fh\,"\<​:raw​:mmap"\,"2G" or die;say"eof?"\,eof$fh?"yes"​:"no"'

will be easier to read for all that do not have P\, never use P and never will use P\, how easy it may be for you. To have people test bugs\, please stick to the language as close as possible.

Besides that\, say is shorter than "use P;" plus P.

-- H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX\, AIX\, and openSUSE http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/ http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

From @iabyn

On Tue\, Aug 27\, 2013 at 01​:43​:07AM +0200\, Leon Timmermans wrote​:

On Tue\, Aug 27\, 2013 at 1​:32 AM\, Linda Walsh \perlbug\-followup@&#8203;perl\.orgwrote​:

perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; use P; P "eof?"; if (eof $h) { P "yes"} else {P "no"} ' ----- "2G" is a 2G file (dd if={source} of=2G bs=1G count=2)

it works at 2GB-1\, but at 2GB and greater\, it hits the eof function and never comes back.

5.18 fixed a number of bugs related to >2GB strings\, could you check if a more recent version of perl shows this same behavior?

I can confirm that its still present in blead. strace shows that it gets stuck in this loop​:

  mmap(NULL\, 2147483648\, PROT_READ\, MAP_SHARED\, 4\, 0) = 0x7fb1831ad000   lseek(4\, 0\, SEEK_SET) = 0   lseek(4\, 0\, SEEK_CUR) = 0   munmap(0x7fb1831ad000\, 2147483648) = 0   lseek(4\, 0\, SEEK_SET) = 0   lseek(4\, 0\, SEEK_CUR) = 0   fstat(4\, {st_mode=S_IFREG|0664\, st_size=2147483648\, ...}) = 0   mmap(NULL\, 2147483648\, PROT_READ\, MAP_SHARED\, 4\, 0) = 0x7fb1831ad000   lseek(4\, 0\, SEEK_SET) = 0   lseek(4\, 0\, SEEK_CUR) = 0   munmap(0x7fb1831ad000\, 2147483648) = 0   ...

I haven't looked into it any further.

-- "Strange women lying in ponds distributing swords is no basis for a system of government. Supreme executive power derives from a mandate from the masses\, not from some farcical aquatic ceremony."   -- Dennis\, "Monty Python and the Holy Grail"

[p5pRT]

From perl-diddler@tlinx.org

On Mon Aug 26 23​:46​:39 2013\, hmbrand wrote​:

reporting "bug"s using unneeded and possibly confusing modules is to say at least confusing. Please do NOT use something as P when "say" will do

I know complex usages like "P" are difficult and confusing for some people\, but when I use say in my example\, I get​: perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; say "eof?"; if (eof $h) { say "yes"} else {say "no"}' String found where operator expected at -e line 3\, near "say "eof?""   (Do you need to predeclare say?) String found where operator expected at -e line 5\, near "say "yes""   (Do you need to predeclare say?) String found where operator expected at -e line 5\, near "say "no""   (Do you need to predeclare say?) syntax error at -e line 3\, near "say "eof?"" syntax error at -e line 5\, near "say "yes"" syntax error at -e line 5\, near "say "no"" Execution of -e aborted due to compilation errors.

$ perl -wE'open my$fh\,"\<​:raw​:mmap"\,"2G" or die;say"eof?"\,eof$fh?"yes"​:"no"' will be easier to read for all that do not have P\, never use P and never will use P\, how easy it may be for you. To have people test bugs\,

Think of 'P' as a generic print thing that does the "right thing".

"say" doesn't handle formats or printing of objects....

I often print things with formats\, "say" doesn't handle that.
I also\, often print "objects" or refs to ARRAYs/HASHs\, "say" doesn't handle that. My data is often a mess and "undef"s need to be handled without disrupting the output. Does 'say' do that?

Just think of it as a 1 char abbreviation that stands for your favorite print 'thing'and you can feel free to substitute in whatever you want for it.

Besides that\, say is shorter than "use P;" plus P.

---

You got me there. I should have used

  perl -weMP'use strict;...'

Then My prog would have been equal in length with 1 'say'\, but shorter in my actual program where I used P twice.

Also problematic I just realized\, is that "say" is just as likely as "when"/"given" to give experimental warnings as it was introduced in the same version. It could just as easily issue warnings as not.

=========================

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

OS? HW? ...!!!.. choice of generic "print thing"? ;^) (I did think about whether to throw that last one in\, honestly\, but it still came out).

======================================================================

Re​: from​: Dave Mitchell \<davem(at)iabyn.com>

I can confirm that its still present in blead. v.

[p5pRT]

From @rjbs

* Linda Walsh via RT \perlbug\-followup@&#8203;perl\.org [2013-08-27T17​:51​:47]

Think of 'P' as a generic print thing that does the "right thing".

Please submit minimal reproducers that don't require installing external libraries. Even if P.pm is pretty simple\, it's a bunch more complicated to test against blead to see if it's fixed in blead. Without "use P\," the program can be run *without installing* in the build dir\, which makes it really easy for anybody to casually check the status of the bug.

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

That is definitely a more interesting question\, though!

-- rjbs

[p5pRT]

From @tux

On Tue\, 27 Aug 2013 21​:03​:05 -0400\, Ricardo Signes \perl\.p5p@&#8203;rjbs\.manxome\.org wrote​:

* Linda Walsh via RT \perlbug\-followup@&#8203;perl\.org [2013-08-27T17​:51​:47]

Think of 'P' as a generic print thing that does the "right thing".

On Mon Aug 26 23​:46​:39 2013\, hmbrand wrote​:

reporting "bug"s using unneeded and possibly confusing modules is to say at least confusing. Please do NOT use something as P when "say" will do

I know complex usages like "P" are difficult and confusing for some people\, but when I use say in my example\, I get​: perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; say "eof?"; if (eof $h) { say "yes"} else {say "no"}' String found where operator expected at -e line 3\, near "say "eof?"" (Do you need to predeclare say?)

perl -e vs perl -E

Please submit minimal reproducers that don't require installing external libraries. Even if P.pm is pretty simple\, it's a bunch more complicated to test against blead to see if it's fixed in blead. Without "use P\," the program can be run *without installing* in the build dir\, which makes it really easy for anybody to casually check the status of the bug.

100% agree

And however useful P might be in your daily work\, and I believe for you it probably is\, it is absolutely unneeded in this example.

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

That is definitely a more interesting question\, though!

Threading issues? My perls are unthreaded I personally would not subscribe to Linda's opinion that it is SUSE's fault\, also because Dave is able to reproduce

Here's my full perl -V

Summary of my perl5 (revision 5 version 16 subversion 2) configuration​:

  Platform​:   osname=linux\, osvers=3.4.11-2.16-desktop\, archname=x86_64-linux-ld   uname='linux pc09 3.4.11-2.16-desktop #1 smp preempt wed sep 26 17​:05​:00 utc 2012 (259fc87) x86_64 x86_64 x86_64 gnulinux '   config_args='-Duse64bitall -Duselongdouble -des'   hint=recommended\, useposix=true\, d_sigaction=define   useithreads=undef\, usemultiplicity=undef   useperlio=define\, d_sfio=undef\, uselargefiles=define\, usesocks=undef   use64bitint=define\, use64bitall=define\, uselongdouble=define   usemymalloc=n\, bincompat5005=undef   Compiler​:   cc='ccache cc'\, ccflags ='-fPIC -fno-strict-aliasing -pipe -fstack-protector -I/pro/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\,   optimize='-O2'\,   cppflags='-fPIC -fno-strict-aliasing -pipe -fstack-protector -I/pro/local/include'   ccversion=''\, gccversion='4.7.1 20120723 [gcc-4_7-branch revision 189773]'\, gccosandvers=''   intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16   ivtype='long'\, ivsize=8\, nvtype='long double'\, nvsize=16\, Off_t='off_t'\, lseeksize=8   alignbytes=16\, prototype=define   Linker and Libraries​:   ld='ccache cc'\, ldflags ='-L/pro/local/lib -fstack-protector'   libpth=/pro/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib /usr/local/lib /lib64 /usr/lib64 /usr/local/lib64   libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat   perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc   libc=/lib/libc-2.15.so\, so=so\, useshrplib=false\, libperl=libperl.a   gnulibc_version='2.15'   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E'   cccdlflags='-fPIC'\, lddlflags='-shared -O2 -L/pro/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:   Compile-time options​: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV   PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_ALL   USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE   USE_LOCALE_COLLATE USE_LOCALE_CTYPE   USE_LOCALE_NUMERIC USE_LONG_DOUBLE USE_PERLIO   USE_PERL_ATOF   Built under linux   Compiled at Nov 1 2012 17​:27​:02   @​INC​:   /pro/lib/perl5/site_perl/5.16.2/x86_64-linux-ld   /pro/lib/perl5/site_perl/5.16.2   /pro/lib/perl5/5.16.2/x86_64-linux-ld   /pro/lib/perl5/5.16.2   .

-- H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX\, AIX\, and openSUSE http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/ http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

From @iabyn

On Wed\, Aug 28\, 2013 at 10​:04​:09AM +0200\, H.Merijn Brand wrote​:

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

That is definitely a more interesting question\, though!

Threading issues? My perls are unthreaded I personally would not subscribe to Linda's opinion that it is SUSE's fault\, also because Dave is able to reproduce

I can reproduce on threaded and unthreaded. Its a 32/64 signed integer wrapping issue.

PerlIOBuf_get_cnt() returns SSize_t\, while Perl_PerlIO_get_cnt() which calls it\, returns int.

This code demonstrates the issue​:

  #include \<stdio.h>

  /* this is what's in my stdio.h​:   typedef long int __ssize_t;   typedef __ssize_t ssize_t;   */

  #define SSize_t ssize_t

  SSize_t f () { return 0x80000000; }   int g () { return f(); }

  int main(int argc\, char**argv)   {   SSize_t avail = g();   printf("sizeof(int)=%d sizeof(SSize_t)=%d\n"\, sizeof(int)\, sizeof(SSize_t));   if (avail > 0)   printf("avail id positive\n");   }

which on my system outputs​:

  sizeof(int)=4 sizeof(SSize_t)=8

But I don't know my round PerlIO\, so I'm not comfortable just changing the int return type to SSize_t.

-- Technology is dominated by two types of people​: those who understand what they do not manage\, and those who manage what they do not understand.

[p5pRT]

From @Leont

On Wed\, Aug 28\, 2013 at 1​:11 PM\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Wed\, Aug 28\, 2013 at 10​:04​:09AM +0200\, H.Merijn Brand wrote​:

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

That is definitely a more interesting question\, though!

Threading issues? My perls are unthreaded I personally would not subscribe to Linda's opinion that it is SUSE's fault\, also because Dave is able to reproduce

I can reproduce on threaded and unthreaded. Its a 32/64 signed integer wrapping issue.

PerlIOBuf_get_cnt() returns SSize_t\, while Perl_PerlIO_get_cnt() which calls it\, returns int.

This code demonstrates the issue​:

\#include \<stdio\.h>

/\* this is what's in my stdio\.h&#8203;:
typedef long int \_\_ssize\_t;
typedef \_\_ssize\_t ssize\_t;
\*/

\#define SSize\_t ssize\_t

SSize\_t f \(\) \{ return 0x80000000; \}
int     g \(\) \{ return f\(\); \}

int main\(int argc\, char\*\*argv\)
\{
    SSize\_t avail = g\(\);
    printf\("sizeof\(int\)=%d sizeof\(SSize\_t\)=%d\\n"\, sizeof\(int\)\,

sizeof(SSize_t)); if (avail > 0) printf("avail id positive\n"); }

which on my system outputs​:

sizeof\(int\)=4 sizeof\(SSize\_t\)=8

But I don't know my round PerlIO\, so I'm not comfortable just changing the int return type to SSize_t.

Given that the Get_cnt method in the PerlIO vtable returns a SSize_t\, I'd say it's safe to change Perl_PerlIO_get_cnt in blead to return a SSize_t. Sadly I don't think we can do that in maint\, as that would break binary compatibility.

Leon

[p5pRT]

From perl-diddler@tlinx.org

On Wed Aug 28 01​:04​:55 2013\, hmbrand wrote​:

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

That is definitely a more interesting question\, though!

Threading issues? My perls are unthreaded I personally would not subscribe to Linda's opinion that it is SUSE's fault\, also because Dave is able to reproduce

---

  Would you care to attribute your statement that I blamed Suse for this? My feelings were that it was a bug in perl or libc where 64 bits are not being used for file offsets -- most likely how perl calls libc\, but that I would assume that would likely be "offensive"\, despite the appropriateness of using probabilistic reasoning.

[p5pRT]

From perl-diddler@tlinx.org

On Wed Aug 28 05​:03​:49 2013\, LeonT wrote​:

which on my system outputs​:

sizeof\(int\)=4 sizeof\(SSize\_t\)=8

But I don't know my round PerlIO\, so I'm not comfortable just changing the int return type to SSize_t. Given that the Get_cnt method in the PerlIO vtable returns a SSize_t\, I'd say it's safe to change Perl_PerlIO_get_cnt in blead to return a SSize_t. Sadly I don't think we can do that in maint\, as that would break binary compatibility.

---

On my (Suse's) perl\, I have :

use64bitint=define\, use64bitall=define\, (as well as intsize=4\, )

Isn't binary compatibility already broken in maint as perl isn't really using 64bitint or 64bitall?

[p5pRT]

From @Leont

On Thu\, Aug 29\, 2013 at 12​:20 AM\, Linda Walsh via RT \< perlbug-followup@​perl.org> wrote​:

On my (Suse's) perl\, I have :

use64bitint=define\, use64bitall=define\, (as well as intsize=4\, )

Isn't binary compatibility already broken in maint as perl isn't really using 64bitint or 64bitall?

64bitall affects the size of an IV\, not an int. We can't affect the size of the latter even if we wanted to\, that's why we use the IV abstraction. That is working exactly as intended.

Leon

[p5pRT]

From perl-diddler@tlinx.org

On Wed Aug 28 15​:20​:32 2013\, LAWalsh wrote​:

On my (Suse's) perl\, I have :

use64bitint=define\, use64bitall=define\, (as well as intsize=4\, )

Isn't binary compatibility already broken in maint as perl isn't really using 64bitint or 64bitall?

---

Actually\, that shouldn't be a question. If perl is using 32bit int's in parameter passing\, internally\, when it has been directed to use 64-bit int's\, perl has already broken binary compat\, because you can't use 32-bit int's for sizes on 64-bit machines.

I.e binary compatibility is already broken. Changing the source so that it makes the size match on 64-bit machines would be fixing binary compatibility.

Conversely\, can you think of a use case where using the the binary compatibility with the native cause a problem during code execution?

[p5pRT]

From @doy

On Wed\, Aug 28\, 2013 at 03​:28​:38PM -0700\, Linda Walsh via RT wrote​:

On Wed Aug 28 15​:20​:32 2013\, LAWalsh wrote​:

On my (Suse's) perl\, I have :

use64bitint=define\, use64bitall=define\, (as well as intsize=4\, )

Isn't binary compatibility already broken in maint as perl isn't really using 64bitint or 64bitall? --- Actually\, that shouldn't be a question. If perl is using 32bit int's in parameter passing\, internally\, when it has been directed to use 64-bit int's\, perl has already broken binary compat\, because you can't use 32-bit int's for sizes on 64-bit machines.

I.e binary compatibility is already broken. Changing the source so that it makes the size match on 64-bit machines would be fixing binary compatibility.

Conversely\, can you think of a use case where using the the binary compatibility with the native cause a problem during code execution?

"Binary compatibility" in this context means compatibility with binary perl extensions\, not compatibility with libc.

-doy

[p5pRT]

From perl-diddler@tlinx.org

On Wed Aug 28 15​:32​:51 2013\, doy@​tozt.net wrote​:

Conversely\, can you think of a use case where using the the binary compatibility with the native cause a problem during code execution?

"Binary compatibility" in this context means compatibility with binary perl extensions\, not compatibility with libc.

---

If those extensions are only using 32 bits for file offsets and sizes\, they are already broken on 32-bit machines using large-file support as well as on 64-bit machines.

As it stands now\, anyplace where this filters through to the outside api and they are using 32-bit offsets is broken.

That I've found 2 of these bugs in working with large files indicates that perl and perl testing is seeing a lack of sufficient exposure to areas where large file usage is common enough to be normal and certainly common enough to require a mandatory fix.

I'm not entirely sure what you mean by 'maint' (5.16.3\, 5.18.1?) but not fixing that is saying that perl programs shouldn't *reliably* able to be manipulate or use files beyond what 32-bits will support. In this day and age\, I would call that unacceptable.

[p5pRT]

From @doy

On Wed\, Aug 28\, 2013 at 05​:41​:43PM -0700\, Linda Walsh via RT wrote​:

On Wed Aug 28 15​:32​:51 2013\, doy@​tozt.net wrote​:

Conversely\, can you think of a use case where using the the binary compatibility with the native cause a problem during code execution?

"Binary compatibility" in this context means compatibility with binary perl extensions\, not compatibility with libc. ---- If those extensions are only using 32 bits for file offsets and sizes\, they are already broken on 32-bit machines using large-file support as well as on 64-bit machines.

As it stands now\, anyplace where this filters through to the outside api and they are using 32-bit offsets is broken.

That I've found 2 of these bugs in working with large files indicates that perl and perl testing is seeing a lack of sufficient exposure to areas where large file usage is common enough to be normal and certainly common enough to require a mandatory fix.

I'm not entirely sure what you mean by 'maint' (5.16.3\, 5.18.1?) but not fixing that is saying that perl programs shouldn't *reliably* able to be manipulate or use files beyond what 32-bits will support. In this day and age\, I would call that unacceptable.

If this is changed in a maint release\, it will be broken for files of all sizes (probably actually causing a segfault)\, rather than only for files of sizes larger than the size of an int. This is unacceptable breakage for a maint release.

-doy

[p5pRT]

From perl-diddler@tlinx.org

On Wed Aug 28 17​:44​:43 2013\, doy@​tozt.net wrote​:

If this is changed in a maint release\, it will be broken for files of all sizes (probably actually causing a segfault)\, rather than only for files of sizes larger than the size of an int. This is unacceptable breakage for a maint release.

---

Then you need to make sure that people know that perl is not capable of reliably processing or handling files >=2G.

As such I would call the product broken -- as I said\, it wasn't clear what you would call a maintenance release. But it seems to be the case that you are unable to release a working perl that reliably does file I/O if the file sizes are > 2G -- even if it requires a recompile of modules with a binary API.

Having to do a "cpan -r"\, once is better than having a perl that doesn't work.

[p5pRT]

From @Smylers

Linda Walsh via RT writes​:

On Mon Aug 26 23​:46​:39 2013\, hmbrand wrote​:

reporting "bug"s using unneeded and possibly confusing modules is to say at least confusing. Please do NOT use something as P when "say" will do

... when I use say in my example\, I get​: perl -we'use strict; open (my $h\, "\<​:raw​:mmap"\, "2G") or "die"; say "eof?"; if (eof $h) { say "yes"} else {say "no"}' String found where operator expected at -e line 3\, near "say "eof?"" (Do you need to predeclare say?) String found where operator expected at -e line 5\, near "say "yes"" (Do you need to predeclare say?) String found where operator expected at -e line 5\, near "say "no"" (Do you need to predeclare say?) syntax error at -e line 3\, near "say "eof?"" syntax error at -e line 5\, near "say "yes"" syntax error at -e line 5\, near "say "no"" Execution of -e aborted due to compilation errors.

Hi Linda. But what you typed isn't actually what was recommended to you​:

$ perl -wE'open my$fh\,"\<​:raw​:mmap"\,"2G" or die;say"eof?"\,eof$fh?"yes"​:"no"'

Note the upper-case E in the above\, which makes say available.

Just think of [P] as a 1 char abbreviation that stands for your favorite print 'thing'and you can feel free to substitute in whatever you want for it.

It's clearer for people reproducing and investigating bugs you report in they can straightforwardly copy and paste your _exact_ code\, not having to substitute in placeholders. That's the advantage of only using core Perl features when reporting a bug in the Perl core -- we all know we're running precisely the same code as each other.

Besides that\, say is shorter than "use P;" plus P.

You got me there. I should have used

perl \-weMP'use strict;\.\.\.'

Did you try that?

Remember that the quotes are interpreted by your shell\, not seen by perl. What you wrote above is equivalent to​:

  perl '-weMPuse strict;...'

Or\, more clearly​:

  perl -w -e 'MPuse strict;...'

Then My prog would have been equal in length with 1 'say'\, but shorter in my actual program where I used P twice.

Except it still wouldn't have worked\, because MPuse isn't a known Perl function.

Also problematic I just realized\, is that "say" is just as likely as "when"/"given" to give experimental warnings as it was introduced in the same version.

No it isn't. Experimental status isn't allocated based on the version a feature was introduced\, but on the feature itself.

That one\, complex\, feature introduced in a version has turned out to have sufficient awkwardness that it was retrospectively deemed to be experimental in no way implies that another\, much simpler\, feature introduced in the same version and which hasn't been found to be awkward will be likely also to be branded experimental.

In fact\, the complete opposite to your claim is more likely to be true​: given that a feature from that version has been marked as experimental\, clearly enough time since that release has elapsed for awkwardness to surface\, and clearly other features from that release could've been made experimental at the same time but weren't. So one could reasonably claim such features are now pretty safe from being retrospectively dubbed experimental.

It could just as easily issue warnings as not.

Anyway\, what say might do in the future is irrelevant. The point is it doesn't issue warnings right now\, when reporting your bug\, in the version of perl you're reporting the bug against\, nor does it do so in blead. So using say is perfectly safe in your bug report.

Please follow the advice of those who are trying to help you\, and make life easy for them by reporting bugs without irrelevant external dependencies\, so they aren't distracted by them\, don't have to spend time replacing them\, and can be focussed on the actual bug you've found.

Cheers

Smylers -- Stop drug companies hiding negative research results. Sign the AllTrials petition to get all clinical research results published. Read more​: http​://www.alltrials.net/blog/the-alltrials-campaign/

[p5pRT]

From @Leont

On Thu\, Aug 29\, 2013 at 2​:41 AM\, Linda Walsh via RT \< perlbug-followup@​perl.org> wrote​:

If those extensions are only using 32 bits for file offsets and sizes\, they are already broken on 32-bit machines using large-file support as well as on 64-bit machines.

As it stands now\, anyplace where this filters through to the outside api and they are using 32-bit offsets is broken.

You're drawing conclusions too quickly. We are *not* using 32 bits for file offsets and sizes; or better said we aren't on any perl where lseeksize=8\, which should only happen on platforms that don't support large files. A quick scan of perlbugs only gives me VMS on VAX\, Android and Hurd systems implicated such (I suspect the latter is a bug though).

That I've found 2 of these bugs in working with large files indicates that perl and perl testing is seeing a lack of sufficient exposure to areas where large file usage is common enough to be normal and certainly common enough to require a mandatory fix.

Again\, drawing conclusions too quickly. The only :mmap layer layer is broken\, nothing else. The issue was that we used a 32 bit integer for the *buffer size*. Normal IO doesn't use such unusually large buffers so is not affected.

You're correct :mmap is almost entirely untested. That is a known issue and the only reason why it hasn't been dual-lifed or evicted from core yet. The fact that no one noticed this before is probably because hardly anyone is using it. It's a piece of crap that I hope we can evict in a future release of perl. Patches welcome.

I'm not entirely sure what you mean by 'maint' (5.16.3\, 5.18.1?)

Yes\, I meant releases that don't end with a .0

Leon

[p5pRT]

From perl-diddler@tlinx.org

On Thu Aug 29 02​:49​:35 2013\, LeonT wrote​:

On Thu\, Aug 29\, 2013 at 2​:41 AM\, Linda Walsh via RT \<

Again\, drawing conclusions too quickly. The only :mmap layer layer is broken\, nothing else. The issue was that we used a 32 bit integer for the *buffer size*. Normal IO doesn't use such unusually large buffers so is not affected.

You're correct :mmap is almost entirely untested. That is a known issue and the only reason why it hasn't been dual-lifed or evicted from core yet. The fact that no one noticed this before is probably because hardly anyone is using it. It's a piece of crap that I hope we can evict in a future release of perl. Patches welcome.

So I don't understand this fear of a patch causing problem. Let's look at the smallest affect possible -- fixing it just for 64bit intel machines. On those machines int's are aligned in memory by quads. If you only are using 32bits and interfaces are only expecting 32bits\, now\, and you start putting stuff in the high DWORD of the returned quad\, how would that cause a segfault?

Registers and stack are allocated on 64-bit boundaries on 64-bit machines. Is that ouput of those calls pushed on to a stack or returned in a register or written to memory?

Even the later would be likely to write a qword boundary. So existing perl api's that use this (which sound like no one except me?) that read 32 bits would still get the low dword and the top would just be unused -- but I don't see how it would cause a segfault\, if backported.

[p5pRT]

From perl-diddler@tlinx.org

On Thu Aug 29 01​:31​:06 2013\, smylers@​stripey.com wrote​:

Besides that\, say is shorter than "use P;" plus P.

You got me there. I should have used

perl \-weMP'use strict;\.\.\.'

Did you try that?

---

I have used those switches but not in that order.

This works\, if you were curious...

perl -MP -we'use strict; P "test"' test

[p5pRT]

From perl-diddler@tlinx.org

On Thu Aug 29 02​:49​:35 2013\, LeonT wrote​:

On Thu\, Aug 29\, 2013 at 2​:41 AM\, Linda Walsh via RT \< perlbug-followup@​perl.org> wrote​:

If those extensions are only using 32 bits for file offsets and sizes\, they are already broken on 32-bit machines using large-file support as well as on 64-bit machines.

As it stands now\, anyplace where this filters through to the outside api and they are using 32-bit offsets is broken.

You're drawing conclusions too quickly. We are *not* using 32 bits for file offsets and sizes; or better said we aren't on any perl where lseeksize=8\, which should only happen on platforms that don't support large files. A quick scan of perlbugs only gives me VMS on VAX\, Android and Hurd systems implicated such (I suspect the latter is a bug though).

---

  That is the behavior I would expect... I said any place they are using 32-bit offsets would be broken -- not that 32-bit machines were thought to be using such in normal operations. It's _just_ "some operations" that misbehave with mmap.

  You say you use a 32-bit integer to hold the size of some buffer? What buffer?... a modify buffer? I'm not writing to the files (they are opened read-only) that involve mmap. Is it necessary to use a buffer at all?

Looking at the mmap call\, it doesn't appear to take a buffer address though it will take a "hint"\, but it recommended to use NULL and let the kernel choose the memmapped buff location.

  So why is my open( fh\, "\<​:raw​:mmap"\, name) needing that 2G buffer anyway?

That I've found 2 of these bugs in working with large files indicates that perl and perl testing is seeing a lack of sufficient exposure to areas where large file usage is common enough to be normal and certainly common enough to require a mandatory fix.

Again\, drawing conclusions too quickly. The only :mmap layer layer is broken\, nothing else. The issue was that we used a 32 bit integer for the *buffer size*. Normal IO doesn't use such unusually large buffers so is not affected.

---

  Then I wouldn't propose changing anything but the mmap layer -- that's the only problem area that I would feel needs a mandatory backpatch. Other areas that are not broken don't need fixing. I hope you wouldn't think I would suggest fixing/rewriting MORE than is necessary -- that would not be very productive.

You're correct :mmap is almost entirely untested. That is a known issue and the only reason why it hasn't been dual-lifed or evicted from core yet.

---

  I think this is a good example about how CPAN testing doesn't provide a complete usage picture of the perl-devs "at large".

The fact that no one noticed this before is probably because hardly anyone is using it. It's a piece of crap that I hope we can evict in a future release of perl.

---

  Maybe they don't know it is twice as fast (at least in some usages).

Patches welcome.

---

=backstory

  Patches to do exactly what? I don't understand why you have a 2G buffer you are using anyway -- could you explain that? Is it just for writing? Should it be used in reading?

 

[p5pRT]

From perl-diddler@tlinx.org

On Tue Aug 27 18​:03​:51 2013\, perl.p5p@​rjbs.manxome.org wrote​:

* Linda Walsh via RT \perlbug\-followup@&#8203;perl\.org [2013-08-27T17​:51​:47]

Think of 'P' as a generic print thing that does the "right thing".

Please submit minimal reproducers that don't require installing external libraries. Even if P.pm is pretty simple\, it's a bunch more complicated to test against blead to see if it's fixed in blead.

---

Not if you sub the word 'say' or use perl -l 'printf'. The issue was already reproduced by people who used or auto-changed it to an equiv rather than complaining on here about my minimal code.

The one who complained about having to do their own substitution\, most vehemently\, was also the one who couldn't reproduce the bug. So\, I vote more for looking at the issue\, and getting over how I spell say".

More importantly​: why does your testing not show the error at hand when Dave's testing with blead did?

That is definitely a more interesting question\, though!

---

I still don't see the answer to that -- It shouldn't have worked\, given the length of the mmap buff len only holds 0 up to 2G-1. Odd theirs would work...

[p5pRT]

From @tux

On Thu\, 29 Aug 2013 16​:45​:55 -0700\, "Linda Walsh via RT" \perlbug\-followup@&#8203;perl\.org wrote​:

The one who complained about having to do their own substitution\, most vehemently\, was also the one who couldn't reproduce the bug. So\, I vote more for looking at the issue\, and getting over how I spell say".

If you by that mean me\, I was put off in trying harder on other machines by your attitude. That plus the fact that others *did* in the meantime find a way to reproduce and show the cause of the error made me not want to try harder at all.

I wonder why you are trying so hard to demotivate people that can fix bugs you find.

-- H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX\, AIX\, and openSUSE http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/ http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

From @Leont

On Fri\, Aug 30\, 2013 at 1​:25 AM\, Linda Walsh via RT \< perlbug-followup@​perl.org> wrote​:

 That is the behavior I would expect\.\.\.  I said any place they are

using 32-bit offsets would be broken -- not that 32-bit machines were thought to be using such in normal operations. It's _just_ "some operations" that misbehave with mmap.

 You say you use a 32\-bit integer to hold the size of some buffer?

What buffer?... a modify buffer? I'm not writing to the files (they are opened read-only) that involve mmap. Is it necessary to use a buffer at all?

Looking at the mmap call\, it doesn't appear to take a buffer address though it will take a "hint"\, but it recommended to use NULL and let the kernel choose the memmapped buff location.

So why is my open\( fh\, "\<&#8203;:raw&#8203;:mmap"\, name\) needing that 2G buffer

anyway?

The :mmap layer is implemented as an IO layer that uses the memory map as its read/write buffer.

  Then I wouldn't propose changing anything but the mmap layer --

that's the only problem area that I would feel needs a mandatory backpatch. Other areas that are not broken don't need fixing. I hope you wouldn't think I would suggest fixing/rewriting MORE than is necessary -- that would not be very productive.

The bottleneck is in the buffering API\, so the obvious place to fix it would be there. It is probably work around it in :mmap itself\, but that may involve rewriting half of it at little gain.

I just pushed a fix to smoke-me/leont/big-mmap. Don't think we can test it systematically though\, given the filesize.

  Maybe they don't know it is twice as fast (at least in some usages).

I don't suspect using it in an IO layer is one of those circumstances. I'd be most interested in benchmarks proving me wrong.

Leon

[p5pRT]

From perl-diddler@tlinx.org

On Fri Aug 30 02​:28​:02 2013\, LeonT wrote​:

I just pushed a fix to smoke-me/leont/big-mmap. Don't think we can test it systematically though\, given the filesize.

---

How might one download/access the fix?

When I tried through cpan\, I keep getting it asking for a username/password​: Fetching with LWP​: http​://mirrors.kernel.org/CPAN/authors/id/s/s//s/sm/smoke-me/leont/big-mmap.gz

Proxy authentication needed! (Note​: to permanently configure username and password run   o conf proxy_user your_username   o conf proxy_pass your_password   ) Username​:
Password​:

---

(note -- local proxy doesn't require a username/pw\, so I'm guessing it is the remote end?)

[p5pRT]

From @tonycoz

On Thu\, Sep 05\, 2013 at 08​:03​:44PM -0700\, Linda Walsh via RT wrote​:

On Fri Aug 30 02​:28​:02 2013\, LeonT wrote​:

I just pushed a fix to smoke-me/leont/big-mmap. Don't think we can test it systematically though\, given the filesize. ---- How might one download/access the fix?

When I tried through cpan\, I keep getting it asking for a username/password​: Fetching with LWP​: http​://mirrors.kernel.org/CPAN/authors/id/s/s//s/sm/smoke-me/leont/big-mmap.gz

Proxy authentication needed! (Note​: to permanently configure username and password run o conf proxy_user your_username o conf proxy_pass your_password ) Username​:
Password​:

---- (note -- local proxy doesn't require a username/pw\, so I'm guessing it is the remote end?)

He pushed it to the git respository at perl5.git.perl.org\, if you have git installed​:

  git clone -b smoke-me/leont/big-mmap git​://perl5.git.perl.org/perl.git

Your linux distribution should include a git package.

Otherwise​:

  wget http​://perl5.git.perl.org/perl.git/snapshot/smoke-me/leont/big-mmap.tar.gz

or its curl or LWP equivalent should download it.

Note that this is a change against bleadperl\, so if you build this\, you're building a development perl\, not a final release.

Tony

[p5pRT]

From @tux

On Fri\, 6 Sep 2013 14​:14​:31 +1000\, Tony Cook \tony@&#8203;develop\-help\.com wrote​:

On Thu\, Sep 05\, 2013 at 08​:03​:44PM -0700\, Linda Walsh via RT wrote​:

On Fri Aug 30 02​:28​:02 2013\, LeonT wrote​:

I just pushed a fix to smoke-me/leont/big-mmap. Don't think we can test it systematically though\, given the filesize. ---- How might one download/access the fix?

When I tried through cpan\, I keep getting it asking for a username/password​: Fetching with LWP​: http​://mirrors.kernel.org/CPAN/authors/id/s/s//s/sm/smoke-me/leont/big-mmap.gz

Proxy authentication needed! (Note​: to permanently configure username and password run o conf proxy_user your_username o conf proxy_pass your_password ) Username​:
Password​:

---- (note -- local proxy doesn't require a username/pw\, so I'm guessing it is the remote end?)

He pushed it to the git respository at perl5.git.perl.org\, if you have git installed​:

git clone -b smoke-me/leont/big-mmap git​://perl5.git.perl.org/perl.git

Or visit the web interface at http​://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/leont/big-mmap

And the single patch from there http​://perl5.git.perl.org/perl.git/commit/40206cde2a95959c16dd7ff4b4c3e46f09de43ab

Or look at the diff on the web http​://perl5.git.perl.org/perl.git/commitdiff/40206cde2a95959c16dd7ff4b4c3e46f09de43ab

Your linux distribution should include a git package.

Otherwise​:

wget http​://perl5.git.perl.org/perl.git/snapshot/smoke-me/leont/big-mmap.tar.gz

or its curl or LWP equivalent should download it.

Note that this is a change against bleadperl\, so if you build this\, you're building a development perl\, not a final release.

Tony

-- H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/ using perl5.00307 .. 5.19 porting perl5 on HP-UX\, AIX\, and openSUSE http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/ http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

[p5pRT]

From @Leont

On Fri Aug 30 02​:28​:02 2013\, LeonT wrote​:

I just pushed a fix to smoke-me/leont/big-mmap. Don't think we can test it systematically though\, given the filesize.

Maybe they don't know it is twice as fast (at least in some usages).

I don't suspect using it in an IO layer is one of those circumstances. I'd be most interested in benchmarks proving me wrong.

Pushed the fix with b66f3475d343bb78e55b4ba343433044f5966b6b\, which was part of perl 5.19.4. Closing this bug as it has been fixed.

Leon

[p5pRT]

@Leont - Status changed from 'open' to 'resolved'