Out of memory!, while extending scalar with vec()

[p5pRT]

Migrated from rt.perl.org#41065 (status was 'resolved')

Searchable as RT41065$

[p5pRT]

From kcronan@oversee.net

Created by kcronan@oversee.net

This is a bug report for perl from kcronan@​oversee.net\, generated with the help of perlbug 1.35 running under perl v5.8.8.

----------------------------------------------------------------- Hi there\,

I believe I've found some kind of allocator bug. The script​:

#!/usr/bin/perl my $foo = ''; vec($foo\, (1\<\<$ARGV[0])-1\, 8)=1;

Works fine for me when run with the parameter 27\, but fails with "Out of memory!" when it is 28--a 256M string\, even though there is plenty of virtual memory.

See also http​://www.mail-archive.com/beginners%40perl.org/msg81854.html

I describe there some of the resolutions I attempted\, including compiling with -DNO_FANCY_MALLOC -DPLAIN_MALLOC. But none of them were successful. In the end\, I created an array of several strings of length 2^26. With this approach I was able to allocate\, and address with vec()\, at least 1GB.

Thanks!

Kyle Cronan \kcronan@&#8203;oversee\.net \kyle@&#8203;pbx\.org

Perl Info

``` Flags: category=core severity=medium This perlbug was built using Perl v5.8.8 in the Red Hat build system. It is being executed now by Perl v5.8.8 - Tue Oct 3 11:01:05 EDT 2006. Site configuration information for perl v5.8.8: Configured by Red Hat, Inc. at Tue Oct 3 11:01:05 EDT 2006. Summary of my perl5 (revision 5 version 8 subversion 8) configuration: Platform: osname=linux, osvers=2.6.9-34.elsmp, archname=i386-linux-thread-multi uname='linux hs20-bc2-2.build.redhat.com 2.6.9-34.elsmp #1 smp fri feb 24 16:56:28 est 2006 i686 i686 i386 gnulinux ' config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Dversion=5.8.8 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dinc_version_list=5.8.7 5.8.6 5.8.5 -Dscriptdir=/usr/bin' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm', optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm' ccversion='', gccversion='4.1.1 20060928 (Red Hat 4.1.1-28)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='gcc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='2.5' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -L/usr/local/lib' Locally applied patches: @INC for perl v5.8.8: /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 . Environment for perl v5.8.8: HOME=/home/kyle LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/kyle/bin PERL_BADLANG (unset) SHELL=/bin/bash ```

[p5pRT]

From @iabyn

On Sat\, Dec 09\, 2006 at 09​:36​:49PM -0800\, Kyle Cronan wrote​:

#!/usr/bin/perl my $foo = ''; vec($foo\, (1\<\<$ARGV[0])-1\, 8)=1;

Works fine for me when run with the parameter 27\, but fails with "Out of memory!" when it is 28--a 256M string\, even though there is plenty of virtual memory.

Thanks for the report. Present in bleed too.

Perl_do_vecset() and Perl_do_vecget() in doop.c start by calculating the required size of the string in bits\, which overflows if the byte length of the string is 2^31/8.

Presumably the code needs re-engineering to do the calculations in bytes+fractions rather than bits.

Hopefully someone will volunteer....

-- This email is confidential\, and now that you have read it you are legally obliged to shoot yourself. Or shoot a lawyer\, if you prefer. If you have received this email in error\, place it in its original wrapping and return for a full refund. By opening this email\, you accept that Elvis lives.

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @mhx

On 2006-12-10\, at 20​:21​:46 +0000\, Dave Mitchell wrote​:

On Sat\, Dec 09\, 2006 at 09​:36​:49PM -0800\, Kyle Cronan wrote​:

#!/usr/bin/perl my $foo = ''; vec($foo\, (1\<\<$ARGV[0])-1\, 8)=1;

Works fine for me when run with the parameter 27\, but fails with "Out of memory!" when it is 28--a 256M string\, even though there is plenty of virtual memory.

Thanks for the report. Present in bleed too.

Perl_do_vecset() and Perl_do_vecget() in doop.c start by calculating the required size of the string in bits\, which overflows if the byte length of the string is 2^31/8.

Presumably the code needs re-engineering to do the calculations in bytes+fractions rather than bits.

Something like the attached patch?

Actually\, I think all those I32's should rather be IV's. then building with 64-bit support would (hopefully) automatically solve the problem that blocks larger than 2GB still cannot be handled. But that would involve a bit more changes.

Marcus

[p5pRT]

From @mhx

doop.c.diff

```diff --- doop.c.orig 2006-12-10 21:38:46.000000000 +0100 +++ doop.c 2006-12-10 22:18:23.000000000 +0100 @@ -726,7 +726,7 @@ Perl_do_vecget(pTHX_ SV *sv, I32 offset, I32 size) { dVAR; - STRLEN srclen, len, uoffset; + STRLEN srclen, len, uoffset, bitoffs = 0; const unsigned char *s = (const unsigned char *) SvPV_const(sv, srclen); UV retnum = 0; @@ -738,13 +738,20 @@ if (SvUTF8(sv)) (void) Perl_sv_utf8_downgrade(aTHX_ sv, TRUE); - uoffset = offset*size; /* turn into bit offset */ - len = (uoffset + size + 7) / 8; /* required number of bytes */ + if (size < 8) { + bitoffs = ((offset%8)*size)%8; + uoffset = offset/(8/size); + } + else if (size > 8) + uoffset = offset*(size/8); + else + uoffset = offset; + + len = uoffset + (bitoffs + size + 7)/8; /* required number of bytes */ if (len > srclen) { if (size <= 8) retnum = 0; else { - uoffset >>= 3; /* turn into byte offset */ if (size == 16) { if (uoffset >= srclen) retnum = 0; @@ -821,9 +828,8 @@ } } else if (size < 8) - retnum = (s[uoffset >> 3] >> (uoffset & 7)) & ((1 << size) - 1); + retnum = (s[uoffset] >> bitoffs) & ((1 << size) - 1); else { - uoffset >>= 3; /* turn into byte offset */ if (size == 8) retnum = s[uoffset]; else if (size == 16) @@ -865,7 +871,7 @@ Perl_do_vecset(pTHX_ SV *sv) { dVAR; - register I32 offset; + register I32 offset, bitoffs = 0; register I32 size; register unsigned char *s; register UV lval; @@ -894,8 +900,14 @@ if (size < 1 || (size & (size-1))) /* size < 1 or not a power of two */ Perl_croak(aTHX_ "Illegal number of bits in vec"); - offset *= size; /* turn into bit offset */ - len = (offset + size + 7) / 8; /* required number of bytes */ + if (size < 8) { + bitoffs = ((offset%8)*size)%8; + offset /= 8/size; + } + else if (size > 8) + offset *= size/8; + + len = offset + (bitoffs + size + 7)/8; /* required number of bytes */ if (len > targlen) { s = (unsigned char*)SvGROW(targ, len + 1); (void)memzero((char *)(s + targlen), len - targlen + 1); @@ -904,14 +916,11 @@ if (size < 8) { mask = (1 << size) - 1; - size = offset & 7; lval &= mask; - offset >>= 3; /* turn into byte offset */ - s[offset] &= ~(mask << size); - s[offset] |= lval << size; + s[offset] &= ~(mask << bitoffs); + s[offset] |= lval << bitoffs; } else { - offset >>= 3; /* turn into byte offset */ if (size == 8) s[offset ] = (U8)( lval & 0xff); else if (size == 16) { ```

[p5pRT]

From @rgarcia

On 10/12/06\, Marcus Holland-Moritz \mhx\-perl@&#8203;gmx\.net wrote​:

Presumably the code needs re-engineering to do the calculations in bytes+fractions rather than bits.

Something like the attached patch?

Thanks\, applied as change #29506.

[p5pRT]

@rgs - Status changed from 'open' to 'resolved'