Memory leak oddities with various versions of Perl (magic related?)

[p5pRT]

Migrated from rt.perl.org#122791 (status was 'rejected')

Searchable as RT122791$

[p5pRT]

From @wolfsage

Created by wolfsage@gmail.com

The attached script displays some bizarre memory characteristics on different versions of Perl.

With perl-5.20.0\, the loop at the bottom will leak memory if the Dumper($content) call is commented out.

If you uncomment that line\, there's no leak. This behaviour is mirrored on perl-5.10.1.

With bleadperl\, it leaks either way.

With perl-5.14.1\, it will not leak\, but without the Dumper Perl uses ~58M resident; with the Dumper it uses upwards of 130M resident. (Possibly a separate\, unrelated issue (or not an issue at all?).

I originally thought this was UTF8 related\, but I just tested with replacing the smiley face with a normal character\, and still see the weird behaviour.

This *could* be a bug with XML​::Simple and/or XML​::SAX\, but the different behaviours on different Perls is suspect.

You'll need to install XML​::Simple and Variable​::Magic. To install Variable​::Magic on blead\, you'll need to skip tests for now or apply sprouts patch here​: https://rt.cpan.org/Public/Ticket/Attachment/1407023/746879/open_BUc83oif.txt

-- Matthew Horsfall (alh)

Perl Info

``` Flags: category=core severity=medium Site configuration information for perl 5.14.2: Configured by Debian Project at Tue Feb 4 23:09:53 UTC 2014. Summary of my perl5 (revision 5 version 14 subversion 2) configuration: Platform: osname=linux, osvers=2.6.42-37-generic, archname=x86_64-linux-gnu-thread-multi uname='linux panlong 2.6.42-37-generic #58-ubuntu smp thu jan 24 15:28:10 utc 2013 x86_64 x86_64 x86_64 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.14 -Darchlib=/usr/lib/perl/5.14 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.14.2 -Dsitearch=/usr/local/lib/perl/5.14.2 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.14.2 -des' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.6.3', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=, so=so, useshrplib=true, libperl=libperl.so.5.14.2 gnulibc_version='2.15' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector' Locally applied patches: @INC for perl 5.14.2: /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl . Environment for perl 5.14.2: HOME=/home/mhorsfall LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/mhorsfall/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games PERLDOC=-oman PERL_BADLANG (unset) SHELL=/bin/bash ```

[p5pRT]

From @wolfsage

why.pl

[p5pRT]

From @wolfsage

I originally thought this was UTF8 related\, but I just tested with replacing the smiley face with a normal character\, and still see the weird behaviour.

And if I remove the Variable​::Magic\, I still get the weird behaviour....

-- Matthew Horsfall (alh)

[p5pRT]

From @iabyn

On Tue\, Sep 16\, 2014 at 11​:00​:14AM -0400\, Matthew Horsfall (alh) wrote​:

I originally thought this was UTF8 related\, but I just tested with replacing the smiley face with a normal character\, and still see the weird behaviour.

And if I remove the Variable​::Magic\, I still get the weird behaviour....

I've had a play with it with blead (minus any Variable​::Magic-related lines).

VM usage gradually grows whether or not I call Data​::Dumper.

I then looked at it under valgrind. Running it for 1\,2\,3 and 4 iterations all showed the same final memory usage. Perl doesn't normally free up SV arenas before exit\, which would account for non-zero memory usage at the end. Doing the valgrind with PERL_DESTRUCT_LEVEL=2 (which forces perl to free up everyhing it knows about before exit) gives no significant leaked memory.

So think it most likely that its just malloc() fragmentation.

Decreasing the size of the data structure below about 33500 elements stopped the VM growth for some reason.

-- That he said that that that that is is is debatable\, is debatable.

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @wolfsage

On Sat\, Sep 20\, 2014 at 5​:44 PM\, Dave Mitchell \davem@​iabyn\.com wrote​:

So think it most likely that its just malloc() fragmentation.

Huh. So is this "not a bug"\, just unfortunate behaviour?

-- Matthew Horsfall (alh)

[p5pRT]

From @iabyn

On Mon\, Sep 22\, 2014 at 12​:24​:10PM -0400\, Matthew Horsfall (alh) wrote​:

On Sat\, Sep 20\, 2014 at 5​:44 PM\, Dave Mitchell \davem@​iabyn\.com wrote​:

So think it most likely that its just malloc() fragmentation.

Huh. So is this "not a bug"\, just unfortunate behaviour?

I guess so.

-- This email is confidential\, and now that you have read it you are legally obliged to shoot yourself. Or shoot a lawyer\, if you prefer. If you have received this email in error\, place it in its original wrapping and return for a full refund. By opening this email\, you accept that Elvis lives.

[p5pRT]

@iabyn - Status changed from 'open' to 'rejected'