search

Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.85k stars 527 forks source link

memory leak in threads #8968

Closed p5pRT closed 16 years ago

p5pRT commented 16 years ago

Migrated from rt.perl.org#43711 (status was 'resolved')

Searchable as RT43711$

p5pRT commented 16 years ago

From michael.scondo@phil.stud.uni-erlangen.de

The resident memory usage of the script below grows indefinitely\, and in steps.

Originally I wrote something more complex\, but after creating and destroying a few 100 threads I'll get an "Out of memory". So I tried to track from where the problem came\, with the help of some perlmonks (http​://perlmonks.org/?node_id=625622)

Undefining of variables at several points seems to slow down the memory growth\, as well as a longer sleep in the line select undef\,undef\,undef\,1;

------------ use threads; use threads​::shared;

our $MAX ||= 100;

$|++;

while( 1 ) { my @​threads = ();   push @​threads\, threads->create( sub{ undef @​_;undef;} ) while @​threads \< $MAX;

  for (@​threads) {   $_->join;   undef $_;   }   undef @​threads;   select undef\,undef\,undef\,1; } -----------

Here's an example ouput of "forever{ cat /proc/PID/statm;sleep 1;}"

22091 13306 394 289 0 21382 0 22091 13306 394 289 0 21382 0 22091 13306 394 289 0 21382 0 75365 13332 394 289 0 74656 0 22091 13306 394 289 0 21382 0 22091 13306 394 289 0 21382 0 151178 13369 394 289 0 150469 0 22091 13306 394 289 0 21382 0 22091 13306 394 289 0 21382 0 22091 13306 394 289 0 21382 0 97904 13343 394 289 0 97195 0 22347 13529 394 289 0 21638 0 22347 13529 394 289 0 21638 0 182169 13607 394 289 0 181460 0 22347 13529 394 289 0 21638 0 22347 13529 394 289 0 21638 0 22347 13529 394 289 0 21638 0 92013 13563 394 289 0 91304 0 22347 13529 394 289 0 21638 0 22347 13529 394 289 0 21638 0 194463 13613 394 289 0 193754 0 22347 13529 394 289 0 21638 0 22347 13529 394 289 0 21638 0 22347 13590 394 289 0 21638 0 26445 13592 394 289 0 25736 0 22347 13584 394 289 0 21638 0 22347 13584 394 289 0 21638 0 53082 13599 394 289 0 52373 0 22347 13584 394 289 0 21638 0 22347 13584 394 289 0 21638 0 149385 13646 394 289 0 148676 0

-------- I finally wrote a script with the slowest memory growth\, I was able to produce.

#!/home/micha/localperl/bin/perl5.9.5 -w use threads; $|++;

while( 1 ) {   my $t = threads->create( sub{ undef @​_;undef;} );

  my $f = $t->join();   undef $t;   undef $f;   threads->yield();   select undef\,undef\,undef\,0.2; }

------ forever{ cat /proc/PID/statm;sleep 1;}

22015 13247 394 289 0 21306 0 22015 13247 394 289 0 21306 0 22015 13247 394 289 0 21306 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 108054 13288 394 289 0 107345 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13246 394 289 0 21287 0 21996 13247 394 289 0 21287 0 21996 13247 394 289 0 21287 0 21996 13247 394 289 0 21287 0 21996 13247 394 289 0 21287 0 21996 13247 394 289 0 21287 0 22252 13259 394 289 0 21543 0 22252 13259 394 289 0 21543 0 22252 13259 394 289 0 21543 0 22252 13259 394 289 0 21543 0 22252 13259 394 289 0 21543 0 162094 17990 394 289 0 161385 0 29164 20199 394 289 0 28455 0 (!) 29164 20199 394 289 0 28455 0 29164 20199 394 289 0 28455 0 29164 20199 394 289 0 28455 0 29164 20199 394 289 0 28455 0 29164 20195 394 289 0 28455 0

Thanks\, Michael Scondo

Perl Info ``` Flags: category=core severity=medium Site configuration information for perl 5.9.5: Configured by micha at Mon Jul 9 18:18:15 UTC 2007. Summary of my perl5 (revision 5 version 9 subversion 5) configuration: Platform: osname=linux, osvers=2.6.18-32-splash, archname=i686-linux-thread-multi uname='linux laptop 2.6.18-32-splash #3 smp preempt mon feb 5 14:12:53 local time zone must be set--s i686 amd turion(tm) 64 x2 mobile technology tl-60 authenticamd gnulinux ' config_args='-Dprefix=/home/micha/localperl -Dusethreads -des -Dusedevel' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='4.1.1 (Gentoo 4.1.1-r1)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.4.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.4' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib' Locally applied patches: @INC for perl 5.9.5: /home/micha/localperl/lib/5.9.5/i686-linux-thread-multi /home/micha/localperl/lib/5.9.5 /home/micha/localperl/lib/site_perl/5.9.5/i686-linux-thread-multi /home/micha/localperl/lib/site_perl/5.9.5 . Environment for perl 5.9.5: HOME=/home/micha LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH=/usr/lib/xorg:/usr/lib/fglrx LOGDIR (unset) PATH=/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.1.1:/usr/i686-pc-linux-gnu/gcc-bin/4.1.1:/opt/ati/bin:/opt/blackdown-jdk-1.4.2.03/bin:/opt/blackdown-jdk-1.4.2.03/jre/bin:/usr/kde/3.5/bin:/usr/qt/3/bin:/usr/games/bin:/home/micha/prog/perl/scripts PERL_BADLANG (unset) SHELL=/bin/bash ```
p5pRT commented 16 years ago

From @iabyn

On Tue\, Jul 10\, 2007 at 05​:09​:11AM -0700\, Michael Scondo wrote​:

# New Ticket Created by Michael Scondo # Please include the string​: [perl #43711] # in the subject line of all future correspondence about this issue. # \<URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=43711 >

This is a bug report for perl from michael.scondo@​phil.stud.uni-erlangen.de generated with the help of perlbug 1.35 running under perl 5.9.5.

----------------------------------------------------------------- [Please enter your report here]

The resident memory usage of the script below grows indefinitely\, and in steps.

Originally I wrote something more complex\, but after creating and destroying a few 100 threads I'll get an "Out of memory". So I tried to track from where the problem came\, with the help of some perlmonks (http​://perlmonks.org/?node_id=625622)

Undefining of variables at several points seems to slow down the memory growth\, as well as a longer sleep in the line select undef\,undef\,undef\,1;

------------ use threads; use threads​::shared;

our $MAX ||= 100;

$|++;

while( 1 ) { my @​threads = (); push @​threads\, threads->create( sub{ undef @​_;undef;} ) while @​threads \< $MAX;

for \(@&#8203;threads\) \{
        $\_\->join;
        undef $\_;
        \}
    undef @&#8203;threads;
    select undef\,undef\,undef\,1;

}

I can't see this leak in bleedperl\, so I presume it's been fixed there.

-- Red sky at night - gerroff my land! Red sky at morning - gerroff my land!   -- old farmers' sayings #14

p5pRT commented 16 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 16 years ago

@rgs - Status changed from 'open' to 'resolved'

p5pRT commented 16 years ago

From @jdhedden

Dave Mitchell wrote​:

I can't see this leak in bleedperl\, so I presume it's been fixed there.

Using bleadperl under Cygwin\, the following shows a continuous increase in memory usage as reported by the Windows Task Manager​:

  use threads;   while (1) {   threads->create(sub{})->join();   threads->yield();   }

Even if scoping is applied\, the memory usage is not recovered​:

  use threads;   {   for (1..1000) {   threads->create(sub{})->join();   threads->yield();   }   }   print("Check memory now\n");   sleep(10);

Unfortunately\, I can't tell if this is an issue with Cygwin/Windows or with Perl.

p5pRT commented 16 years ago

From @iabyn

On Wed\, Jul 11\, 2007 at 09​:22​:50AM -0400\, Jerry D. Hedden wrote​:

Dave Mitchell wrote​:

I can't see this leak in bleedperl\, so I presume it's been fixed there.

Using bleadperl under Cygwin\, the following shows a continuous increase in memory usage as reported by the Windows Task Manager​:

use threads; while (1) { threads->create(sub{})->join(); threads->yield(); }

With linux\, I see constant memory usage as reported by top.

-- Red sky at night - gerroff my land! Red sky at morning - gerroff my land!   -- old farmers' sayings #14

p5pRT commented 16 years ago

From guest@guest.guest.xxxxxxxx

On Wed Jul 11 06​:57​:58 2007\, davem wrote​:

On Wed\, Jul 11\, 2007 at 09​:22​:50AM -0400\, Jerry D. Hedden wrote​:

Dave Mitchell wrote​:

I can't see this leak in bleedperl\, so I presume it's been fixed there.

Using bleadperl under Cygwin\, the following shows a continuous increase in memory usage as reported by the Windows Task Manager​:

use threads; while (1) { threads->create(sub{})->join(); threads->yield(); }

With linux\, I see constant memory usage as reported by top.

I downloaded bleadperl and the leak still exists here. (Hopefully I did the right thing in order to get bleadperl\, I ran rsync -av rsync​://ftp.linux.activestate.com/perl-current/ /usr/src/bleadperl)

I'm running linux\, 2.6.18 32bit on a dual core (turion x2).

I wrote the same program in C\, there are no leaks there.

p5pRT commented 16 years ago

From michael.scondo@phil.stud.uni-erlangen.de

On Wednesday 11 July 2007 12​:03\, Rafael Garcia-Suarez via RT wrote​:

According to our records\, your request regarding "memory leak in threads" has been resolved.

If you have any further questions or concerns\, please respond to this message.

Thanks for your effort\, however\, I downloaded bleadperl and the leak still exists. (Hopefully I did the right thing in order to get bleadperl\, I ran rsync -av rsync​://ftp.linux.activestate.com/perl-current/ /usr/src/bleadperl)

Regarding to http​://perlmonks.org/?node_id=625622\, the leak seems to exist only on linux/unix.

There's also still the probability the leak shows up only with smp.

perl5.9.5 -V

Summary of my perl5 (revision 5 version 9 subversion 5 patch 31587) configuration​:   Platform​:   osname=linux\, osvers=2.6.18-32-splash\, archname=i686-linux-thread-multi   uname='linux laptop 2.6.18-32-splash #3 smp preempt mon feb 5 14​:12​:53 local time zone must be set--s i686 amd turion(tm) 64 x2 mobile technology tl-60 authenticamd gnulinux '   config_args='-Dprefix=/home/micha/localperl -Dusethreads -des -Dusedevel'   hint=previous\, useposix=true\, d_sigaction=define   useithreads=define\, usemultiplicity=define   useperlio=define\, d_sfio=undef\, uselargefiles=define\, usesocks=undef   use64bitint=undef\, use64bitall=undef\, uselongdouble=undef   usemymalloc=n\, bincompat5005=undef   Compiler​:   cc='cc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\,   optimize='-O2'\,  
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'   ccversion=''\, gccversion='4.1.1 (Gentoo 4.1.1-r1)'\, gccosandvers=''   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=1234   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12   ivtype='long'\, ivsize=4\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8   alignbytes=4\, prototype=define   Linker and Libraries​:   ld='cc'\, ldflags =' -L/usr/local/lib'   libpth=/usr/local/lib /lib /usr/lib   libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc   perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc   libc=/lib/libc-2.4.so\, so=so\, useshrplib=false\, libperl=libperl.a   gnulibc_version='2.4'   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E'   cccdlflags='-fPIC'\, lddlflags='-shared -O2 -L/usr/local/lib'

Characteristics of this binary (from libperl)​:   Compile-time options​: MULTIPLICITY PERL_DONT_CREATE_GVSV   PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP   USE_ITHREADS USE_LARGE_FILES USE_PERLIO   USE_REENTRANT_API   Locally applied patches​:   DEVEL   Built under linux   Compiled at Jul 11 2007 14​:42​:20   @​INC​:   /home/micha/localperl/lib/5.9.5/i686-linux-thread-multi   /home/micha/localperl/lib/5.9.5   /home/micha/localperl/lib/site_perl/5.9.5/i686-linux-thread-multi   /home/micha/localperl/lib/site_perl/5.9.5   .