Closed p5pRT closed 15 years ago
p5pRT
commented
16 years ago Under perl-5.10.0 the following simple program leaks memory\, while under perl-5.8.8 it does not. DBI module version is the same under both perl versions (1.605). The leak seems to be not specific to DBD driver\, as it also happens with mysql and oracle drivers.
use DBI; my $dbh=DBI->connect('DBI:NullP:'); while(1) { my $q=$dbh->prepare(q{select 1 from dual}); }
p5pRT
commented
16 years ago On Mon Jul 14 00:40:45 2008\, lav@yar.ru wrote:
This is a bug report for perl from lav@yar.ru\, generated with the help of perlbug 1.36 running under perl 5.10.0.
----------------------------------------------------------------- [Please enter your report here] Under perl-5.10.0 the following simple program leaks memory\, while under perl-5.8.8 it does not. DBI module version is the same under both perl versions (1.605). The leak seems to be not specific to DBD driver\, as it also happens with mysql and oracle drivers.
use DBI; my $dbh=DBI->connect('DBI:NullP:'); while(1) { my $q=$dbh->prepare(q{select 1 from dual}); }
DBI does some sneaky things with the internals\, in the pursuit of raw performance.
The internals were heavily reworked during the development track up to 5.10\, and in the process DBI was dinged a couple of times by changes in assumptions. As such\, and given the absence of response so far (in spite of a delightfully short test snippet -- thank you)\, I recommend you post your report to the dbi-users mailing list.
I've cc'ed Tim Bunce on this reply. It just may turn out that Tim has a easy solution regardless\, and was not aware of the bug report.
Thanks\, David Landgren
p5pRT
commented
16 years ago The RT System itself - Status changed from 'new' to 'open'
p5pRT
commented
16 years ago Umm. I've reproduced it using 5.10 but not 5.8.6.
p5pRT
commented
16 years ago I have reproduced this bug with DBI versions 1.604 and 1.607 with Perl 5.10 on Linux (using the DBD::Pg driver\, as well). It looks to be approximately 72 bytes leaked every time we call ->prepare (specifically from somewhere in DBI::_new_handle). Unfortunately our application has some long-running processes which prepare a lot of database statements.
perl -V information:
Platform: osname=linux\, osvers=2.6.9-42.0.3.elsmp\, archname=i386-linux-thread-multi uname='linux cent4-rpmbuild.dev.airwave.com 2.6.9-42.0.3.elsmp #1 smp fri oct 6 06:21:39 cdt 2006 i686 i686 i386 gnulinux ' config_args='-des -Doptimize=-O2 -g -march=i386 -mcpu=i686 -Dversion=5.10.0 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat\, Inc. -Dinstallprefix=/opt/airwave -Dprefix=/opt/airwave -Dprivlib=/opt/airwave/lib/perl5/5.10.0 -Dsitelib=/opt/airwave/local/lib/perl5/site_perl/5.10.0 -Dvendorlib=/opt/airwave/lib/perl5/vendor_perl/5.10.0 -Darchlib=/opt/airwave/lib/perl5/5.10.0/i386-linux-thread-multi -Dsitearch=/opt/airwave/local/lib/perl5/site_perl/5.10.0/i386-linux-thread-multi -Dvendorarch=/opt/airwave/lib/perl5/vendor_perl/5.10.0/i386-linux-thread-multi -Darchname=i386-linux-thread-multi -Dvendorprefix=/opt/airwave -Dsiteprefix=/opt/airwave/local -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/opt/airwave/bin' hint=recommended\, useposix=true\, d_sigaction=define useithreads=define\, usemultiplicity=define useperlio=define\, d_sfio=undef\, uselargefiles=define\, usesocks=undef use64bitint=undef\, use64bitall=undef\, uselongdouble=undef usemymalloc=n\, bincompat5005=undef Compiler: cc='gcc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm'\, optimize='-O2 -g -march=i386 -mcpu=i686'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm' ccversion=''\, gccversion='3.4.6 20060404 (Red Hat 3.4.6-9)'\, gccosandvers='' intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=1234 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12 ivtype='long'\, ivsize=4\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8 alignbytes=4\, prototype=define Linker and Libraries: ld='gcc'\, ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.3.4.so\, so=so\, useshrplib=true\, libperl=libperl.so gnulibc_version='2.3.4' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E -Wl\,-rpath\,/opt/airwave/lib/perl5/5.10.0/i386-linux-thread-multi/CORE' cccdlflags='-fPIC'\, lddlflags='-shared -O2 -g -march=i386 -mcpu=i686 -L/usr/local/lib'
Characteristics of this binary (from libperl): Compile-time options: DEBUGGING MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_TRACK_MEMPOOL USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Built under linux Compiled at Jun 30 2008 15:26:47
p5pRT
commented
16 years ago Citeren "lav@yar.ru (via RT)" \perlbug\-followup@​perl\.org:
# New Ticket Created by lav@yar.ru # Please include the string: [perl #56908] # in the subject line of all future correspondence about this issue. # \<URL: http://rt.perl.org/rt3/Ticket/Display.html?id=56908 >
This is a bug report for perl from lav@yar.ru\, generated with the help of perlbug 1.36 running under perl 5.10.0.
----------------------------------------------------------------- [Please enter your report here] Under perl-5.10.0 the following simple program leaks memory\, while under perl-5.8.8 it does not. DBI module version is the same under both perl versions (1.605). The leak seems to be not specific to DBD driver\, as it also happens with mysql and oracle drivers.
use DBI; my $dbh=DBI->connect('DBI:NullP:'); while(1) { my $q=$dbh->prepare(q{select 1 from dual}); }
I've done a binary search on it with these results: (with DBI-1.607)
----Program---- #!/usr/bin/perl -l
use DBI;
my $dbh=DBI->connect('DBI:NullP:'); my $i = 0; my $mem1 = qx/ps u $$/;
while ($i++ \< 100_000) { my $q=$dbh->prepare(q{select 1 from dual}); }
my $mem2 = qx/ps u $$/;
my ($vsz1) = $mem1 =~ m/^perl\s+\d+\s+[0-9.]+\s+[0-9.]+\s+(\d+)/m; my ($vsz2) = $mem2 =~ m/^perl\s+\d+\s+[0-9.]+\s+[0-9.]+\s+(\d+)/m; if ($vsz1 + 1000 \< $vsz2) { print "not ok"; } else { print "ok"; }
----Output of .../pUmCekJ/perl-5.9.2@26529/bin/perl---- ok
----EOF ($?='0')---- ----Output of .../pn6SMVj/perl-5.9.2@26530/bin/perl---- not ok
----EOF ($?='0')----
The output with blead is the same. (meaning: it hasn't changed since)
http://public.activestate.com/cgi-bin/perlbrowse/p/26530 Change 26530 by nicholas@nicholas-saigo on 2005/12/30 01:08:46
RMAGIC on symbol tables is bad\, m'kay. Allow hashes (and therefore all symbol tables) to store the backreference array in the hv_aux structure\, and thereby undo the performance damage of 24966\, which resulted in 60% of all hash lookups trying to mg_find tiehash magic.
I'll leave it up to someone else to figure out if it's a bug in the
CORE or a bug in DBI...
Kind regards\,
Bram
p5pRT
commented
16 years ago On Tue\, Aug 05\, 2008 at 10:00:12PM +0200\, Bram wrote:
Citeren "lav@yar.ru (via RT)" \perlbug\-followup@​perl\.org:
# \<URL: http://rt.perl.org/rt3/Ticket/Display.html?id=56908 >
Under perl-5.10.0 the following simple program leaks memory\, while under perl-5.8.8 it does not. DBI module version is the same under both perl versions (1.605). The leak seems to be not specific to DBD driver\, as it also happens with mysql and oracle drivers.
use DBI; my $dbh=DBI->connect('DBI:NullP:'); while(1) { my $q=$dbh->prepare(q{select 1 from dual}); }
I've done a binary search on it with these results: (with DBI-1.607)
The output with blead is the same. (meaning: it hasn't changed since)
http://public.activestate.com/cgi-bin/perlbrowse/p/26530 Change 26530 by nicholas@nicholas-saigo on 2005/12/30 01:08:46
RMAGIC on symbol tables is bad\, m'kay\. Allow hashes \(and therefore all symbol tables\) to store the backreference array in the hv\_aux structure\, and thereby undo the performance damage of 24966\, which resulted in 60% of all hash lookups trying to mg\_find tiehash magic\.I'll leave it up to someone else to figure out if it's a bug in the CORE or a bug in DBI...
Having looked at the patch\, and felt my eyes glazing over\, I'm going to point the finger at Nick first :)
Nick\, in case it helps... DBI uses weakrefs for only one thing. Whenever a new handle is created the parent handle's ChildHandles attribute (an array ref) gets a weak ref to the new child pushed onto it. This is the code that does that (search for weak in DBI.xs):
AV *av; /* add weakref to new (outer) handle into parents ChildHandles array */ tmp_svp = hv_fetch((HV*)SvRV(parent)\, "ChildHandles"\, 12\, 1); if (!SvROK(*tmp_svp)) { SV *ChildHandles_rvav = newRV_noinc((SV*)newAV()); sv_setsv(*tmp_svp\, ChildHandles_rvav); sv_free(ChildHandles_rvav); } av = (AV*)SvRV(*tmp_svp); av_push(av\, (SV*)sv_rvweaken(newRV((SV*)SvRV(orv)))); if (av_len(av) % 120 == 0) { /* time to do some housekeeping to remove dead handles */ I32 i = av_len(av); /* 0 = 1 element */ while (i-- >= 0) { SV *sv = av_shift(av); if (SvOK(sv)) av_push(av\, sv); else sv_free(sv); } }
Let me know if that's doing anything wrong\, or sub-optimally. (I can see a few cleanups but nothing fundamentally wrong.)
One thing that's unusual is that these are weakrefs to *tied* hashes. Perhaps that's an odd case not handled by the patch.
Tim.
p5pRT
commented
16 years ago On Wed\, Aug 06\, 2008 at 12:12:44PM +0100\, Tim Bunce wrote:
Nick\, in case it helps... DBI uses weakrefs for only one thing. Whenever a new handle is created the parent handle's ChildHandles attribute (an array ref) gets a weak ref to the new child pushed onto it. This is the code that does that (search for weak in DBI.xs):
One thing that's unusual is that these are weakrefs to *tied* hashes. Perhaps that's an odd case not handled by the patch.
Well\, it should have still worked...
I doubt that I'll get any time to look at this this side of September.
Nicholas Clark
p5pRT
commented
15 years ago fixed in bleed by change 34209\, also integrated into 5.10.1
p5pRT
commented
15 years ago @iabyn - Status changed from 'open' to 'resolved'
Migrated from rt.perl.org#56908 (status was 'resolved')
Searchable as RT56908$