Closed GoogleCodeExporter closed 9 years ago
I tried to re produce the above problem, and found that above mentioned finding
is not correct.
Initially you may found "gc.collect()" is returning some value after removing
all the object but if you wait for some time then again you run "gc.collect()"
then it will return 0.
This is because of python takes some time to remove unreferenced object.
Your findings about memory uses through pmap is expected because during the
operation some of the dynamic library gets loaded in memory. These are one time
loaded libraries so, this will not cause any problem with long running programs.
Please let us know if you still have any concern about memory leak.
Thanks,
Rahul Priyadarshi
Original comment by rahul.pr...@in.ibm.com
on 18 Jun 2012 at 8:26
Hi Rahul,
Thanks for your answer
I still can't explain why memory size is increasing even if code seems
to be closing objects, the test below iterates 30 times executing 3
queries each one with its own connection and cursor and memory is
bigger each time a query is added. is there some kind of cache in the
middle? Enabling garbage collector debugging will show collectable
objects but memory and number of objects still grow.
Which are the system requirements in terms of memory to use ibm_db_dbi?
gc: collectable <Connection 0x7fffee461b10>
gc: collectable <dict 0x7b6890>
gc: collectable <list 0x7fffee45e6c8>
gc: collectable <Cursor 0x7fffee461b50>
gc: collectable <dict 0x7b4e50>
gc: collectable <list 0x7fffee45e5f0>
reference: http://code.google.com/p/ibm-db/issues/detail?id=114
#!/usr/bin/python
import os
import time
import ibm_db_dbi
cmd = "pmap %d | gawk '{ if ($0 ~ /Total/) print
substr($3,1,length($3) - 1)}'" % os.getpid()
print "ini mem %d " % int(os.popen(cmd).read())
for i in (range(1,30)):
dbh = ibm_db_dbi.connect("db2")
cur = dbh.cursor();cur.execute('SELECT col1 FROM T1);rs = cur.fetchall()
cur.close();del cur; cur = None;del rs; rs = None
dbh.close();del dbh; dbh = None;
dbh = ibm_db_dbi.connect("db2")
cur = dbh.cursor();cur.execute('SELECT * FROM T2 LIMIT 100');rs =
cur.fetchall()
cur.close();del cur; cur = None;del rs; rs = None
dbh.close();del dbh; dbh = None;
dbh = ibm_db_dbi.connect("db2")
cur = dbh.cursor();cur.execute('SELECT * FROM T3');rs = cur.fetchall()
cur.close();del cur; cur = None;del rs; rs = None
dbh.close();del dbh; dbh = None;
t=60
print "sleeping %d seconds" % t
time.sleep(60)
print "[DEBUG] memory %d"% int(os.popen(cmd).read())
Original comment by south.mi...@gmail.com
on 19 Jun 2012 at 7:14
I am taking it up for bigger loop that should run at least 20 to 30 min, if
memory uses will increase continuously then definitely there will be a memory
leak. And then after I will start looking the code to find the memory leak.
Will get back to you very soon after my findings.
Thanks,
Rahul Priyadarshi
Original comment by rahul.pr...@in.ibm.com
on 20 Jun 2012 at 7:01
Thanks Rahul, let me know if you need any tests on my side.
Original comment by south.mi...@gmail.com
on 21 Jun 2012 at 1:57
I tried with a long running test case and found that there is gradually
increase in memory but it is in very small amount
I ran the same program with valgrind to detect the memory leak but this tool
dosn't show anything which differentiate the memory leak.
The amount in increase in memory is very small so, we stuck here.
If you have any program in your environment which shows some significant
gradually increase in memory, then please run that program with valgrind and
send us the output. This may help us to find the memory leak.
Original comment by rahul.pr...@in.ibm.com
on 22 Jun 2012 at 10:30
Hi Rahul,
See attached the valgrind output for these command lines
valgrind --tool=memcheck --suppressions=valgrind-python.supp
--log-file=leak-valgrind-yes.log --leak-check=yes python -E -tt
test.py
valgrind --tool=memcheck --suppressions=valgrind-python.supp
--show-reachable=yes --log-file=leak-valgrind-yes-reachable.log
--leak-check=yes python -E -tt test.py
test.py is the real app and runs for 19 loops and exits, at prod it
runs continuously but as far as I've seen it behaves the same after a
24+ hs running.
Original comment by south.mi...@gmail.com
on 22 Jun 2012 at 5:09
I am not able to find the attachment. Please try to attach it again.
Original comment by rahul.pr...@in.ibm.com
on 28 Jun 2012 at 7:34
Here it goes again
Original comment by south.mi...@gmail.com
on 28 Jun 2012 at 1:06
Attachments:
I looked into the output file, but it only contain the possible memory leak due
to "PyMalloc". So it doesn't help us more.
If possible then try to run your script on "--without-pymalloc" configured
python and send us the valgrind output file to us.
Original comment by rahul.pr...@in.ibm.com
on 2 Jul 2012 at 9:32
ok, will try to compile python and test.
Original comment by south.mi...@gmail.com
on 3 Jul 2012 at 4:08
[deleted comment]
Thanks Rahul. JOOC, is issue#120 related to #114?
Original comment by south.mi...@gmail.com
on 22 Oct 2012 at 5:37
Sorry, issue#120 is not related with 114. Please try the new version and let us
know how it works for you.
Original comment by rahul.pr...@in.ibm.com
on 23 Oct 2012 at 7:26
ok, #120 looks similar to what I had on #114 so it might be fixed
also. Will try and let you know.
Original comment by south.mi...@gmail.com
on 23 Oct 2012 at 8:33
Hi,
I tested the new version and looks good. I had some warnings at
installation, see attached log.
Original comment by south.mi...@gmail.com
on 24 Oct 2012 at 3:56
Looks like you have missed to attach the log file.
Original comment by rahul.pr...@in.ibm.com
on 25 Oct 2012 at 6:25
It might be stripped down by mail server, sending it again compressed.
Original comment by south.mi...@gmail.com
on 25 Oct 2012 at 2:12
It looks like mail server again removed the attachment.
Original comment by rahul.pr...@in.ibm.com
on 2 Nov 2012 at 6:55
Plain text now :p
amenabar:~ # easy_install --upgrade ibm-db
/usr/lib64/python2.6/site-packages/setuptools/package_index.py:7:
DeprecationWarning: the md5 module is deprecated; use hashlib instead
from md5 import md5
Searching for ibm-db
Reading http://pypi.python.org/simple/ibm-db/
Reading http://code.google.com/p/ibm-db/downloads/list
Best match: ibm-db 2.0.0
Downloading
http://pypi.python.org/packages/source/i/ibm_db/ibm_db-2.0.0.tar.gz#md5=709c576c
0ec2379ca15049f5c861beb1
Processing ibm_db-2.0.0.tar.gz
Running ibm_db-2.0.0/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-XGKZqD/ibm_db-2.0.0/egg-dist-tmp-l5a0V0
Detected 64-bit Python
In file included from ibm_db.c:27:
ibm_db.h:31:1: warning: "PyObject_CheckBuffer" redefined
In file included from /usr/include/python2.6/Python.h:132,
from ibm_db.c:26:
/usr/include/python2.6/abstract.h:534:1: warning: this is the location
of the previous definition
zip_safe flag not set; analyzing archive contents...
testfunctions: module MAY be using inspect.stack
tests: module references __file__
tests.test_144_BindParamInsertStmtPARAM_FILE: module references __file__
tests.test_000_PrepareDb: module references __file__
Removing ibm-db 1.0.6 from easy-install.pth file
Adding ibm-db 2.0.0 to easy-install.pth file
Installed
/usr/local/lib64/python2.6/site-packages/ibm_db-2.0.0-py2.6-linux-x86_64.egg
Processing dependencies for ibm-db
Finished processing dependencies for ibm-db
Original comment by south.mi...@gmail.com
on 2 Nov 2012 at 1:39
The first warning "DeprecationWarning: the md5 module is deprecated; use
hashlib instead" is coming through setuptools, so they may resolve it in coming
coming release.
Python have made some changes in python-2.6.x and Python-2.7.x to make it
compatible with Python-3.x, Many of these are wrapper kind of thing which they
defined in header file. we supports 2.5.x and above and treat all Python2.x
version as same that's why PyObject_CheckBuffer get redefined. so, we can also
ignore this warning.
Thanks,
Rahul
Original comment by rahul.pr...@in.ibm.com
on 10 Dec 2012 at 8:57
Original comment by rahul.pr...@in.ibm.com
on 1 Mar 2013 at 10:09
Original issue reported on code.google.com by
south.mi...@gmail.com
on 15 Jun 2012 at 5:40