Memory leak in ibm_db_dbi

[GoogleCodeExporter]

ibm_db_dbi wrapper does not free memory after deleting objects (the issue was 
present on 1.0.5 too)

What steps will reproduce the problem?

test:~ # python
Python 2.6 (r26:66714, May  5 2010, 14:02:39) 
[GCC 4.3.4 [gcc-4_3-branch revision 152973]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gc
>>> import ibm_db_dbi
>>> len(gc.get_objects())
7823
>>> dbh = ibm_db_dbi.connect("db2")
>>> cur = dbh.cursor();cur.execute('SELECT col1 FROM test LIMIT 10');rs = 
cur.fetchall()
True
>>> len(gc.get_objects())
7876
>>> cur.close();del cur; cur = None; dbh.close(); del dbh; dbh = None; del rs
True
>>> len(gc.get_objects())
7864
>>> gc.collect()
6
>>> len(gc.get_objects())
7858

Memory usage with pmap $(pgrep '^python$') | grep 'Total'

# before connecting
Total:           116992K  16928K   9770K   7240K      0K

#after closing and deleting objects
Total:           168776K  22168K  12213K   8308K      0K

What is the expected output? What do you see instead?
Python should return to initial memory usage ( or close)

What version of the product are you using? On what operating system?
ibm_db-1.0.6-py2.6-linux-x86_64
Suse enterprise 11.1 SP1 X86_64

Please provide any additional information below.
Problem gets worse if python process is continuously running

Original issue reported on code.google.com by south.mi...@gmail.com on 15 Jun 2012 at 5:40

[GoogleCodeExporter]

I tried to re produce the above problem, and found that above mentioned finding 
is not correct.

Initially you may found "gc.collect()" is returning some value after removing 
all the object but if you wait for some time then again you run "gc.collect()" 
then it will return 0.
This is because of python takes some time to remove unreferenced object.

Your findings about memory uses through pmap is expected because during the 
operation some of the dynamic library gets loaded in memory. These are one time 
loaded libraries so, this will not cause any problem with long running programs.

Please let us know if you still have any concern about memory leak.

Thanks,
Rahul Priyadarshi

Original comment by rahul.pr...@in.ibm.com on 18 Jun 2012 at 8:26

[GoogleCodeExporter]

Hi Rahul,
Thanks for your answer

I still can't explain why memory size is increasing even if code seems
to be closing objects, the test below iterates 30 times executing 3
queries each one with its own connection and cursor and memory is
bigger each time a query is added. is there some kind of cache in the
middle? Enabling garbage collector debugging will show collectable
objects but memory and number of objects still grow.
Which are the system requirements in terms of memory to use ibm_db_dbi?

gc: collectable <Connection 0x7fffee461b10>
gc: collectable <dict 0x7b6890>
gc: collectable <list 0x7fffee45e6c8>
gc: collectable <Cursor 0x7fffee461b50>
gc: collectable <dict 0x7b4e50>
gc: collectable <list 0x7fffee45e5f0>

reference: http://code.google.com/p/ibm-db/issues/detail?id=114

#!/usr/bin/python

import os
import time
import ibm_db_dbi

cmd = "pmap %d | gawk '{ if ($0 ~ /Total/) print
substr($3,1,length($3) - 1)}'" % os.getpid()

print "ini mem %d " % int(os.popen(cmd).read())
for i in (range(1,30)):
    dbh = ibm_db_dbi.connect("db2")
    cur = dbh.cursor();cur.execute('SELECT col1 FROM T1);rs = cur.fetchall()
    cur.close();del cur; cur = None;del rs; rs = None
    dbh.close();del dbh; dbh = None;
    dbh = ibm_db_dbi.connect("db2")
    cur = dbh.cursor();cur.execute('SELECT * FROM T2 LIMIT 100');rs =
cur.fetchall()
    cur.close();del cur; cur = None;del rs; rs = None
    dbh.close();del dbh; dbh = None;
    dbh = ibm_db_dbi.connect("db2")
    cur = dbh.cursor();cur.execute('SELECT * FROM T3');rs = cur.fetchall()
    cur.close();del cur; cur = None;del rs; rs = None
    dbh.close();del dbh; dbh = None;

t=60
print "sleeping %d seconds" % t
time.sleep(60)
print "[DEBUG] memory  %d"% int(os.popen(cmd).read())

Original comment by south.mi...@gmail.com on 19 Jun 2012 at 7:14

[GoogleCodeExporter]

I am taking it up for bigger loop that should run at least 20 to 30 min, if 
memory uses will increase continuously then definitely there will be a memory 
leak. And then after I will start looking the code to find the memory leak.

Will get back to you very soon after my findings. 

Thanks,
Rahul Priyadarshi

Original comment by rahul.pr...@in.ibm.com on 20 Jun 2012 at 7:01

[GoogleCodeExporter]

Thanks Rahul, let me know if you need any tests on my side.

Original comment by south.mi...@gmail.com on 21 Jun 2012 at 1:57

[GoogleCodeExporter]

I tried with a long running test case and found that there is gradually 
increase in memory but it is in very small amount
I ran the same program with valgrind to detect the memory leak but this tool 
dosn't show anything which differentiate the memory leak.

The amount in increase in memory is very small so, we stuck here. 

If you have any program in your environment which shows some significant 
gradually increase in memory, then please run that program with valgrind and 
send us the output. This may help us to find the memory leak.

Original comment by rahul.pr...@in.ibm.com on 22 Jun 2012 at 10:30

[GoogleCodeExporter]

Hi Rahul,
See attached the valgrind output for these command lines
valgrind --tool=memcheck --suppressions=valgrind-python.supp
--log-file=leak-valgrind-yes.log --leak-check=yes python -E -tt
test.py
valgrind --tool=memcheck --suppressions=valgrind-python.supp
--show-reachable=yes --log-file=leak-valgrind-yes-reachable.log
--leak-check=yes python -E -tt test.py

test.py is the real app and runs for 19 loops and exits, at prod it
runs continuously but as far as I've seen it behaves the same after a
24+ hs running.

Original comment by south.mi...@gmail.com on 22 Jun 2012 at 5:09

[GoogleCodeExporter]

I am not able to find the attachment. Please try to attach it again.

Original comment by rahul.pr...@in.ibm.com on 28 Jun 2012 at 7:34

[GoogleCodeExporter]

Here it goes again

Original comment by south.mi...@gmail.com on 28 Jun 2012 at 1:06

Attachments:

• valgrind.tar.gz

[GoogleCodeExporter]

I looked into the output file, but it only contain the possible memory leak due 
to "PyMalloc". So it doesn't help us more.

If possible then try to run your script on "--without-pymalloc" configured 
python and send us the valgrind output file to us.

Original comment by rahul.pr...@in.ibm.com on 2 Jul 2012 at 9:32

[GoogleCodeExporter]

ok, will try to compile python and test.

Original comment by south.mi...@gmail.com on 3 Jul 2012 at 4:08

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Thanks Rahul. JOOC, is issue#120 related to #114?

Original comment by south.mi...@gmail.com on 22 Oct 2012 at 5:37

[GoogleCodeExporter]

Sorry, issue#120 is not related with 114. Please try the new version and let us 
know how it works for you. 

Original comment by rahul.pr...@in.ibm.com on 23 Oct 2012 at 7:26

• Changed state: New

[GoogleCodeExporter]

ok, #120 looks similar to what I had on #114 so it might be fixed
also. Will try and let you know.

Original comment by south.mi...@gmail.com on 23 Oct 2012 at 8:33

[GoogleCodeExporter]

Hi,
I tested the new version and looks good. I had some warnings at
installation, see attached log.

Original comment by south.mi...@gmail.com on 24 Oct 2012 at 3:56

[GoogleCodeExporter]

Looks like you have missed to attach the log file.

Original comment by rahul.pr...@in.ibm.com on 25 Oct 2012 at 6:25

[GoogleCodeExporter]

It might be stripped down by mail server, sending it again compressed.

Original comment by south.mi...@gmail.com on 25 Oct 2012 at 2:12

[GoogleCodeExporter]

It looks like mail server again removed the attachment. 

Original comment by rahul.pr...@in.ibm.com on 2 Nov 2012 at 6:55

[GoogleCodeExporter]

Plain text now :p

amenabar:~ # easy_install --upgrade ibm-db
/usr/lib64/python2.6/site-packages/setuptools/package_index.py:7:
DeprecationWarning: the md5 module is deprecated; use hashlib instead
  from md5 import md5
Searching for ibm-db
Reading http://pypi.python.org/simple/ibm-db/
Reading http://code.google.com/p/ibm-db/downloads/list
Best match: ibm-db 2.0.0
Downloading 
http://pypi.python.org/packages/source/i/ibm_db/ibm_db-2.0.0.tar.gz#md5=709c576c
0ec2379ca15049f5c861beb1
Processing ibm_db-2.0.0.tar.gz
Running ibm_db-2.0.0/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-XGKZqD/ibm_db-2.0.0/egg-dist-tmp-l5a0V0
Detected 64-bit Python
In file included from ibm_db.c:27:
ibm_db.h:31:1: warning: "PyObject_CheckBuffer" redefined
In file included from /usr/include/python2.6/Python.h:132,
                 from ibm_db.c:26:
/usr/include/python2.6/abstract.h:534:1: warning: this is the location
of the previous definition
zip_safe flag not set; analyzing archive contents...
testfunctions: module MAY be using inspect.stack
tests: module references __file__
tests.test_144_BindParamInsertStmtPARAM_FILE: module references __file__
tests.test_000_PrepareDb: module references __file__
Removing ibm-db 1.0.6 from easy-install.pth file
Adding ibm-db 2.0.0 to easy-install.pth file

Installed 
/usr/local/lib64/python2.6/site-packages/ibm_db-2.0.0-py2.6-linux-x86_64.egg
Processing dependencies for ibm-db
Finished processing dependencies for ibm-db

Original comment by south.mi...@gmail.com on 2 Nov 2012 at 1:39

[GoogleCodeExporter]

The first warning "DeprecationWarning: the md5 module is deprecated; use 
hashlib instead" is coming through setuptools, so they may resolve it in coming 
coming release.

Python have made some changes in python-2.6.x and Python-2.7.x to make it 
compatible with Python-3.x, Many of these are wrapper kind of thing which they 
defined in header file. we supports 2.5.x and above and treat all Python2.x 
version as same that's why PyObject_CheckBuffer get redefined. so, we can also 
ignore this warning.

Thanks,
Rahul

Original comment by rahul.pr...@in.ibm.com on 10 Dec 2012 at 8:57

[GoogleCodeExporter]

Original comment by rahul.pr...@in.ibm.com on 1 Mar 2013 at 10:09

• Changed state: Fixed