Memory leak

[sbrisard]

The following code causes a memory leak in janus.discretegreenop:

import janus.discretegreenop as discretegreenop
import janus.greenop as greenop

from janus.matprop import IsotropicLinearElasticMaterial as Material
from pympler import tracker

def run():
    mat = Material(1.0, 0.3, 2)
    n0 = 128
    n1 = 128
    n = (n0, n1)

    greenc = greenop.create(mat)
    greend = discretegreenop.TruncatedGreenOperator2D(greenc, n, 1.)

if __name__ == '__main__':
    memory_tracker = tracker.SummaryTracker()
    i = 0
    for i in range(10):
        i += 1
        run()
        print("Loop: ",i)
        memory_tracker.print_diff()

Log file generated by pympler show that janus.discretegreenop.memoryview was not be freed. By adding the following function in discretegreenop.pyx, the memory leak has gone away:

#!python

def __dealloc__(self):
    self.green = None

Same problem observed in module janus.operators.

Initially reported by @vptran.

[sbrisard]

Hello, I've looked into the fix you propose. It seems to me that even an empty __dealloc__() function solves the problem

def __dealloc__(self):
    pass

I need to check whether the problem concerns all extension types with extension types as attributes, or only extension types with memoryviews (Cython arrays) as attributes.

[sbrisard]

Hi again,

I've managed to create a minimal working example illustrating this leak issue. Here is memory_leak.pyx

from cython.view cimport array

cdef class DoesntLeak:
    cdef double[:] buffer

    def __cinit__(self, n):
        self.buffer = array((n,), sizeof(double), 'd')

cdef class Leaks(DoesntLeak):
    cdef double[:] buffer2

    def __cinit__(self, n):
        self.buffer2 = array((n,), sizeof(double), 'd')

cdef class NoLongerLeaks(DoesntLeak):
    cdef double[:] buffer2

    def __cinit__(self, n):
        self.buffer2 = array((n,), sizeof(double), 'd')

    def __dealloc__(self):
        pass

Please note that the only difference between classes Leaks and NoLongerLeaks is the empty __dealloc__() function. To compile this Cython module, use the following setup.py

import setuptools

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

extensions = [Extension('memory_leak',
                        sources=['memory_leak.pyx'])]

setup(name='Memory leak',
      cmdclass = {'build_ext': build_ext},
      ext_modules=extensions)

Finally, the script below illustrates the problem

import gc

from memory_leak import DoesntLeak
from memory_leak import Leaks
from memory_leak import NoLongerLeaks

def num_memoryviews():
    gc.collect()
    return sum(1 if 'memoryview' in str(type(o)) else 0
               for o in gc.get_objects())

def test(cls, num_iters):
    def f():
        x = cls(1024)
    for i in range(num_iters):
        f()
        print('Iteration {0},  {1} memoryviews.'.format(i + 1,
                                                        num_memoryviews()))

if __name__ == '__main__':
    n = 10
    for cls in [DoesntLeak, Leaks, NoLongerLeaks]:
        print('*** With class {0} ***'.format(cls.__name__))
        test(cls, 10)
        print('')

Produces the following output

*** With class DoesntLeak ***
Iteration 1,  0 memoryviews.
Iteration 2,  0 memoryviews.
Iteration 3,  0 memoryviews.
Iteration 4,  0 memoryviews.
Iteration 5,  0 memoryviews.
Iteration 6,  0 memoryviews.
Iteration 7,  0 memoryviews.
Iteration 8,  0 memoryviews.
Iteration 9,  0 memoryviews.
Iteration 10,  0 memoryviews.

*** With class Leaks ***
Iteration 1,  1 memoryviews.
Iteration 2,  2 memoryviews.
Iteration 3,  3 memoryviews.
Iteration 4,  4 memoryviews.
Iteration 5,  5 memoryviews.
Iteration 6,  6 memoryviews.
Iteration 7,  7 memoryviews.
Iteration 8,  8 memoryviews.
Iteration 9,  9 memoryviews.
Iteration 10,  10 memoryviews.

*** With class NoLongerLeaks ***
Iteration 1,  10 memoryviews.
Iteration 2,  10 memoryviews.
Iteration 3,  10 memoryviews.
Iteration 4,  10 memoryviews.
Iteration 5,  10 memoryviews.
Iteration 6,  10 memoryviews.
Iteration 7,  10 memoryviews.
Iteration 8,  10 memoryviews.
Iteration 9,  10 memoryviews.
Iteration 10,  10 memoryviews.

This is all very confusing, and I am going to ask for clarification on the Cython ML before I modify the code.

[sbrisard]

New script illustrating memory leak in 7098066bc6560478424a9fb6d08a07a1821a626d.

[sbrisard]

This patch to Cython should fix the bug (to be confirmed). It is implemented in Cython 0.20.2.