Memory leak in Model

[pmichel31415]

When running this code :

import _dynet as dy

dp=dy.DynetParams()
dp.from_args()
dp.set_mem(512)
dp.init()

def create_model():
    m=dy.Model()
    w=m.add_parameters((1000,1000))
    del m

for i in range(100):
    print '=== Iteration %d ===' % i
    try:
        create_model()
    except e:
        exit()
    print '===================='

With an old version (without dynamic memory pool), I get the following error

[dynet] random seed: 2856107851
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
=== Iteration 0 ===
====================
=== Iteration 1 ===
====================
=== Iteration 2 ===
====================
=== Iteration 3 ===
====================
=== Iteration 4 ===
====================
=== Iteration 5 ===
====================
=== Iteration 6 ===
====================
=== Iteration 7 ===
====================
=== Iteration 8 ===
====================
=== Iteration 9 ===
====================
=== Iteration 10 ===
====================
=== Iteration 11 ===
====================
=== Iteration 12 ===
====================
=== Iteration 13 ===
====================
=== Iteration 14 ===
====================
=== Iteration 15 ===
====================
=== Iteration 16 ===
====================
=== Iteration 17 ===
====================
=== Iteration 18 ===
====================
=== Iteration 19 ===
====================
=== Iteration 20 ===
====================
=== Iteration 21 ===
====================
=== Iteration 22 ===
terminate called after throwing an instance of 'std::runtime_error'
  what():  CPU parameter memory is out of memory, try increasing with --dynet-mem (current capacity: 178257920). See http://dynet.readthedocs.io/en/latest/commandline.html for details.
[1]    3281 abort (core dumped)  python test_mem.py

There are two big problems here :

• There is a memory leak
• With the new dynamic memory pool this is much worse since there is no way to limit the memory used by dynet

Is there a way to limit the memory (ideally an option like --dynet-mem [initial mem]-[maximum-mem]) ?

[jujbob]

I've got the similar problem for running my parser several times in a system using dynet. If you are really urgent to run your system properly, you can run it step by step by using "os.system(command)" like me.

for a_eval in data: os.system(command) --> #command must have load and save model. e.g.) if has_prev_model: load model

In my case, a saved model is almost 1.5G so I have to delete model in the memory before running a next job(need to load another model) but it was impossible. I have spent a lot of time to solve it temporally because I have thought the memory problem seems to be my system fault.

Thank you share it :+1:

[yoavg]

Did you verify it is indeed a python issue and does not happen in cpp?

[yoavg]

By looking at the cpp code, I think this is because there are no destructors for ParameterStorage and LookupParametersStorage.

[pmichel31415]

Yes I confirmed this is a c++ problem

[neubig]

So I took a look at this, and it's basically a result of the DyNet memory allocator not having any concept of garbage collection. When a parameter is allocated the amount of memory the memory allocator is using is incremented, but there's no way to free memory (other than freeing all the memory, which would kill all of the parameters that we have loaded).

I thought about this a bit, and I think it might be useful to generalize the DyNet memory management so that we can support different types of allocators:

• A "normal" memory allocator that uses standard OS function calls to allocate and free memory. This could be used for things that don't need to be super-fast.
• The super-fast memory allocator that we're using now.
• Some other kind of smart memory allocator that falls between the two.

[gth879w]

(other than freeing all the memory, which would kill all of the parameters that we have loaded).

Is this done by calling cleanup() declared in init.h?

I have a process that needs to repeatedly create and delete independent dynet models. In this case, would I call initialize() and cleanup() repeatedly?