Rapidly increasing memory usage on InitPlan() call

[kb0n]

Hi,

I was trying to figure out why the memory on my machine gets used up so quickly. It happens on InitPlan() call (I'm usibg BiRRT), slowly occupying some memory. But after the ~20th call it occupies additional 100MBs of RAM, almost doublich after each call, reaching the RAM limits pretty quickly.

After spending hours of debugging, I narrowed it down to a problem with boost::pool, used in "plugins/rplanners.h". Not sure if it's a Boost-bug or intenden pool-design, I read so much text about it lately that I got confused :)

Anyhow, it seems that the "purge_memory" does not what is intended here, which would be freeing all allocated memory and start over again. Looking at Boost's "pool.hpp" that may be because it sets "this->first = 0;", which would mean the internal list of free memory is empty, leading to a request to resize the pool with the next usage. But I don't really know, it's a little over my head.

I tried a lot of stuff, use ordered_malloc(), ordered_free(), release_memory(), explicitly call free() before purging... Nothing worked, memory usage kept increasing. The only thing that helped was to destroy the pool, which according to Boost:

Destroying a Pool implicitly frees all chunks that have been allocated from it. 

So my suggestion would be to do exactly that each time the Init() Method of SpatialTree is called. There is a line which iirc does something like

nodePool->reset(new boost::pool<>....)

but only if the pool is not available yet. Removing the if-clause wrapping will always use a new pool, destroying the old one and reliably releasing all its memory.

Now I imagine you might have more insight into this? :)

Btw, using boost-1.54.

Cheers

[mkoval]

@es92 was also noticing memory leaks with some OpenRAVE planners. Tagging him so he can take a look at this.

[rdiankov]

wow, that's a great catch! and also very weird! boost::pool's destructor just calls purge_memory...

i'll do some experiments on this in the next couple of days to see if there's any methods just as fast that don't leak, but for now, i'm putting in bahram's suggestion:

0b0c5dba3126ba9ea635cf87435a6124e94c8d29

[kb0n]

Thanks, that would be great.

The destructor does more than calling purge_memory, more specific it deletes all its internal data like the free-list of memory blocks. I guess that list is what's growing, and I guess it is somehow related to purge_memory not properly resetting everything. But I might be wrong of course.

Why investigating other methods, is there any downside in the suggestion I made? I mean if the pool is recreated only at Init(), which calls Reset() anyway.... I'm just asking because I'm curious and don't have the insight :)

[rdiankov]

ideally a planner wouldn't have to continue re-allocating memory to the SpatialTree when it is called subsequent times. Therefore, after the first couple of calls, the system really stabilizes.

i was looking for such a solution and decided to try boost::pool. Unfortunately i found too late that its API is pretty resctirive, and in the end i had to make a purge_memory call, which was the fastest way to re-initialize the pool.

i can see a couple of different ways to optimize the node allocation by using a custom allocator solution rather than boost::pool. and its definitely worth the time spent ;0)

[kb0n]

Ah, so you want a "pool" as described in http://en.wikipedia.org/wiki/Object_pool_pattern? That's not what boost::pool does though, it's just a way to faster allocate/deallocate memory, especially useful when you need to allocate/deallocate a lot of small objects.

If you say it's worth the time spent then I guess it is :) As long as it's not my time though, haha ;)

I might be wrong though, but that's what I undestood after reading http://www.boost.org/doc/libs/1_54_0/libs/pool/doc/html/boost_pool/pool/pooling.html and a lot of stackoverflow discusssion, especially this one http://stackoverflow.com/questions/15847359/how-does-boost-pool-achieve-re-use-of-allocated-memory :)