search

kutsurak / python-bitstring

Automatically exported from code.google.com/p/python-bitstring
0 stars 0 forks source link

Memory leak in BitStream #120

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What's the problem?
There is a  memory leak somewhere in BitStream. I don't really know where, but 
I have spent a lot of days on this and also have implemented the functions with 
plain strings and the code does not leak. I attach a replication of the issue. 
Just run and watch the heap grow.

Using OSX Lion, Homebrew python 2.7.2, bitstring 2.2/3.0.1

Original issue reported on code.google.com by rui.lour...@gmail.com on 21 Jan 2012 at 6:24

Attachments:

GoogleCodeExporter commented 9 years ago 
It looks as though it's a side-effect with some cacheing code that's used to 
speed up initialisation, but I'll need to investigate further to see how much 
of a problem it is.

For your example it seems to be fixed if you re-write your initialisation as 

            ps.append(BitStream(bin=signature))

which is also a bit faster as it doesn't have to parse a string to work out 
that's it's binary.

Original comment by dr.scott...@gmail.com on 21 Jan 2012 at 6:56

GoogleCodeExporter commented 9 years ago 
Well it's not strictly a memory leak. What's happening is that the string being 
passed to the BitStream initialiser (as an 'auto' parameter) is being cached 
along with the Bits object that it creates. This is done as it's a very nice 
optimisation, especially if more complex strings are being used to initialise 
bitstrings and the code is being evaluated many times.

In general this won't be a big problem with memory usage, as the idea is that 
the string literals used to initialise are actually in the code, so the amount 
of caching is bounded by the length of the program. What's happening in this 
case though is that the 'auto' initialiser is being created programmatically 
and so many thousands are being cached. Using the more efficient and direct 
initialisation bypasses the caching and everything works OK.

So it's a feature, not a bug :). But there is a good case for limiting the 
cache in some way so that it doesn't grow unbounded so I'll leave this open as 
a defect.

Thanks for the report, hope this helps.

Original comment by dr.scott...@gmail.com on 21 Jan 2012 at 11:03

GoogleCodeExporter commented 9 years ago 
This feature has stolen me a lot of sleep :) I agree that this cache must be 
limited in some way.
I have replaced all the auto  initializations in my project and it does not 
leak anymore.

Glad to contribute.

Original comment by rui.lour...@gmail.com on 21 Jan 2012 at 11:25

GoogleCodeExporter commented 9 years ago 
Fixed in r.950. Cache sizes limited to 1000 elements.

Original comment by dr.scott...@gmail.com on 6 Feb 2012 at 4:22