Fatal error in gc - Too many heap sections

[GoogleCodeExporter]

First up - much appreciation for all the awesome work you've put into shedskin. 
I'm getting a 10x speed increase for some monte carlo sampling routines. I've 
detailed my problem below ... excuse any rookie stuff I'm still on a big 
learning curve at the moment. Any pointers you can provide - obvious mistakes 
I'm making that expose me to the problem detailed below would be much 
appreciated.

What steps will reproduce the problem?
1. Windows 7 - compile an extension module that runs a multi-year, multi loop 
monte carlo re-sampling function (ie. many, many loops done mostly as list 
comprehensions and a substantial number of multi-dimensional lists computed) - 
mostly look something like this:

resampled_AEP_for_year = [precalcs.b_AEPs[haz]*(1 + 
resample_AEP(precalcs.b_AEPs_hilo[0], 
precalcs.b_AEPs_hilo[1],precalcs.AEP_mods[year][haz])) for haz in 
range(num_hazards)]

2. pack resulting computations in a class that's defined within the compiled 
extension module itself, and imported to the main python script e.g.

class Asset_Output:
   def __init__ (self):
       self.var = 0
       ...etc

The initial values the plugin works on are also passed in as an instance of a 
class that's defined in the plugin and imported for population by the main 
script. Mostly dimensional lists again.

3. Wrap that all in a Django web app and ask the compiled plugin to run for 
10000 resamples for 30 years. Collect output instances for repacking through 
javascript etc.

What is the expected output? What do you see instead?

Module compiles fine with no errors, using only the -e option and the compiler 
shedskin ships with.

Expected output while it is working is perfect - and for small object samples, 
or low monte carlo iteration values it works. However after about 150 calls to 
the plugin have been made it dies with a "fatal error in gc" - "Too many heap 
sections" popup - and a gc.log full of "GC Warning: Repeated allocation of very 
large block (appr. size 135168):May lead to memory leak and poor performance." 
- which seems to be what kills it.

What version of the product are you using? On what operating system?

shedskin 0.9.2, on Windows 7 Professional 64-bit, running Python 2.7

Please provide any additional information below.

I have tried to recompile the boehm garbage collector with a variety of config 
options (--enable-large-config, --with-large-heap, --enable-big-arrays), with 
both 32 and 64 bit mingw compilers. Still get the same error for large runs.

I've also notice there doesn't appear to be any memory released from a python 
process that keeps growing even after error free runs. It seems related to 
using the compiled version of the module - as I don't see the same continued 
growth with a straight python version of the module (grows once then stays 
static for subsequent equivalent runs). The python version is also obviously 
much much slower, but doesn't die for large runs.

Cheers, and thanks again.

Shane

Original issue reported on code.google.com by shane.br...@gmail.com on 6 Jul 2012 at 12:50

[GoogleCodeExporter]

thanks for reporting, and apologies for the late reply!

would it be possible to send me a version of your code that exhibits the 
problem, preferrably as small as possible of course, so I can see if I can 
reproduce it here..?

Original comment by mark.duf...@gmail.com on 14 Jul 2012 at 8:49

[GoogleCodeExporter]

Hello,
thanks heaps for getting in contact. The code is attached - it's some
ugly stuff I've whacked together for some rushed prototyping - I'm
sure it could be prettier but time > style at the moment.

Apologies for the length - but I think you'll get the drift - lots of
loops and list comprehensions. I reworked an earlier one which had
even more comprehensions into this one with extra ugly for loops to
test to see if it was the sheer bulk of dynamic comprehension work
that was the problem. It didn't help.

Might be worth mentioning that I also have dramas with scipy/numpy -
which I sorted with threading.Rlock(). I think there's some strange
stuff going on with windows 7,  python & c/c++ libraries. Anyway -
here's the wall of code.

Thanks again,

Shane

Original comment by shane.br...@gmail.com on 16 Jul 2012 at 12:16

[GoogleCodeExporter]

hi shane,

sorry for being so slow again. it looks like google ate your attachment?

thanks again,
mark.

Original comment by mark.duf...@gmail.com on 25 Jul 2012 at 9:30

[GoogleCodeExporter]

Slow? no way. I responded directly through gmail - I guess all attachments 
aren't equal. Try this one..
Cheers & thanks again,

Shane

Original comment by shane.br...@gmail.com on 25 Jul 2012 at 11:29

Attachments:

• risk_plugin5.py

[GoogleCodeExporter]

please also attach a small test program that utilizes the extension module and 
exhibits the problem.. :-)

Original comment by mark.duf...@gmail.com on 25 Jul 2012 at 1:20

[GoogleCodeExporter]

No worries - carving all the django off it to a stripped down call should be 
achievable. I'll do that in the near future. Thanks again for your attention to 
this.

Original comment by shane.br...@gmail.com on 26 Jul 2012 at 1:11