Compiling large functions runs out of memory

[GoogleCodeExporter]

The following program:

import sys
copies = int(sys.argv[1])
print "Running %d copies..." % copies

longexpr = 'x = x or ' + '-x' * 2500
code = ('''
def f(x):
''' + '''    %s
''' * copies + '''
    # the expressions above have no effect, x == argument
    while x:
        x -= 1
    return x
''') % ((longexpr,) * copies)
exec code
print f(5)

demonstrates non-linear memory use and running times with Unladen Swallow
r573. This was extracted from test_compile, which takes more than 4GB of
memory which exhausts a 32-bit address space.

The memory use below is from watching the "real memory" column in the apple
activity monitor and taking the highest number I saw.

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  1
Running 1 copies...
0

real    0m37.979s
user    0m36.972s
sys 0m0.602s
memory 174MB

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  2
Running 2 copies...
0

real    1m15.750s
user    1m13.479s
sys 0m1.368s
memory 491MB  (delta 317)

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  3
Running 3 copies...
0

real    2m6.118s
user    2m2.631s
sys 0m2.328s
memory 944MB  (delta 453)

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  4
Running 4 copies...
0

real    5m59.303s
user    3m10.135s
sys 0m14.223s
memory 1500MB  (delta 556)

Watching memory use of the "3" case, it seems to rise to ~30-40MB up to and
during SimplifyCFG, then rise by ~3 MB per second through CodeGenAndEmitDAG
up to about 150MB, and then rise by ~50MB per second through
LiveVariables::runOnMachineFunction up to about 750MB. LiveIntervals seems
to account for most of the rest of the memory use.

Original issue reported on code.google.com by jyass...@gmail.com on 22 May 2009 at 2:59

[GoogleCodeExporter]

r583 brings the memory use for test_compile down to <1GB:
/usr/bin/time -l:
     2924.16 real      2723.06 user        32.51 sys
 987070464  maximum resident set size
...

$ /usr/bin/time -l ./python.exe -S -L0 ./use_lots_of_memory.py  4
Running 40 copies...
0
      197.85 real       189.50 user         2.63 sys
 403173376  maximum resident set size

I think there's still quite a bit of low-hanging fruit here, but I'm moving on 
to
other things for now.

Original comment by jyass...@gmail.com on 26 May 2009 at 11:41

[GoogleCodeExporter]

Moving back to the free pool.

Original comment by collinw on 19 Jun 2009 at 7:14