search

google-code-export / ccc-gistemp

Automatically exported from code.google.com/p/ccc-gistemp
Other
0 stars 0 forks source link

MemoryError on full run #109

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
When running the full ccc-gistemp (python tool/run.py) I get a MemoryError.  
This is fairly recent; it used to be the case that the analysis would have no 
problem on this machine.  Work-around seems to be to run one step at a time 
(python tool/run.py -s 0; python tool/run.py -s 1; etc).

====> STEPS 0, 1, 2, 3, 4, 5  ====
reading input/oiv2mon.201102
Load GHCN records
Load USHCN records
(Reading average temperature)
Load HOHENPEISSENBERG records
Load SCAR records
Correct the GHCN Hohenpeissenberg record.
Adjust USHCN records
1 USHCN records had no GHCN counterpart.
Step 0: closing output file.
Step 1: closing output file.
Step 2: closing output file.
Region (+64/+90 S/N -180/-090 W/E): 1 empty cell.
Region (+64/+90 S/N -090/+000 W/E): 0 empty cells.
Region (+64/+90 S/N +000/+090 W/E): 0 empty cells.
Region (+64/+90 S/N +090/+180 W/E): 0 empty cells.
Region (+44/+64 S/N -180/-135 W/E): 0 empty cells.
Region (+44/+64 S/N -135/-090 W/E): 0 empty cells.
Region (+44/+64 S/N -090/-045 W/E): 0 empty cells.
Region (+44/+64 S/N -045/+000 W/E): 0 empty cells.
Region (+44/+64 S/N +000/+045 W/E): 0 empty cells.
Region (+44/+64 S/N +045/+090 W/E): 0 empty cells.
Region (+44/+64 S/N +090/+135 W/E): 0 empty cells.
Region (+44/+64 S/N +135/+180 W/E): 0 empty cells.
Region (+24/+44 S/N -180/-150 W/E): 34 empty cells.
Region (+24/+44 S/N -150/-120 W/E): 2 empty cells.
Region (+24/+44 S/N -120/-090 W/E): 0 empty cells.
Region (+24/+44 S/N -090/-060 W/E): 0 empty cells.
Region (+24/+44 S/N -060/-030 W/E): 10 empty cells.
Region (+24/+44 S/N -030/+000 W/E): 0 empty cells.
Region (+24/+44 S/N +000/+030 W/E): 0 empty cells.
Region (+24/+44 S/N +030/+060 W/E): 0 empty cells.
Region (+24/+44 S/N +060/+090 W/E): 0 empty cells.
Region (+24/+44 S/N +090/+120 W/E): 0 empty cells.
Region (+24/+44 S/N +120/+150 W/E): 0 empty cells.
Region (+24/+44 S/N +150/+180 W/E): 26 empty cells.
Region (+00/+24 S/N -180/-158 W/E): 22 empty cells.
Region (+00/+24 S/N -158/-135 W/E): 41 empty cells.
Region (+00/+24 S/N -135/-112 W/E): 83 empty cells.
Region (+00/+24 S/N -112/-090 W/E): 27 empty cells.
Region (+00/+24 S/N -090/-068 W/E): 0 empty cells.
Region (+00/+24 S/N -068/-045 W/E): 10 empty cells.
Region (+00/+24 S/N -045/-022 W/E): 41 empty cells.
Region (+00/+24 S/N -022/+000 W/E): 1 empty cell.
Region (+00/+24 S/N +000/+022 W/E): 0 empty cells.
Region (+00/+24 S/N +022/+045 W/E): 0 empty cells.
Region (+00/+24 S/N +045/+068 W/E): 3 empty cells.
Region (+00/+24 S/N +068/+090 W/E): 0 empty cells.
Region (+00/+24 S/N +090/+112 W/E): 0 empty cells.
Region (+00/+24 S/N +112/+135 W/E): 0 empty cells.
Region (+00/+24 S/N +135/+158 W/E): 0 empty cells.
Region (+00/+24 S/N +158/+180 W/E): 2 empty cells.
Region (-24/-00 S/N -180/-158 W/E): 0 empty cells.
Region (-24/-00 S/N -158/-135 W/E): 0 empty cells.
Region (-24/-00 S/N -135/-112 W/E): 55 empty cells.
Region (-24/-00 S/N -112/-090 W/E): 81 empty cells.
Region (-24/-00 S/N -090/-068 W/E): 7 empty cells.
Region (-24/-00 S/N -068/-045 W/E): 0 empty cells.
Region (-24/-00 S/N -045/-022 W/E): 9 empty cells.
Region (-24/-00 S/N -022/+000 W/E): 2 empty cells.
Region (-24/-00 S/N +000/+022 W/E): 0 empty cells.
Region (-24/-00 S/N +022/+045 W/E): 0 empty cells.
Region (-24/-00 S/N +045/+068 W/E): 2 empty cells.
Region (-24/-00 S/N +068/+090 W/E): 61 empty cells.
Region (-24/-00 S/N +090/+112 W/E): 1 empty cell.
Region (-24/-00 S/N +112/+135 W/E): 0 empty cells.
Region (-24/-00 S/N +135/+158 W/E): 0 empty cells.
Region (-24/-00 S/N +158/+180 W/E): 0 empty cells.
Region (-44/-24 S/N -180/-150 W/E): 25 empty cells.
Region (-44/-24 S/N -150/-120 W/E): 37 empty cells.
Region (-44/-24 S/N -120/-090 W/E): 48 empty cells.
Region (-44/-24 S/N -090/-060 W/E): 2 empty cells.
Region (-44/-24 S/N -060/-030 W/E): 23 empty cells.
Region (-44/-24 S/N -030/+000 W/E): 18 empty cells.
subbox at -40.9+013.5 (5 empty)Traceback (most recent call last):
  File "tool/run.py", line 280, in <module>
    sys.exit(main())
  File "tool/run.py", line 261, in main
    data = step_fn[step](data)
  File "tool/run.py", line 104, in run_step5
    result = step5.step5(data)
  File "/home/nb/ccc-gistemp/trunk/code/step5.py", line 541, in step5
    land,ocean,mixed = as_boxes(subboxes)
  File "/home/nb/ccc-gistemp/trunk/code/step5.py", line 36, in as_boxes
    return land_ocean_boxes(meta, data)
  File "/home/nb/ccc-gistemp/trunk/code/step5.py", line 74, in land_ocean_boxes
    for landweight, landcell, oceancell in cells:
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 1687, in step5_mask_output
    for datum in data:
  File "/home/nb/ccc-gistemp/trunk/code/step5.py", line 514, in ensure_weight
    for _,land,ocean in data:
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 1567, in step5_input
    for land, ocean in data:
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 1601, in ensure_landocean
    for series in data:
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 1544, in step4_output
    for land,ocean in data:
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 1424, in step3_output
    out.write(thing)
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 183, in write
    self._flush(len(record))
  File "/home/nb/ccc-gistemp/trunk/tool/gio.py", line 171, in _flush
    b.stations, b.station_months, b.d, *b.series)
  File "/usr/local/lib/python2.5/struct.py", line 63, in pack
    return o.pack(*args)
MemoryError
$

Original issue reported on code.google.com by nick.bar...@gmail.com on 8 Mar 2011 at 1:20