search

OGGM / oggm

Open Global Glacier Model
http://oggm.org
BSD 3-Clause "New" or "Revised" License
218 stars 105 forks source link

OGGM Memory Usage #722

Open sumnonpuella opened 5 years ago

sumnonpuella commented 5 years ago

Hello all - I apologize if this is a small issue, but I went to run the Arctic Canada North RGI directory today and found a memory issue I have never seen before with this directory, and was wondering if anyone else had this issue. When I go to initialize the glacier directories with

cfg.set_intersects_db(utils.get_rgi_intersects_region_file('03', version='6')) path = utils.get_rgi_region_file('03', version='61') rgidf = gpd.read_file(path) gdirs = oggm.workflow.init_glacier_regions(rgidf)

my computer froze for 10 minutes, and when it finally unfroze, I had the error

oggm.core.gis: MemoryError occurred during task define_glacier_region on RGI60-03.00085:

repeated for nearly every error in the directory. I reran the code while monitoring the memory usage, and OGGM was using 99+% of the memory, where it normally uses less than 40%. I'm working on resolving the issue on my end, but I was wondering if anyone has ever seen this issue before. Thank you in advance for your time.

fmaussion commented 5 years ago

Never seen this error before, sorry...

Have you considered starting from preprocessed directories directly?

https://docs.oggm.org/en/v1.1/input-data.html#pre-processed-directories

TimoRoth commented 5 years ago

How much memory are we talking about in absolute numbers? I wouldn't run OGGM, specially in multiprocessing mode, on a machine anywhere short of 16 or even 32GB of RAM.

sumnonpuella commented 5 years ago

@TimoRoth Our computer has 62GB of RAM, and OGGM typically only uses ~30GB. Suddenly, it jumped to the full 62 when I tried to run a newly initialized glacier directory. It looks like we don't have the issue when its running without multiprocessing.