search

cwerner / lpjguesstools

BSD 3-Clause "New" or "Revised" License
2 stars 2 forks source link

Out of memory error #5

Open lukasbaumbach opened 4 years ago

lukasbaumbach commented 4 years ago

When running lgt_createinput I can usually process 13-14 tiles before I run out of memory (in my case 32GB) and the process is killed. Some part of the code apparently keeps writing into memory, so that it fills up after a few tiles. When trying to debug with filprofiler, the process crashes with a segmentation fault at the end of the first tile processing (probably when trying to write to netcdf). What could be done to track down the error?

lukasbaumbach commented 4 years ago

I added some function decorators to the code and created a memory profile over time with memory_profiler. It seems that memory leaks mainly during the writing/reprojecting/processing of the DEM files (elevation, aspect, slope, tpi) in convert_dem_files. memprof_labelled

Possibly some objects are not garbage collected due to persisting references, however I could not backtrack these yet. Could any of the loops be responsible for that?

itamarst commented 3 years ago

Hi, author of filprofiler here—sorry it didn't work for you!

I've been working on it for past months, so it's hopefully more robust. Could you give the latest version a try? If it still crashes, I would love more details so I can fix it.