clcr / pyeo

Python Earth Observation processing chain
GNU General Public License v3.0
96 stars 29 forks source link

MemoryError when processing large areas #39

Closed ValentinLouis closed 5 years ago

ValentinLouis commented 5 years ago

I recieved another Memory Error but this time at a different stage and instead of a GDAL VirtualMemory error it is a numpy MemeoryError. Here the details:

in_masked = in_masked = get_masked_array(in_raster, mask_paths[i]) File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1140, in get_masked_array mask_array = project_array(mask_array, raster_array.shape[0], 0) File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1147, in project_array array_in = np.repeat(array_in, depth, axis) File "/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 445, in repeat return _wrapfunc(a, 'repeat', repeats, axis=axis) File "/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 51, in _wrapfunc return getattr(obj, method)(*args, **kwds) MemoryError

Am I correct in assumeing that it is another RAM issue and it should be fine to increase the required vmem?