steps-dev / steps

steps: Spatially- and Temporally- Explicit Population Simulator
19 stars 7 forks source link

Feature request: Write rasters to disc #22

Open geryan opened 5 years ago

geryan commented 5 years ago

Would be totes fab to be able to specify a filepath/pattern and write output rasters to disc rather than forcibly hold them in memory when running simulation.

Especially for long runs and large habitat suitability inputs this would help memory management much more betterer.

geryan commented 5 years ago

NB reading rasters from disc makes the plot method very slow, in particular getting the population size. Using future_lapply instead of lapply in get_pop_replicate can significantly improve this

goldingn commented 5 years ago

raster should automatically be reading and writing rasters from disk (in a temporary file directory) if it looks like you don't have enough memory for each object. You can control this behaviour a bit with rasterOptions()

Agreed there could be more use of parallelism in the plotting bit

geryan commented 4 years ago

The other part of the issue I guess isn't the writing to disk per se, but being able to save, export and read-in results later, which isn't easily managed from temp files.

cvisintin commented 4 years ago

Can you please expand upon this bit: "being able to save, export and read-in results later, which isn't easily managed from temp files" What is your current workflow? What is difficult? What would you like the behaviour to be?

geryan commented 4 years ago

I guess it's OK if the rasters are in memory, you can export and read in again later. If the rasters are large and written to a temp file somewhere then it's not possible to simply export the results object to read in elsewhere because the temp files will be missing.