Long-term development targets

ropensci / chopin

Computation of Spatial Data by Hierarchical and Objective Partitioning of Inputs for Parallel Processing

https://niehs.github.io/chopin/

Other

12 stars 2 forks source link

Open sigmafelix opened 1 year ago

sigmafelix commented 1 year ago

Bridging python packages (dask-geopandas, geopolars, cuSpatial) to R workflow
Optimal splitting of computational regions regarding shape complexity and input data resolutions
Streamlined integration of sparklyr and sf-derived functions

--- more ideas to comments ---

sigmafelix commented 9 months ago

sigmafelix commented 8 months ago

mirai keeps using the low rate of CPU and ends up taking longer time than future runs.
future.batchtools integration into targets pipeline in other projects are promising, but I want to note that it will not guarantee to reduce the total runtime compared to running separate pipeline nodes with future.
crew is relevant to targets pipeline.

sigmafelix commented 8 months ago

[x] Integration of callr for background processing (semi-automatic and asynchronous, I would say a la parallel) -- future.callr can handle this.
[x] Passing paths instead of sf/terra objects in vector processing use cases
[x] Memory saving solutions: no global object export as default