search

davidtedfordholt / fable.bsts

Implementation of the bsts package for use with the fable framework
8 stars 3 forks source link

use of future package integration doesn't load libraries #14

Open davidtedfordholt opened 4 years ago

davidtedfordholt commented 4 years ago

In the following example, the dtwclust library is explicitly loaded into each worker. I'm not sure how to explicitly load things like rlang into the workers when using the implementation of the future packages in fable. @mitchelloharawild , I assume I'm just failing to put the required packages someplace appropriate, to ensure that they get called?

require("doParallel")
# Create parallel workers
workers <- makeCluster(2L)
# Preload dtwclust in each worker; not necessary but useful invisible(clusterEvalQ(workers, library("dtwclust")))
# Register the backend; this step MUST be done registerDoParallel(workers)
mitchelloharawild commented 4 years ago

For now I approach parallel at the batch modelling level (via model()) and avoid any parallelisation within the model estimator itself. Later I hope to also allow parallelisation within the model estimation (further splitting up the compute), however I haven't thought about the best way to do this yet.

I would expect that within-estimation parallelisation should use the same method as model() parallelisation, and that fabletools should provide some helpers for this. But again, I haven't thought about this problem much yet.

davidtedfordholt commented 4 years ago

No problem. That's definitely on the lower end of importance, even in terms of implementing parallelisation (probably below both refit and stream).