Open krlmlr opened 7 years ago
BatchJobs development has been moved to batchtools I believe.
I have some reservations about this:
storr
datastore; connecting these together has been a long-term goal (see #84) (the batch* packages and future are both nice though in that they are one of few parallel focussed packages that don't focus purely on doing just a blocking parallel map).The future package I really like the look of, though I've not used it in my own work yet (but then I'm not actually doing my own computational work anymore).
Given that all that is really needed in any case is a submit/fetch, perhaps a useful general approach would be some sort of generic interface that could support any of the queuing packages. That does end up implementing an interface-to-interfaces though, which seems a bit overcomplicated
Thanks.
To me it looks like we'd need to replace lapply(plan, ...)
by a scheduler in remake_make1()
. That scheduler would:
The future
package has only two dependencies, globals
and listenv
; perhaps globals
could even be made "suggested". I think future
is already a nice interface-to-interfaces; I'd expect the overhead of the "eager local" operation mode of future
to be sufficiently small if at all noticeable. What remains is the overhead of our scheduler.
An adapter between future
and querer
/rrq
would be a way of integrating your queues with future
, and consequently with remake
.
The globals package could be possibly used internally for some of the code scouring (as mooted in #64, #121)
BatchJobs development has been moved to batchtools I believe.
this is correct. if you want to go further here, please use batchtools.
FYI, it's pretty high up on my todo list to get a first version of future.batchtools up and running. Most of the work will be to port existing future.BatchJobs code over and adjust according to what's in the suggested migration docs. It will require me to find some free deep-focus time, but after than it should be quick, e.g. the API and all package / redundancy tests are basically already in place (= the same).
So, obviously biased but also why future exists in the first, by utilizing the Future API you should be able to get what you need and then the user can choose to use whatever backend they want.
UPDATE: The future.batchtools package is on CRAN since 2017-06-03.
It would be awesome if
remake
usedBatchJobs
as backend to perform its computation.BatchJobs
has also a "local" mode that performs the computation in the same R process, so this could well be the default mode of operation.The
future
andfuture.BatchJobs
packages could simplify the implementation.CC @mllg @berndbischl @HenrikBengtsson.