search

wlandau / crew

A distributed worker launcher
https://wlandau.github.io/crew/
Other
129 stars 4 forks source link

Prepare {crew} to do without a dispatcher process #60

Closed wlandau closed 1 year ago

wlandau commented 1 year ago

crew currently relies on being able to get a valid dispatcher process ID from attr(dimnames(daemons())[[1]], "dispatcher_pid"). To prepare for potential improvements to mirai which would eliminate the dispatcher process, I plan to disable the current dispatcher checks when attr(dimnames(daemons())[[1]], "dispatcher_pid") is NULL. FYI @shikokuchuo.

shikokuchuo commented 1 year ago

Cool! I am still waiting for CRAN Winbuilder-devel to update to nanonext 0.8.1 before releasing the current version of mirai...

Might necessitate another nanonext version before I can integrate the dispatcher for mirai, but shall see...

wlandau commented 1 year ago

Sounds great!

When the next mirai is released, I will make the new version the required minimum version for crew, and I will remove this note in the README. And if the dispatcher process goes away (which would be super exciting), I will remove the checks commented in R/crew_router.R.

I am planning to release the next version of crew on April 24, a month after the most recent release. (Except for emergency bugfixes, I prefer to space CRAN releases a least a month apart as per CRAN guidelines.) After that, I will make formal announcements on social media and invite others to write launcher plugins.

Integration into targets is my very next step, the only thing from keeping me from that is the other projects I am juggling. After that, I plan to write launcher plugins for traditional clusters. I will elicit whatever help I can get for the cloud stuff.

shikokuchuo commented 1 year ago

Great plan. Super exciting! I am making some optimisations in nanonext in preparation...

shikokuchuo commented 1 year ago

Oh, great to see you got the hang of condition variables so quickly! I think they can be very powerful...

shikokuchuo commented 1 year ago

mirai 0.8.2 is now on CRAN!

While there may not be another mirai version before your next release, I plan to put out a new version of nanonext (0.8.2) next week. This has some important low level improvements, particularly around memory management.

I do also plan to keep the dispatcher around (if not the default) in mirai as if nothing else it can be used standalone remotely. Plan is to benchmark any new implementation against this as we will have to see what ultimately scales well.

wlandau commented 1 year ago

Fantastic! That non-polling dispatcher really is a game-changer.

I do also plan to keep the dispatcher around (if not the default) in mirai as if nothing else it can be used standalone remotely. Plan is to benchmark any new implementation against this as we will have to see what ultimately scales well.

Yeah, a gradual transition away from the dispatcher makes sense. Happy to test if/when it is available in dev.