Refactor `parallel` cluster code to allow packages to provide alternate/custom backends

[gmbecker]

Endorsed by Luke.

Currently (AFAIK) functionality is spread across the parallel package and some hardcoded links to functionality in the snow package.

• [ ] - Refactor cluster code to make use of alternate backends
• [ ] - Allow packages to register backends they implement (?)
• [ ] - API for querying which backends are available based on installed/loaded packages (??)

[gmbecker]

Functionality needed:

cluster object which can:

• [ ] create node (start process with work, ie submit)
• [ ] blocking retrieve one
• [ ] blocking retrieve first from set (block on set)
• [ ] stopCluster (tear down all nodes)

[gmbecker]

@shikokuchuo I believe you'er the right one here. Last I heard you had something that conceptualloy works but may need some previously internal generics in parallel exported. Is that correct?

[shikokuchuo]

Conceptually yes, actually not yet. The unexported generics I discussed with Luke this morning and it shouldn't be an issue for them to be exported (just a historical artefact that they weren't). Once I have something that's working I'll detail what exactly needs to be changed.

[gmbecker]

Ultimately its up to @ltierney but my 2c is to remember that parallel is a very well established existing system. Pure extensions (such as exporting things that weren't before, or adding a package backend registration system eventually) may be doable, but any backwards incompatible changes would likely need really strong, general justifications to be considered.

[shikokuchuo]

Bugzilla issue 18587 filed. https://bugs.r-project.org/show_bug.cgi?id=18587

[ltierney]

Thanks! Let's continue working on this on bugzilla.