search

rstudio / promises

A promise library for R
https://rstudio.github.io/promises
Other
201 stars 19 forks source link

Support for extending promises #103

Closed wlandau closed 5 months ago

wlandau commented 8 months ago

mirai and crew are new packages for distributed distributed computing similar to future. They can both leverage their parallel computing capabilities to create custom promise objects, as described at https://shikokuchuo.net/mirai/articles/promises.html for mirai and at https://wlandau.github.io/crew/articles/promises.html for crew (also c.f. https://github.com/wlandau/crew/issues/147, https://github.com/wlandau/crew/pull/149). The promises seem to work, but they do rely on polling with later::later(), which may not be as efficient or as kind to the CPU as a truly event-driven approach.

It would be great if promises could provide a general solution for triggering callbacks asynchronously, or if the documentation could describe how to implement a successful event-driven pattern. I am not sure if anything like this currently exists, but there may be possibilities using the nanonext package. nanonext is a low-level R client for the NNG messaging library, which in turn provides network protocols, asynchronous I/O, and perhaps most importantly for this issue, synchronization primitives such as condition variables which can be configured to respond to local network events.

This thread is part of a follow-up on recent conversations with @shikokuchuo and @jcheng5.

shikokuchuo commented 7 months ago

I think I've figured a way of implementing this completely event-driven approach by polling read activity on file descriptors internal to NNG sockets – plugging into R’s event handlers similar to later. I’ve touched on this with @wlandau already, @jcheng5 happy to discuss with you whenever you wish.

jcheng5 commented 5 months ago

Ended up discussing this with @wlandau and @shikokuchuo extensively over email and Zoom, my take on it is this draft PR on nanonext. The TL;DR is that it's not really something for the promises package itself to handle, rather, each execution engine needs its own way of detecting relevant events and can use the {later} package to either poll (if necessary) or to move notifications from background threads to the main thread (preferred).