search

ropensci / targets

Function-oriented Make-like declarative workflows for R
https://docs.ropensci.org/targets/
Other
916 stars 74 forks source link

Avoid polling in crew-powered pipelines #1183

Closed wlandau closed 9 months ago

wlandau commented 10 months ago

Like the other pipeline algorithms in targets, https://github.com/ropensci/targets/blob/main/R/class_crew.R uses polling and exponential backoff to balance latency with CPU usage. nanonext condition variables may help us avoid polling altogether.

This issue depends on https://github.com/wlandau/crew/issues/108, which depends on https://github.com/shikokuchuo/nanonext/discussions/20 and https://github.com/shikokuchuo/nanonext/discussions/21.

wlandau commented 9 months ago

Started this in branch 1183. The basic idea is there, but I think we need to throttle scaling using a time interval configurable by the user. I think we can create the infrastructure for this in crew, decouple it from any of the existing structures, and make it interact with the controller through the scale argument of push(), pop(), and wait().

wlandau commented 9 months ago

Implemented.