Not to propose a new direction in development, but this does get me thinking big-picture about the mirai/crew integration. mirai has a dispatcher as a separate process which frees the user session. But in crew, because of auto-scaling and transient workers, the main R process needs to be an event loop that continuously calls push(), pop(), or wait() (similar to Gabor's task queue idea for callr). The crew event loop is even more important now that there is throttling. All this makes me wonder if crew itself could manually run an iteration of dispatch each time e.g. pop() is called, rather than overburdening the mirai dispatcher. It might not be feasible due to the scope and intent of mirai, but I keep wondering about this.
Not to propose a new direction in development, but this does get me thinking big-picture about the
mirai/crewintegration.miraihas a dispatcher as a separate process which frees the user session. But increw, because of auto-scaling and transient workers, the main R process needs to be an event loop that continuously callspush(),pop(), orwait()(similar to Gabor's task queue idea forcallr). Thecrewevent loop is even more important now that there is throttling. All this makes me wonder ifcrewitself could manually run an iteration of dispatch each time e.g.pop()is called, rather than overburdening themiraidispatcher. It might not be feasible due to the scope and intent ofmirai, but I keep wondering about this.Originally posted by @wlandau in https://github.com/wlandau/crew/issues/76#issuecomment-1551827347