Promises in shiny app combined with future's or foreach's parallelization

[bguillod]

Hi,

First of all thanks for the great package, which is really useful!

This issue is more of a question than an bug in the package's code.

Essentially, I have a large shiny app that uses foreach and future's parallelization for some expensive operations, and I am wondering how to combine it with promises, because the latter also uses future's parallelization via multisession. My understanding is that when a promise is called in the shiny app, the calculation will be sent to a separate process using future's backend. So far so good. But now let's assume that the calculation itself uses parallelization with either foreach or future's multisession. Would this work? I guess the expensive calculation would be a promise but wouldn't be calculated with multiple cores (if I wrong, let me know how it would work and why!).

Therefore my question is: Is there a way to use promises when the calculations themselves are already parallelized?

The reason for turning to promises is to avoid other user's sessions to be blocked during the long operation; however, if the operation takes 1 minute using 10 cores (based on, say a foreach with %dopar% statement) but would take about 9-10 minutes using a single core, my assumption is that I need to choose one of the following options:

1. Using promises, in which case the user that triggers the calculation will now have to wait for 10 minutes rather than 1 minute, but other users won't notice the app is hanging.
2. Keep as is (i.e. without using promises), so the user will have his output within 1 minute but other users will all be waiting during this time.
3. Migrate these expensive calculation to a separate application (e.g., a plumber API that can do the calculation in parallel and return the results whenever they're ready, in which case it wouldn't compete with the shiny app's resources or at least its parallel backend).

I'm confident option 3 would work (promises also work for API calls that take long until they return a response, right?), but I'd prefer to avoid splitting the shiny app into several application, if possible. Is there any other way? Or do I misunderstand how promises/future/foreach work and they could be directly combined without any issue/competition for resources? If so, how?

Thank you very much for your support.

[bguillod]

So, I've gone through the details of the future package and found out that nested futures are possible. Therefore, I think my question has been answered - I can use future for promises and then that promise itself runs something with a multisession, using something like:

plan(list(tweak(multisession, workers = 3), tweak(multisession,  workers = 10)))

With this I close this issue.