Use inboxes in parallel linear solver algorithms?

[nilsvu]

Feature request:

Investigate whether the parallel linear solvers that are used in the elliptic solver should switch to an inbox control flow. Currently they do this:

• An element action contributes to a reduction to the "residual monitor" singleton component, then terminates the algorithm.
• Once the reduction is complete, the residual monitor broadcasts back to all elements. They proceed with the algorithm.

Instead, perhaps they should do this:

• An element action contributes to the reduction. The next element action monitors its inbox in the is_ready method.
• Once the reduction is complete, the residual monitor broadcasts back to all elements, contributing to the inboxes of the elements.

I am not sure at this point if the inbox control flow is 1) possible and 2) better. Here's a few notes and potential issues:

• The singleton residual monitor has to broadcast doubles into all elements' inboxes. That may or may not be currently possible.
• In the current control flow the linear solver iteration is triggered from a single iterable action on the element. What follows is bouncing reductions and simple actions back and forth until the iteration is complete. With the inbox control flow the linear solver would have to expose an action list, instead of one perform_step action.
• I am not clear on the advantages and disadvantages of either control flow to decide which one is better.

Component:

• [x] Code
• [ ] Documentation
• [ ] Build system
• [ ] Continuous integration

[nilsvu]

In a call with @nilsdeppe in response to #2425 we identified a few issues and potential solutions:

• Issue: The current control flow where reductions (essentially simple actions) are called on the residual monitor singleton has the disadvantage that the code in ResidualMonitor.hpp (where the singleton component is defined) provides no indication which actions are going to be invoked on it. Currently these actions are defined in ResidualMonitorActions.hpp, but they are only associated with the residual monitor component by the name of the file and the fact that they are going to be invoked on the residual monitor.
• Potential solution: Reduce to a generic action that calls receive_data on the residual monitor. Convert simple to iterable actions on the residual monitor.