Introduce new runtime protocol

[eirrgang]

Introduce scalems.radical.executor.RPExecutor, a SCALEMS concurrent.futures.Executor, and some context manager utilities. Prepare to use RuntimeManager to coordinate potentially competing Raptor sessions. Reorganize resource management scoping.

• Decouple RP Session from Raptor task. Remove RuntimeSession.raptor.
• Let RPDispatchingExecutor manager the Raptor task separately.
• Begin to use some of the new Raptor API.

Split get_scheduler into an asyncio version and a threaded version. The coroutine version will go away as we migrate to the new RP management protocol.

Minor updates to the worker description. Needs further revision to allow expression of multiple workers and to be completely removed from the Master configuration.

Establish a more complete framework for RPExecutor. Use one thread to manage the Raptor session and process the sequence of submit() calls and additional control messages. submit() is not yet fully implemented.

Expand the RuntimeManager acquire()/release() machinery. Add timeout support.

We diverge between the abstract scalems.execution.RuntimeManager.cpi() and the new CPI servicer based on whether/where the Raptor task lives. Note that only something with access to a Raptor session is able to fully service CPI calls for the foreseeable future.

TODO(#335):

• [ ] Migrate or remove old code to new stack.
• [ ] Resource acquisition and release should be moved closer to (or into) the manage_raptor thread.
• [ ] Implement work-load aware Worker provisioning. Master should not directly start Worker(s) in its main script or include Worker requirements in its config file.

TODO(#345):

• [ ] implement submit()
• [ ] rigorously support Future.cancel()

TODO(#23):

• [ ] Define the Compute Provider Interface and identify the participants.

TODO(#277, #299, #338):

• [ ] Better tools for access task results.
• [ ] Agent-side task chaining.

TODO(#229, #340):

• [ ] make sure RuntimeManager shuts down Executors
• [ ] support variously aggressive Raptor shut down.

None on the queue should only be a last resort. All controlled shut down should use an explicit stop command task, but we either need at least two levels of urgency in the stop command or we need to separately send control messages to cancel unresolved tasks.

Ref #335

[codecov[bot]]

Codecov Report

Patch coverage: 71.31% and project coverage change: -2.08 :warning:

Comparison is base (ded7fef) 80.60% compared to head (35cdae0) 78.53%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #362 +/- ## ========================================== - Coverage 80.60% 78.53% -2.08% ========================================== Files 35 36 +1 Lines 3543 3987 +444 Branches 531 608 +77 ========================================== + Hits 2856 3131 +275 - Misses 504 643 +139 - Partials 183 213 +30 ``` | [Impacted Files](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS) | Coverage Δ | | |---|---|---| | [src/scalems/radical/manager.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvcmFkaWNhbC9tYW5hZ2VyLnB5) | `67.53% <64.40%> (-1.60%)` | :arrow_down: | | [src/scalems/radical/executor.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvcmFkaWNhbC9leGVjdXRvci5weQ==) | `66.78% <66.78%> (ø)` | | | [src/scalems/radical/runtime.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvcmFkaWNhbC9ydW50aW1lLnB5) | `79.71% <77.77%> (-7.53%)` | :arrow_down: | | [src/scalems/radical/raptor/\_\_init\_\_.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvcmFkaWNhbC9yYXB0b3IvX19pbml0X18ucHk=) | `83.91% <90.56%> (-3.59%)` | :arrow_down: | | [src/scalems/execution.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvZXhlY3V0aW9uLnB5) | `77.21% <100.00%> (+0.31%)` | :arrow_up: | | [src/scalems/radical/session.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvcmFkaWNhbC9zZXNzaW9uLnB5) | `77.50% <100.00%> (-0.34%)` | :arrow_down: | | [src/scalems/radical/task.py](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS#diff-c3JjL3NjYWxlbXMvcmFkaWNhbC90YXNrLnB5) | `75.26% <100.00%> (+0.08%)` | :arrow_up: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/SCALE-MS/scale-ms/pull/362/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SCALE-MS)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.