SCALE-MS / scale-ms

SCALE-MS design and development
GNU Lesser General Public License v2.1
4 stars 4 forks source link

move Worker management to separate CPI operation #377

Open eirrgang opened 1 year ago

eirrgang commented 1 year ago

Let RuntimeManager mediate Raptor resource acquisition.

RPExecutor merely represents the allocated resources, and does not directly implement the resource management. This allows RPExecutor to be responsible for providing the concurrent.futures.Executor interface, which isn't particularly friendly to the asyncio protocols. For regular scalems usage, RPExecutor should be used in a non-main thread via the scalems asyncio utilities.

  1. Acquire the Raptor master task through the RuntimeManager.
  2. Acquire the Worker(s) through CPI call to the RuntimeManager.

Isolate RPExecutor concurrent.futures.Executor support from asyncio support (avoid blocking the event loop by avoiding event loop usage in the main implementation)

We also need this so that we can restore and normalize the "stop" command to shut down everything cleanly and expeditiously. In too many cases right now, tests take an improperly long time because of various timeouts.

Supports #335