tomwhite
commented
1 month ago That's a great suggestion. I'll sketch out something here to start with.
A Cubed executor is a subclass of DagExecutor:
The execute_dag method is responsible for taking a Cubed plan - which is a DAG of operations and arrays - and turning it into sets of parallel tasks that it runs in stages. It is also responsible for issuing callbacks whenever an operation starts or a task finishes, which the client uses to update progress bars, and record statistics.
The simplest example of a Cubed executor is the SingleThreadedExecutor which runs tasks sequentially:
In practice backends will have the following characteristics that make them suitable targets for Cubed:
- Parallel. The ability to efficiently run multiple tasks in parallel. (The task inputs are generated by the the
pipeline.mappableiterator.) - Code distribution. The ability to run arbitrary code in the remote process via some distribution mechanism. (The task function is
pipeline.function.) - Memory guarantees. The backend should guarantee that the task gets a certain (configurable) amount of memory.
- Retries. The backend should have some way of retrying a task if it fails.
- Timeouts. Tasks should fail after a certain amount of time, so they can be retried.
- (Optional) Straggler mitigation. Very slow tasks are detected and retried with a backup task so as to not slow down the entire computation. Most backends do not have support for this, but it can be implemented as a part of the executor.
These features (and a few more) are discussed in #276, which also has a table showing which executor has each feature.
Some of the executors in Cubed use asyncio APIs to call the backend (e.g. Modal, Dask, local threads and processes), which makes implementing some of these features easier since the code can be shared (e.g. backup tasks). However, backends do not have to offer an asyncio API to be integrated with Cubed (e.g. Lithops).
All the executor implementations can be found in cubed.runtime.executors.
For testing, new executors can be conditionally added to the ALL_EXECUTORS list - if the backend dependency is present.
There are a lot of parallel frameworks that Cubed Plans could be converted to (e.g. #499). We have executors for dask & beam, but instead of trying to write more executors ourselves it would be nice to just more clearly document how to create a new executor so that others can contribute.