Open TomNicholas opened 1 month ago
That's a great suggestion. I'll sketch out something here to start with.
A Cubed executor is a subclass of DagExecutor
:
The execute_dag
method is responsible for taking a Cubed plan - which is a DAG of operations and arrays - and turning it into sets of parallel tasks that it runs in stages. It is also responsible for issuing callbacks whenever an operation starts or a task finishes, which the client uses to update progress bars, and record statistics.
The simplest example of a Cubed executor is the SingleThreadedExecutor
which runs tasks sequentially:
In practice backends will have the following characteristics that make them suitable targets for Cubed:
pipeline.mappable
iterator.)pipeline.function
.)These features (and a few more) are discussed in #276, which also has a table showing which executor has each feature.
Some of the executors in Cubed use asyncio APIs to call the backend (e.g. Modal, Dask, local threads and processes), which makes implementing some of these features easier since the code can be shared (e.g. backup tasks). However, backends do not have to offer an asyncio API to be integrated with Cubed (e.g. Lithops).
All the executor implementations can be found in cubed.runtime.executors
.
For testing, new executors can be conditionally added to the ALL_EXECUTORS
list - if the backend dependency is present.
There are a lot of parallel frameworks that Cubed Plans could be converted to (e.g. #499). We have executors for dask & beam, but instead of trying to write more executors ourselves it would be nice to just more clearly document how to create a new executor so that others can contribute.