Open aeroaks opened 1 year ago
The current DaskExecutor
is quite old, and works by attempting to translate the Dagster DAG in to a Dask DAG and submit it to Dask for orchestration. Working this way means it does not support many Dagster features that are managed at the executor/orchestration layer. This is one reason that the dask executor is not generally recommended.
Thanks, Hopefully it helps others looking for it.
So given that I would be using Dask in my compute, and defining Dask as a resource is quite useful in Dagster, which is the preferred executor for this approach? Also, in general, which executor is preferrable?
The default multiprocess/in-process executor is probably the generally preferable choice https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#controlling-job-execution
What's the use case?
We have computation tasks which we execute by using dask. The best approach that I understood from blogs and documentation is to define the cluster as a resource and use in the assets.
With the above tasks,
dagster-dask
as an executor seems not to be an option. What kind of executor is then generally used in such scenario?What is the reason a dask cluster cannot be used for both as
executor
and for parallelising the computation in a task?Ideas of implementation
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.