Open mrocklin opened 4 years ago
Also, I'm super-happy to see that Dask was helpful here :)
@mrocklin that schedule selection is in the documentation and shows how to switch the scheduler interactively.
@mrocklin - yes our reason for having our own context manager was because for large datasets, having the multithreaded/process scheduler by default let to memory issues, so we wanted multi-threading/processing to be opt-in. If there is a better way to do this, please let us know!
I notice in https://spectral-cube.readthedocs.io/en/latest/dask.html that this package takes ownership of which Dask scheduler to use
It is somewhat atypical for packages that use Dask to control Dask configuration. Instead they typically let Dask handle this, which makes it easy for users to combine the use of many packages together.
I'm guessing that you've chosen to own this yourself because you prefer for cubes to have the single-threaded scheduler by default and you weren't sure how to do this without storing this preference as state somewhere. I'm curious about this choice as well.