radio-astro-tools / spectral-cube

Library for reading and analyzing astrophysical spectral data cubes
http://spectral-cube.rtfd.org
BSD 3-Clause "New" or "Revised" License
98 stars 65 forks source link

Use Dask configuration for schedulers #636

Open mrocklin opened 4 years ago

mrocklin commented 4 years ago

I notice in https://spectral-cube.readthedocs.io/en/latest/dask.html that this package takes ownership of which Dask scheduler to use

>>> with cube.use_dask_scheduler('threads'):  
...     cube.max()

It is somewhat atypical for packages that use Dask to control Dask configuration. Instead they typically let Dask handle this, which makes it easy for users to combine the use of many packages together.

I'm guessing that you've chosen to own this yourself because you prefer for cubes to have the single-threaded scheduler by default and you weren't sure how to do this without storing this preference as state somewhere. I'm curious about this choice as well.

mrocklin commented 4 years ago

Also, I'm super-happy to see that Dask was helpful here :)

keflavich commented 4 years ago

@mrocklin that schedule selection is in the documentation and shows how to switch the scheduler interactively.

astrofrog commented 4 years ago

@mrocklin - yes our reason for having our own context manager was because for large datasets, having the multithreaded/process scheduler by default let to memory issues, so we wanted multi-threading/processing to be opt-in. If there is a better way to do this, please let us know!