dask tutorial

[orbeckst]

introductory tutorial on dask

• background: task graph and execution
• parallelizing with dask
• using distributed (setting up a dask cluster)
• using the bokeh status web interface

[orbeckst]

@VOD555 please coordinate with @dotsdl for your PMDA tutorial so that the two tutorials fit together.

[orbeckst]

With the changes in https://github.com/MDAnalysis/pmda/pull/66 and the new way to globally define the scheduler it becomes even more useful to have a solid introduction to dask.

@dotsdl please can you make sure that the new idiom to set the scheduler, which @kain88-de explained in https://github.com/MDAnalysis/pmda/pull/66#issuecomment-434417794, is also covered?

[orbeckst]

With PMDA now using dask's preferred way to select a scheduler, ~we now have to teach people to explicitly set the multiprocessing or distributed scheduler and not the default threaded scheduler.~ Threaded does not work well with our largely Python based code.

EDIT: As @kain88-de says below we still use multiprocessing as default. It will still be useful to show how to set up distributed and talk about the schedulers, though. Also worthwhile pointing out that the threaded scheduler will not work well with PMDA.

[kain88-de]

If people don’t change the scheduler setting we pick multiprocessing. We should add to the pmda docs that thread scheduler doesn’t work with pmda.

On Thu 1. Nov 2018 at 23:51, Oliver Beckstein notifications@github.com wrote:

With PMDA now using dask's preferred way to select a scheduler, we now have to teach people to explicitly set the multiprocessing or distributed scheduler and not the default threaded scheduler. Threaded does not work well with our largely Python based code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MDAnalysis/WorkshopHackathon2018/issues/20#issuecomment-435213632, or mute the thread https://github.com/notifications/unsubscribe-auth/AEGnVlxf_NXdiKjnAX9KYZN5GhX4Me9rks5uq3sEgaJpZM4Xz1L2 .

[orbeckst]

Yes, you're right.

(I didn't think of adding ad note regarding the threaded scheduler in https://github.com/MDAnalysis/pmda/issues/78.)