search

dask / dask-blog

Dask development blog
https://blog.dask.org/
30 stars 35 forks source link

New post: The evolution of a Dask Distributed user #99

Closed jacobtomlinson closed 3 years ago

jacobtomlinson commented 3 years ago

The 2021 Dask Summit has yielded many excellent conversations.

One conversation I've had a few times is "there are so many deployment options, what is right for me?".

A common answer I seem to fall back to is describing the path that many Dask users walk, starting with one machine, reaching out to more, using and abusing compute platforms and eventually ending up with some centralized or managed service.

Describing that journey and the tools which empower each stage has been helpful in orienteering users and helping them find what tools to think about next.

Thought I'd write it up as a blog post.

GenevieveBuckley commented 3 years ago

Thanks for writing this post @jacobtomlinson

I think my two main suggestions are:

  1. Add link to the video recording of your tutorial once it's available on youtube
  2. Some more detail about how to judge if you've crossed the threshold for needing something like Dask Gateway over dask-jobqueue.
GenevieveBuckley commented 3 years ago

I've had a lot of conversations with people with deployment questions (and have been referring them to your tutorial where possible). I'm curious to know if the questions you got during the life sciences discussion were mostly answered by the kind of content in this post, or if they also had some trickier ones as well?

jacobtomlinson commented 3 years ago

I've had a lot of conversations with people with deployment questions (and have been referring them to your tutorial where possible). I'm curious to know if the questions you got during the life sciences discussion were mostly answered by the kind of content in this post, or if they also had some trickier ones as well?

In the life sciences workshop we kinda got distracted with the graph visualisation conversation. The only conversation we did have about deployment was one of the ones which inspired this post.

I think one theme which I'm hoping to address here is folks who ask "I'm new to Dask, but my org is not, where do I start?". Given the rate of Dask adoption I guess this will come up more and more.

jacobtomlinson commented 3 years ago

Thanks for the feedback @DPeterK and @GenevieveBuckley. I've made some changes to reflect that and hopefully expand on some of my points.

I'll also add a note to my calendar to come back and update with a YouTube link if this gets published before the Summit videos make their way up.

jacobtomlinson commented 3 years ago

I intend to merge this in 24 hours if there are no further comments.