kmpaul
commented
5 years ago @jhamman can probably amend or append to this...
5 Reasons We Use Dask at NCAR
In my mind, these are the 5 best things that we get from Dask, here at NCAR:
- Cost: Dask is free and open source, which means we at NCAR do not have to rebalance our budget and staff to address the new immediate need of data analysis tools that we have failed to invest in over the last decade as Big Data problems have reared their ugly head.
- Scalability: Dask provides the scalability we need to deal with the massively large datasets that our existing (primarily serial) data analysis tools cannot.
- Simplicity: By abstracting the parallelism away from the user/developer, our analysis tools can be written by computer science non-experts, such as the scientists themselves, meaning that our software engineers can take on a more supporting role than a leadership role.
- Inspiration: Dask is more than just a tool to NCAR; it is a gateway to thinking about a completely different way of providing computing infrastructure to our users. Dask opens up the door to cloud computing technologies (such as elastic scaling and object storage) and makes us rethink what an HPC center should really look like!
- Collaboration: Dask, and members of the Dask development team, have been critical collaborators with NCAR staff, helping us learn a new way of conducting business in the open source software world.
3 Things That Could Be Better
I probably need some help developing this list, as this list is short and very self-centered. In fact, some of these things could actually already have solutions...which I would be happy to hear about.
- Batch Launching: For many at NCAR, launching a parallel job means 2 things: (1) writing a PBS or SLURM batch script that (2) runs your parallel application with
mpirunormpiexec. While Dask is fantastic for interactive jobs with Jupyter Lab/Notebooks, a great deal of our workflows at NCAR are still (and probably will be for the foreseeable future) chained batch-job workflows. To fit into those workflows (without extra modifications), a single batch job must be submitted that launches the Scheduler, Workers, and the Client (and application, obviously) together. - Coarse-Grained Diagnostics: Dask provides a number of profiling tools that give you diagnostics at the individual task-level, but (as far as I know), there is no way of analyzing or profiling your Dask application at the (for lack of a better term) "unchunked" task-level (or the "operator-level", if that makes sense). The best way I know of getting coarse-grained diagnostics is to scale your application down to a much smaller size so that you can run it serially. However, some applications just don't show problems, or at least do not indicate the truly problematic step, when scaled back.
- Scheduler Profiling: When large graphs get created, there can be long delays before any sign of execution shows up on the dashboard. I can only assume that this is because of work that the Scheduler is doing that is not being displayed. It would be nice to get some information about what the Scheduler is doing at these moments. Can this be done today?
...@jhamman, is there anything you would add?
mrocklin
guillaumeeb
dharhas
willirath
Hi All,
I'd like for a group of us to write a blogpost about using Dask on supercomputers, including why we like it today, and highlighting improvements that could be done in the near future to improve usability. My goal for this post is to show it around to various HPC groups, and to show it to my employer to motivate work in this area. I think that now is a good time for this community to have some impact by sharing its recent experience.
cc'ing some notable users today @guillaumeeb @jhamman @kmpaul @lesteve @dharhas @josephhardinee @jakirkham
To start conversation, if we were to structure the post as five reasons we use Dask on HPC and five things that could be better, what would be those five things? I think it'd be good to get a five-item list from a few people cc'ed above, then maybe we talk about those lists and I (or anyone else if interested) composes an initial draft that we can then all iterate on?