Longer term planning!

[martindurant]

There have been some conversations going on around how to improve communication among the dev team, to be able to prioritize work and actually plan for and execute larger plans. The idea is to reduce friction, and ideally we get consensus on a few action items, to move dask forward

Some suggestions of things that might help. This list will be expanded as thoughts come in.

• Project boards (on github)? Collection of issues reflecting a particular larger work stream, showing the status of the individual issues (on github), who is assigned, etc. Comes with useful github automation, but still needs someone to gather issues and manage the board.
• Milestones? Similar to project boards, a specific label grouping some issues together and, in theory, to be done before some larger action like a merge or release.
• Dask improvement plans? A formal mechanism for submitting designs for bigger implementation changes.
• Focus groups? Small temporary groups of devs drawn together to tackle a larger problem
• Topic champion? Someone who is responsible for execution of a larger plan. Not necessarily the person who writes most of the code, but an organisational figure who knows the current state of things. This would probably be a "product manager" in a commercial context
• ask NumFOCUS! They are our helper non-profit org, and surely have seen and maybe shepherded other OSS projects into long-term sustainability.
• Issue severity requirements? Perhaps we just need more labels and to raise more serious problems at the devs' meeting. This is the simplest solution, but perhaps doesn't really change anything

Some examples for discussion follow. How might the ideas above have helped or yet help these situations? These all have long conversations in issues that could be linked.

• Flaky CI tests, an ongoing source of friction. "Flaky", "skip" and "xfail" tests, as well as CI runs that sometimes fail, are a sign of work that needs to be done and reflect poorly on our project.
• HLG refactor. This was a major change in dask and continues to undergo big rewrites, and conversations about replacing it altogether.
• Worker state refactor. This was/is largely done by a focus group within Coiled.
• cythonization. A large code project across years to attempt a faster scheduler, eventually reverted due to lack of clear large benefit, dev team enthusiasm and complexity.

@dask/maintenance , all encourages to have a say. Time is also set aside at the end of this week's community call.

[hayesgb]

Thanks for posting this @martindurant !

I like the idea of starting with a project roadmap, and then having that flow down into organized projects as boards on GH. Giving better visibility into the direction of dask, timing for new features, would (I think) improve adoption. It also wouldn't hurt to provide better clarity on how new users (or organizations) can make proposals for changes and/or new capabilities -- this would likely fall under the heading of Improvement Proposals.

[MrPowers]

I think the Spark project improvement proposals (SPIP) process works well. It's a great way to give a contributor a signal if a big individual undertaking is worth the effort.

I'd like to make a detailed dask-iceberg proposal and get feedback from the Dask community before doing a ton of development work. Perhaps I can make a "Dask project improvement proposal" via an issue in dask/dask to trial that process? Thoughts @martindurant?

[jcrist]

I'd like to make a detailed dask-iceberg proposal and get feedback from the Dask community before doing a ton of development work. Perhaps I can make a "Dask project improvement proposal" via an issue in dask/dask to trial that process?

Note there was an earlier attempt at a repo for large project design docs earlier (https://github.com/dask/design-docs), which may overlap a bit with this.

[hayesgb]

One action item from the June 2022 Dask Developers meeting was to form a subgroup tasked with identifying proposals to improve product/project management for the Dask organization, where appropriate. This group is comprised of leaders of technical teams within organizations committed to the development of Dask, who can also commit resources to larger efforts. This team met on June 7, 2022 to discuss areas of interest, align on intended outcomes, and identify next steps.

Attendees:

• @hayesgb
• @mmccarty
• @martindurant
• @jsignell
• @quasiben

Some current areas of interest that were discussed included:

• I/O
• dask-awkward
• General usability Issues, specifically dask/dask
• Improving the user experience / performance on distributed ETL workloads (both GPUs & CPUs)
• Simplified deployments including dask-kubernetes including heterogeneous cluster environments
• serialization & buffer management
• cross-project API dependencies
• HighLevelGraphs
• Stability, specificallydask/distributed

There are instances where multiple organizations have a shared interest in improving Dask's performance. These shared interests represent potential points for collaboration, which could be improved by more rigorous project/product management practices. In cases where efforts are small, or resources for the work reside within a single organization, the added complexity likely does not provide incremental value.

Product Roadmaps

The group discussed ways to leverage learnings from the Python community, including product roadmaps like those of Xarray and NumPy. The consensus was that these roadmaps are formatted at a suitable level of abstraction, and creating such a roadmap for Dask is something that should be evaluated.

Project Boards / Design Documents

There is agreement that the added rigor of design documents and project boards would improve visibility into technical deliverables, timelines, and commitments, but acknowledgement that the rigor comes with flexibility tradeoffs. The group discussed a range of options from simple design documents with a required technical review by stakeholders that could be archived as part of larger project boards to formal Enhancement Proposals similar to NEPs.

• [x] @mmccarty will reach out to the NumPy organization about how to learn from their experiences with NEPs for both product & project management

There is consensus that projects involving resources spanning multiple organizations would benefit from a shared project board, assuming a resource has responsibility to maintain the board. Assuming it were properly maintained, there is an interest in testing this approach using either dask-awkward or HighLevelGraphs.

• [x] @hayesgb -- review SPIPs vs NEPs vs Xarray Enhancement Proposals. Are there other approaches that should be considered?
• [x] @hayesgb -- schedule a follow up with this team to review

[hayesgb]

There are number of concurrent discussions underway on this topic. The initial proposal would be to:

1. Create a shared epic-level project board for the Dask org.
2. Add a new meeting to the Dask org for team leaders to discuss priorities.

The intent would be to enable an ongoing conversation about priorities for all teams engaged in Dask development.

[martindurant]

I did create one project, as promised: https://github.com/orgs/dask/projects/3 . Not been noticed so far.

[jakirkham]

Actually did look at that project (after it was posted in the Plasma PR) 🙂

[jacobtomlinson]

@hayesgb I think it would also help to have a design doc (is work spans multiple repos) or high-level issue (if work is contained to a single repo) to capture the work.