Open fjetter opened 2 years ago
A) I really like the idea of an organized changelog and I agree that we can use PR labels to achieve that in a way that does not significantly increase the burden on maintainers.
B) I don't understand how the release cadence affects our ability to fix regressions. In my mind, we agreed to just release where we are at every 2 weeks. With that approach, if there is a regression and it takes a month to fix, then that regression would just be in two releases. I know that is not exactly what we have been doing, but it feels like a fine approach to me. If people want to be able to pin to a release that they know is a pretty good one, that takes us back to the conversation about "blessed" releases.
don't understand how the release cadence affects our ability to fix regressions. In my mind, we agreed to just release where we are at every 2 weeks
I believe two weeks are not sufficient to find and fix the regressions. Due to stability problems in dask/distributed we recently needed to delay multiple releases because we didn't manage to fix them in the given time.
I wanted to add some examples from how other Python projects handle characterizing release notes. Hope it helps.
We start categorizing our changelog in this or a similar way. I believe we could easily do this via a semi-automated process using labels.
We do this in LightGBM using the release-drafter bot.
Release-drafter automatically creates a draft release (only visible to maintainers), and then you have an opportunity to manually edit it before publishing it as an official release.
Alternatively, we could maintain a manual changelog
My personal opinion...asking individual contributors to update a changelog in files adds friction for not a lot of benefit. This is what cuml
does, for example, and I recall needing to fix merge conflicts in that project's CHANGELOG.md
4 times while I patiently waited for a small PR to be reviewed.
For a sort of middle ground between manual and automatic, prefect
asks contributors to add a small YAML file to a directory called changes/
. On each release, prefect
has some automation that sucks up these files and creates release notes, then deletes the files.
That approach is nice because it doesn't result in merge conflicts, but it's still a burden on maintainers to remind contributors to add such a file.
I like the above idea about the release-drafter bot. However, so far, we're not actually using githubs release feature but I guess this is not a deal breaker. Either we start using it or we simply use the bots to generate the markdown. IIUC, this would mostly be based on labels so we should start labeling everyhing properly either way.
My personal opinion...asking individual contributors to update a changelog in files adds friction for not a lot of benefit.
Yes, I pointed out pandas because they are using this and I am aware of that process. It is quite tedious, the results are OK, definitely better than what dask has to offer right now. However, the merge conflicts are very annoying, especially with long CI build times
don't understand how the release cadence affects our ability to fix regressions. In my mind, we agreed to just release where we are at every 2 weeks
I believe two weeks are not sufficient to find and fix the regressions. Due to stability problems in dask/distributed we recently needed to delay multiple releases because we didn't manage to fix them in the given time.
On the flip side releasing less frequently could mean more regressions accumulate in main
(as others are only testing the latest release) and we only find out about more of them later (once we finally release).
On the topic of automating release notes:
We've got a release scheduled for tomorrow (xref https://github.com/dask/community/issues/209). I'll propose we start improving the format of our changelogs by introducing the following sections for each release:
Thoughts on these categories? For tomorrow I'll just handle sorting commits into the above categories manually
👋 hello from a longtime dask/community
lurker.
@jrbourbeau instead of "Highlights" being a bulleted list of PRs, I want to suggest doing something like what XGBoost does for releases. Their release notes typically include a few paragraphs summarizing the content of the release, followed by bulleted lists linking to specific issues / PRs and arranged under headers.
For example: https://github.com/dmlc/xgboost/releases/tag/v1.5.0
That summary is typically written for the audience of "people installing and using XGBoost", and answers the question "why should I care about this release?".
I've really appreciated that because sometimes one feature like "better support for categorical data" is actually a composite of many separate PRs.
As a counterpoint, one advantage of bullet points is they are quicker to read for people that have a limited amount of time
I'd like to revive this conversation again. We've come a long way and have been using the current process (see above, https://github.com/dask/community/issues/199#issuecomment-990167373) for a rather long time now. I think that the changelog improved but is still not where it should be.
A couple of notes about the current state
Even I, as an expert, am sometimes struggling to parse our changelog to know where the notable changes are burried. I suspect that most end users do not even bother reading the changelog any longer.
On top of this, I don't think it is very obvious for users whether a change in dask/dask
or dask/distributed
affects them. I suspect that most users are using distributed as well but are interfacing mostly with dask/dask
. While there are technical reasons to have dedicated repos, this does not extend to having dedicated changelogs.
I would like to have a way to
dask.Array
user will not care about pyarrow strings but they may care about general improvements, e.g. new instrumentation.cc @jrbourbeau @jacobtomlinson @phofl
I agree that we need a better way of filtering out irrelevant stuff (and order things better). The changelog currently contains a lot of things that most folks don't care about.
Quick description how the pandas process works:
The changelog in pandas is updated by the PR author. This means that the author of a PR creates a whatsnew entry in one of the available categories. PRs are only added, if they changed anything that is user-visible, meaning that maintenance PRs or something similar won't show up in the changelog.
This has a couple of advantages:
How does the Pandas release process handle conflicts between adjacent changelog entries added by different PRs?
Good question.
We have a script that sorts entries alphabetically per category that's part of pre-commit. It does not remove all conflicts, but we rarely see conflicts since we introduced it.
I don't think we'll get around manual effort here and considering that the release manager (mostly but not exclusively @jrbourbeau ) isn't always aware of all changes, I'm +1 for a pandas-like approach. Still open for other proposals, of course
I took the changelog of the distributed 2021.10.0 and 2021.09.0 releases and tried to classify the changes into a few categories.
The 2021.09.0 changelog had 22 changes of which 14 (that's >60%) are maintenance related fixes as far as I can tell. Only very few actual user relevant fixes are in this release.
I see two issues
Changelog categorized distributed 2021.09.0
``` # Generic enhancements and regression - Overwrite worker plugins (GH#5248) Matthew Rocklin - Improved IPv6 dask-worker support (GH#5197) Walt Woods - Add support for diskless machines to system monitor (GH#5257) James Bourbeau # Active Memory Managment # Stability # Dashboard - Use non-histogram plots up to 100 workers (GH#5249) Matthew Rocklin # Documentation - Fix “then” -> “than” typo in docs (GH#5247) David Chudzicki - Fix typo (remove extra verb “creates”) in docs (GH#5244) David Chudzicki - Fix “fractiom” -> “fraction” typo in docstring (GH#5245) David Chudzicki - Fix “schedulers” -> “scheduler” typo in docs (GH#5246) David Chudzicki - # Tests, Maintenance, refactoring - Avoid during-iteration scheduler plugin changes (GH#5259) Doug Davis - Properly check for ipv6 availability (GH#5255) crusaderky - Replace atop with blockwise (GH#5289) James Bourbeau - Add pytest color to CI (GH#5276) James Bourbeau - Fix test_map and others (GH#5278) crusaderky - Use name argument with Scheduler.remove_plugin calls (GH#5260) Doug Davis - Downgrade to jupyter_client 6 (GH#5273) crusaderky - Migrate Security HTML repr to Jinja2 (GH#5264) Jacob Tomlinson - Migrate ProcessInterface HTML repr to Jinja2 (GH#5263) Jacob Tomlinson - Fix add_plugin warnings (GH#5267) Doug Davis - Add list around iterator in handle_missing_dep (GH#5285) Matthew Rocklin - Jupyter-client 7 compatiblity (GH#5286) Min RK - Remove GroupProgress scheduler plugin (GH#5256) James Bourbeau - Refactor scheduler plugins; store in a dictionary (GH#5120) Doug Davis ```Moving on to the 2021.10.0 release which was actually delayed for stability reasons, the summary looks like
This feels like a much more meaningful release, however, the changelog is a bit of a mess, I likely misclassified a lot.
Changelog categorized distributed 2021.10.0
``` # Generic enhancements and regressions (8 / 14.5%) - Increase latency for stealing (GH#5390) Florian Jetter - Log original exception upon compute failure (GH#5387) Florian Jetter - Add support for partial functions to iscoroutinefunction util (GH#5344) Michael Adkins - Use dask-spec for SSHCluster (GH#5191) Charles Blackmon-Luca - Update _cluster_info dict in __init__ (GH#5305) Jacob Tomlinson - Pickle exception and traceback immediately (GH#5338) Mads R. B. Kristensen - Ensure dask-worker and dask-scheduler pick up preload configuration values (GH#5365) James Bourbeau - Handle UCXNotConnected error (GH#5449) Peter Andreas Entschev # Active Memory Managment (5 / 9%) - Run multiple AMMs in parallel (GH#5339) crusaderky - Reinstate: AMM ReduceReplicas to iterate only on replicated tasks (GH#5341) crusaderky - Don’t schedule tasks to paused workers (GH#5431) crusaderky - Active Memory Manager to use bulk comms (GH#5357) crusaderky - Sync worker status to the scheduler; new ‘paused’ status (GH#5330) crusaderky # Stability (8 / 14.5%) - Ensure resumed flight tasks are still fetched (GH#5426) Florian Jetter - Ensure reconnecting workers do not loose required data (GH#5436) Florian Jetter - Resolve work stealing deadlock caused by race in move_task_confirm (GH#5379) - Fix regression where unknown tasks were allowed to be stolen (GH#5392) - Do no attempt to fetch keys which are no longer in flight (GH#5160) Florian - Fix zombie worker tasks after missing transition (GH#5316) Florian Jetter - Workers submit a reply to the scheduler if replica removal was rejected (GH#5356) Florian Jetter - Worker state machine refactor (GH#5046) Florian Jetter # Dashboard (1/ 1.8%) - Add scroll to dashboard dropdown (GH#5418) Jacob Tomlinson # Security (1 / 1.8%) - Pass host through LocalCluster to workers (GH#5427) Jim Crist-Harif # Documentation (5 / 9%) - AMM high level documentation (GH#5456) crusaderky - Simple SSHCluster example (GH#5349) Ray Bell - Use new Dask docs theme (GH#5391) Jacob Tomlinson - Enhance AMM docstrings (GH#5340) crusaderky - Fix typo in client side example foundations.rst (GH#5336) Genevieve Buckley # Tests, Maintenance, refactoring (27 / 49%) - Revisit Scheduler.add_plugin / Scheduler.remove_plugin (GH#5394) crusaderky - Provide stack for suspended coro in test timeout (GH#5446) Florian Jetter - Use pip install . instead of calling setup.py (GH#5442) Matthias Bussonnier - Type annotations for Worker and gen_cluster (GH#5438) crusaderky - Mark test_gather_dep* as xfail (GH#5432) crusaderky - Remove zict-related skips (GH#5429) James Bourbeau - Fixes async warnings in UCX tests (GH#5396) Peter Andreas Entschev - Enable mypy in CI 2/2 (GH#5348) crusaderky - Rewrite test_client_timeout (GH#5397) crusaderky - Fix flaky test_WorkerPlugin_overwrite (GH#5398) crusaderky - Add coverage badge to README (GH#5382) James Bourbeau - Mark test_stress_creation_and_deletion as xfail (GH#5393) James Bourbeau - Mark test_worker_reconnects_mid_compute* tests as flaky (GH#5378) James Bourbeau - Remove pytest.mark.repeat from test_prometheus_collect_task_states (GH#5376) James Bourbeau - Add code coverage (GH#4670) James Bourbeau - Mark distributed/tests/test_client.py::test_profile_server as flaky (GH#5375) James Bourbeau - Enable mypy in CI 1/2 (GH#5328) crusaderky - Use Dask temporary file utility (GH#5361) James Bourbeau - Avoid deprecated random set sampling (GH#5360) James Bourbeau - Add check for unsupported NVML metrics (GH#5343) Charles Blackmon-Luca - Add pre-commit to environments (GH#5362) Ray Bell - Generate Cython HTML annotations (GH#5321) crusaderky - Bump RAPIDS_VER for gpuCI (GH#5358) Charles Blackmon-Luca - fsspec and s3fs git tips are incompatible (GH#5346) crusaderky - Worker State Machine Refactor: clean up dead handlers (GH#5359) crusaderky - Fix test_many_Progress and others (GH#5329) crusaderky - Run pyupgrade in CI (GH#5327) crusaderky ```Original changelogs, see http://distributed.dask.org/en/stable/changelog.html#id2
I didn't go through the dask/dask changelog but it is similarly unorganized and I am struggling to decipher what is what and what would be relevant for me if I was a user.
I would like to propose
A) We start categorizing our changelog in this or a similar way. I believe we could easily do this via a semi-automated process using labels. We might even consider dropping the entire maintenance section.
Alternatively, we could maintain a manual changelog, e.g. similar to pandas. That's more effort and a frequent source for merge conflicts in my experience but that would also allow us to provide more context information for a change.
Either way, I believe a more organized changelog would provide a lot of value for users.
B) This will be a more lengthy discussion but I would like to start a conversation about our release cadence. In particular with stability problems all over the place, the two weeks cadence feels a bit rushed sometimes to fix known regressions and properly test them.
Edit: Let's move discussion around B to https://github.com/dask/community/issues/200