Consider pivoting away from the changelog file to something more merge-friendly?

[bitprophet]

This is just spitballing so I don't forget N months from now.

The current curated-changelog-file approach works well enough, but has frustrating side effects I've never found solutions for, such as all the merge conflicts when doing a 'merge-up' from old branches through newer ones. These are usually easy to resolve, but it's still irritating gruntwork/friction, removal of which was the whole point of Releases...

That's on top of the issues endemic to this approach, such as the need for complex logic to figure out which release lines a given issue is, or should be, included within. When the project began, this work seemed a necessary tradeoff vs "use your git commit messages" or "100% manually edited NEWS/CHANGES files" (I contrasted these approaches in http://bitprophet.org/blog/2013/09/14/a-better-changelog/).

I've recently been acting as a sounding board for some colleagues trying to grow an early-stage internal project into a more mature, public-facing one, and was pitching them on using Releases or some variant thereof.

They like the overall approach but are, understandably, leery about the merge-conflict thing. What's an annoyance for a single maintainer like myself would become unreasonable friction for a full team (especially one currently used to the low-friction anything-goes rolling-non-release approach.)

What follows are some ideas about alternatives that came out of that discussion.

---

The core idea is to replace the changelog file with some other 'format' or data source that is more resistant to merge conflicts, and which is dead simple for developers to interact with (two arguably separate concerns.)

The biggest class of approach being considered is to go back to the "git commits" style of changelog data source:

• Originally, the idea was to literally use the old "every commit is a changelog-worthy entry" concept, with a release-oriented script which crawls Git history from each release-line HEAD being released to identify the appropriate changes for then next release from that branch.
  • This is in part because many team members work that way naturally - they only make one big commit per overall change (i.e. they tend to stick together commit and push actions - one commit per push.)
• I point out that this only works if you can enforce that pattern (team members more like me, which make many small commits, would find this hard to deal with) and it's arguably an antipattern - smaller, more granular commits make history spelunking and code review easier, not to mention the benefits of checkpointing your own WIP.
• However, there are two ways to work around this, using git features I'd forgotten about til now:
  • Continue requiring single commits, but use squashed merges to allow individual devs to use whatever commit strategy they prefer, within their feature/fix/etc branches. Github can even let you mandate that PRs use the squash strategy, so as long as the workflow of "which form field becomes the commit message?" works out, could be OK.
  • I do worry a bit about how PR descriptions sometimes want to serve a different purpose than a changelog entry, assuming the former gets used for the commit message; but probably not a major deal.
  • And of course, this does kill the 'better history' angle. Tradeoffs, yo.
  • Allow arbitrary commit styles, and only pull out specifically-annotated (think how CI systems look for [ci skip] for example) commit messages for use in the changelog.
  • This is the idea that sparked this ticket - it's not an approach I had even considered back when I started Releases, and it seems like it could go a long way towards a more slimmed-down, smarter version of the software.
  • The main problem here is: when do you put in that special annotation? It requires some commit's message to be more than just its own description of that commit's contents. Say I've got a 4-commit feature branch of 'doc, test, impl, fix to impl'. Which of those 4 commits should have the (extra?) blurb of eg [support] does support things? This conflicts with a commit message's "real" functionality of labeling what that specific commit changed.
  • However, I realized - git commit --allow-empty lets you add an empty commit to the tree! Normally that's a silly thing to do, but in this case it lets us add Git-level metadata, basically treating Git like the database.
  • So: folks habitually doing single-commit workflows can put the changelog annotation in their single commits no problem; folks habitually using granular commits can add an 'empty' one with the annotated changelog message; everybody's happy?

All that said, I wonder if there's a middle ground not relying (entirely) on Git metadata? If the core problem being solved is the merge conflicts in a single changelog file, could we do other wacky stuff instead? E.g.:

• Single-file changelog entries inside a changelog directory? Seems silly at a glance, but could work?
  • Look at a given HEAD to see what changes are in it, no need to crawl deeper into Git history (but still requires looking at all applicable HEADs, instead of just master, so we're still probably needing release-time script tomfoolery)
  • Would require the ability to tell which changes were added after the previous release/tag, though this could be more git reliance? (find files in that dir added to the tree after previous tag)
• Anything else? Per-release-line changelogs don't work, as avoiding them is the whole point of this project; not sure what else there is offhand.

---

The dump above covers both my own thoughts & those of the internal team I mentioned. Re: Releases itself, I'm torn.

For my own personal use (which is of course how it began), I'm tempted to investigate the annotated-git-commits approach - annotating some of my commits and/or doing occasional 'empty' commits seems like a compelling alternative to the increasing number of merge conflicts I'm dealing with (especially as we move into multiple major release families.)

The big downside is that this approach requires git tomfoolery at changelog creation time and spoils the "purity" of the current setup: you can no longer render the changelog from a static copy of one's Sphinx sources!

One must expect either:

• that the render action occurs within a (fully mirrored!) Git checkout
  • probably not true for a nontrivial number of users, especially on services like RTD where a limited checkout might be compelling, if it's not already being done - can't recall
• that one reverts to the ugly practice of "pre-rendering" the changelog as part of the release process (either to a 'final' normal-RST document, or - shudder - to an 'intermediate' document suitable for use with today's Releases.)
  • which reintroduces the old "whoops, forgot that step!" loophole - fixable with strict release processes, but still not ideal. One can argue the current Releases approach, where the onus is on the developing or merging individual, and not the releasing one, might be "better". Tradeoffs!

[MinchinWeb]

For myself personally, I appreciate the ability to review the Changelog from one place within the source files of my project; I refer to the source changelog.rst much more often then the rendered version.

For the projects I've been involved with, they're been small and so I haven't run into the issues of serious merge conflicts like you describe above.

But for merging up long running branches, maybe it's actual a feature to either work through the merge conflict, or prep the changelog so there isn't a merge conflict (perhaps by backporting the changelog updates), because it forces you to explicitly decide which release lines your new feature or bugfix should be released on.

[jaraco]

It feels to me like there are tools out there that have attempted some of these other approaches (and ran up against the tradeoffs):

• reno attempts to render the changelog based on file-per-change notes, deriving from a full git history.
• towncrier renders the changelog based on file-per-change notes, but materializes those changes into a static commit at release time.

I think releases strikes a good balance here, and I'd recommend that releases not try to re-invent itself.