Improve release notes through full automation, so that users know exactly what is in each release

Is your feature request related to a problem?

The GameMaker release notes have been, historically, created manually and have only ever represented a subset of the actual changes for a given release.

With the move to GitHub issues, we now have a lot more visibility on the kinds of things that might be in any given release, but still don't actually know. There is currently no way for us to answer the question: "What are the comprehensive set of changes made to this release compared to the one we're on?"

This is particularly impossible for beta releases.

On top of just being really frustrating conceptually, this makes it difficult for us to decide if any given GameMaker version is worth upgrading to. And when upgrading causes surprises (which happens nearly every time we upgrade), the lack of comprehensive changelogs makes it so we have no guidance as to why things might have gone wrong (e.g. we can't look through the changes and see that, "OH! This change to ds_maps might explain why this other thing stopped working!").

Long story short: Having comprehensive information about what's changed in a given release would be super valuable to GameMaker users (and staff!).

Describe the solution you'd like

The ideal solution to this problem is a tight coupling between changes and release notes. GitHub provides lots of mechanisms to automate this, and there are tons of ways it could be done depending on your workflows, so I can't present a tailored solution.

So, instead, I'll describe how we solve this problem for our games.

💡 Release Notes should come from Git commit messages

We use a custom system inspired by Conventional Commits, which is a super popular approach to solving this problem in the webdev space. I'm assuming y'all are using Git, but the idea would be the same for any subversioning system.

The rationale is this:

Changes are already being described by product developers when they write commits
High quality commit messages are beneficial for developers themselves
Re-using commit messages as release notes completely prevents any manual re-work, since those changes don't need to be re-described
Using commit messages ensures that release notes are comprehensive and accurate about what's in the release

The approach is this:

Commit message are formatted in a specific way so that they can be parsed by robots.
Developers write the messages for the release notes, rather than just being internal stuff for themselves.
- Developers can still write stuff that doesn't show up in the release notes!
- Many developers balk at this at first, because it feels like extra work. It's not -- it's just a requirement for quality in the work. And then it reduces downstream work, makes QA's lives easier, and makes users' lives easier (which, in turn, reduces a lot of other forms of work for devs!)
- It's better to have a fully documented release where some of the notes are awkward than to not have those notes at all, so it's okay if some developers write awkwardly.
Issues closed by a given commit are mentioned in the commit. This allows:
- Automatic closing of the issue when the commit is pushed
- Automatic closing of tickets in other systems, if needed (e.g. we use Favro for project management rather than GitHub Issues)
- Automatic referencing between the issue and the commit message
- Automatic linking between release notes and the relevant issues

For example, here's a selection of recent commit messages from our current project:

meta: Removed legacy fields from package.json
chore: Lodestone fountain placeholder added and Tendraam decor set painting update
0.1.361-rc.0
chore: Fixed a crash when downloading strings.
fix(dev|Workshop): Fixed an issue that was causing the game to purge your local campaign files if they deviated from the packed campaign file.
meta: Fixed missing credentials
0.1.360-rc.0
fix(dev|Audio): The music should no longer get super loud momentarily while on the loading screen for entering the packed campaign.
fix(Saving): Fixed an issue that would sometimes allow you to load into world maps that are from a higher version of the game than you are currently on.
imp(dev|Saving): Data fields on world elements (such as how many thorns a Creepthorn has) can now have their data types changed in the Game Changer without breaking save files or world map files.

The prefix term (meta, fix, imp, feat, chore, etc) indicates the kind of change (only certain kinds of changes are actually published in our release notes). The "topic" comes next, in parens, allowing grouping of changes by category.

Our system extends Conventional Commits by also adding an optional "stream" (e.g. for fix(dev|Audio), dev is the stream). This allows us to generate distinct sets of patchnotes per stream, which in turn are shown to different audiences (e.g. the dev stream is only shown in our internal release notes).

Our system also extends Conventional Commits to allow us to have multiple entries per commit message, so that we can be more loosey-goosey about the "one change, one commit" philosophy.

When we make builds on our build servers, we have a script that reads all of the commit messages, parses them, composes them into a giant JSON file, and uploads that to our server as a simple file. We can then download and display that content however we want on our release notes pages, in our games, etc.

Here's what the above Git messages end up looking like, for example:

This screegrab shows a portion of the HTML version of the release notes generated from the sample commit messages above. The red number and checkmark next to each entry are used by our QA team to (1) check off an item as tested and (2) see how many QA have checked off that item.

For an interactive experience, check out Levelhead's patchnotes. That page just fetches that big JSON file and then converts it into interactive HTML!

(Note: if we had public issues, we'd include links back to those in our patchnotes. We do include links to our internal system, but strip those out of what's displayed to regular users.)

Basically every major open-source web project uses Conventional Commits and publishes Markdown release notes, so there are many examples of this in action:

GameMaker, and games, have slightly different problems since the actual code and git commits are proprietary and inaccessible to consumers, which is why we've taken our own spin on this and GameMaker would need to, too.

Describe alternatives you've considered

We've tried various ways of searching and reading the GitHub Issues to figure out what's in a given release. The fact is that this just is not possible because there isn't enough information available to figure that out.

But, even if there were, it's a huge amount of work for us to read through the issues, guess what they mean, guess what it means that they were closed, and guess if they are in a given release.

Additional context

No response

I thought I commented on this a few months ago when the change notes started being like this, but if I did I can't find it: this is what the user sees when they open the release notes.

You have to click on a link, which means you've already lost like 80% of readers and you haven't even gotten out of bed.

When you click on the link, you see this.

The actual changes are mostly buried in the "show more" at the top, and most of what you see is the big purple list of closed Github issues. These are long, dense, technical, and honestly not-very-interesting literature that you have to read through just to get at the two or three things in the new version that's relevant to you and your project.

I don't think the way the update notes themselves are written are the biggest problem, and I don't really think generating them from commit messages are any better than having a human write them out by hand (remember, most of the people who are going to look at this are hobbyists without formal training, not professional web devs), but because it's they're so small and hard to find, for all intents and purposes they don't exist. Even I have to do some work just to get to the page where they can be found.

@DragoniteSpam Commit messages are also "written by hand". But yes, they'd need to be written for the target audience.

Commit messages for things end users shouldn't see would be written such that they don't end up in the release notes.

Which is to say that I agree commit messages should not be used to generate release notes if said commit messages are not written for end users. I'm saying that git commits should be written for end users, at least in part, because that solves a whole bunch of problems at once.

We keep technical details in the body of the commit messages, which we exclude from release notes, and the title line only appears as a release note when it is intentionally formatted so that it would appear.

If devs consider how each change may impact end users, and then explain that change for end users (or QA, or any other specific target audience), at the time they are making the commit and documenting the change, that's when you'll get the most accurate and comprehensive description of the changes. You also get a side benefit that git message are more useful for everyone involved (internal developers, internal QA).

Though I'll also note that you don't even need to do this through Git messages written for end-users (though I believe that's ideal):

If git messages include a ref to the issue they close, that would also be sufficient to accomplish all of these goals (assuming all changes go through GitHub Issues, even internal ones). In that case the changelog-building tooling could just fetch the titles all of the closes issues.

Just for the record, if release notes process was to be automated or semi-automated, the issues are better be sorted into features and bugs (considering bugs and FR repositories were merged), and also preferably sorted by their impact. On that note, I guess semi-automation - when baseline is auto-generated, but the human might then edit the output - might be more fitting, as it's easier to decide which is actually a big feature (and thus is better showcased at the top) and which is a minor addition. "We made a 'reload' button for running the game, we added a setting for the default project creation directory, also GUI prefabs are a thing now" 😉

Also, bugfixes related to new changes should be filtered out. For example, if new Code Editor is finally revealed, I'm pretty sure we'll get quite a few bug reports related to it (as with many new features, especially as big as this one). These bugfixes don't relate to something that has been broken in a previous monthly and is fixed in the new release, but rather something that has been broken/introduced and fixed in the same Beta cycle, and thus irrelevant in the monthly-to-monthly scope. Filtering these kinds of bugfixes would hopefully reduce some clutter, too. ^^'

^ Yes, this would all be essential for any system we choose.

Just for the record, if release notes process was to be automated or semi-automated, the issues are better be sorted into features and bugs (considering bugs and FR repositories were merged), and also preferably sorted by their impact. On that note, I guess semi-automation - when baseline is auto-generated, but the human might then edit the output - might be more fitting, as it's easier to decide which is actually a big feature (and thus is better showcased at the top) and which is a minor addition. "We made a 'reload' button for running the game, we added a setting for the default project creation directory, also GUI prefabs are a thing now" 😉

Also, bugfixes related to new changes should be filtered out. For example, if new Code Editor is finally revealed, I'm pretty sure we'll get quite a few bug reports related to it (as with many new features, especially as big as this one). These bugfixes don't relate to something that has been broken in a previous monthly and is fixed in the new release, but rather something that has been broken/introduced and fixed in the same Beta cycle, and thus irrelevant in the monthly-to-monthly scope. Filtering these kinds of bugfixes would hopefully reduce some clutter, too. ^^'

Agreed! This can be achieved pretty easily with the Conventional Commits / Conventional Changelogs framework as described above, via prefixes and "streams". We deal with this pretty regularly in our studio, where we'll have changes that are shown internally to QA but don't end up on the final patch notes for players, which we manage via the methods Adam described in the OP.

YoYoGames / GameMaker-Bugs