Create a release process

poljar commented 1 month ago

It's time, I don't think we can realistically afford to not have a proper release process anymore or invent ever more complicated schemes how EW and EX can use the SDK without the SDK having proper releases.

Tasks

[ ] Get rid of Git dependencies.
[ ] Automated changelog handling.
[ ] Create release automation.
[ ] Write a release playbook.

Get rid of Git dependencies

This one is self-explanatory, we use Git dependencies sometimes because we have to, but we'll need to put an end of this practice if we want to stick to release schedule.

We don't have many of them but some have been here for quite some time:

The following list is used by the bindings and thus doesn't necessarily, need to be solved since we don't plan on releasing the bindings as a crate. The openidconnect crate could be moved to the bindings as well, since it solves a bindings-specific Send isssue:

Automated changelog handling

For changelogs I suggest we adopt git-cliff. It has support for monorepos.

The exact configuration that we're going to set up is up for discussion, though I suggest that we use the conventional commit configuration with one addition.

Add a special git trailer that contains the changelog entry. This should allow people to write longer and more detailed changelog entries.

Please note, the purpose of automated changelogs is not to generate the perfect changelog, manual edits after the changelog has been generated to improve the quality of the changelog is something we should always make time for.

Create release automation

This will need a bit of more investigation, but the gist of it is that we need to somehow do the following:

Make it easier to do major releases as well as patch releases for security issues.
Make it hard or impossible to do patch releases with breaking changes.

Please note, the goal of this task is not to give full control to the Github over our release process, in the end we do want to have a real human being:

Merge a PR that bumps the versions.
Create signed release tag.
Publish the crates to crates.io.

Release automation scripts/projects

There are a couple of existing projects which attempt to automate the release process which also include support for workspaces, there are of course a bunch of custom scripts and xtask written for this as well, some notable examples:

cargo-release, from the author of typos-cli which we heavily use, some additional docs can be found here.
Ruma, uses a custom xtask.
wasmtime, uses a custom script.
Insert you favorite example here...

Write a release playbook

Of course, all of the above should be documented and well-defined so any person in the team can do a release. Inspiration can be found in the wasmtime project.

richvdh commented 1 month ago

It's not my decision, so I'll bang this drum one more time and then leave it to the team to decide, but...

I still think that attempting to use commit messages to generate changelogs just gives you crap commit messages and crap changelogs. (I was inspired to blog about it, in case you're not bored of hearing from me about it yet.)

poljar commented 1 month ago

I still think that attempting to use commit messages to generate changelogs just gives you crap commit messages and crap changelogs. (I was inspired to blog about it, in case you're not bored of hearing from me about it yet.)

The blog post mixes the criticism of conventional commits with the criticism of git commits containing potential changelog entries.

It seems that the suggestion of the git trailer would address all of your concerns, except perhaps the git revert case?

richvdh commented 1 month ago

Ah, I missed the idea of a git trailer, sorry. Yeah, I think that helps a lot.

What I would say is this: I think that changelog entries ought to be reviewed just like code changes, and I think putting those changelog entries in a git commit message -- which frequently isn't even written until after a review happens -- makes that hard. I think it might also make it tricky to have CI which reminds people to write changelogs.

Perhaps I'm just making up problems though.

poljar commented 1 month ago

I think it might also make it tricky to have CI which reminds people to write changelogs.

Hmm, but it's similarly easy/hard to check if a changelog file has been modified or if git commits contain a specific trailer via a github action. The bigger problem is to teach a computer how to decide if a PR requires a changelog entry or not.

bnjbvr commented 1 month ago

The bigger problem is to teach a computer how to decide if a PR requires a changelog entry or not.

By combining cargo-public-api and cargo-insta, it's possible to:

check in the output of cargo-public-api
add a CI pass running cargo-insta running cargo-public-api, and comparing against the checked-in output
if there's a mismatch, then the public API has changed

It's not precise because a public API may not change despite a breaking change (in behavior), but I think that's a good 80/20 solution to the problem of finding what requires changelog entries overall.

bnjbvr commented 1 month ago

Also, we chatted about when to mandate changelog entries, and there could be a simple check that if a PR changes more than N LOC (for N >= 300? 400?), then it requires a changelog entry. Would lead to false positives for sure, but it would likely be a good first approximation.

richvdh commented 1 month ago

(Could you default to requiring changelog entries, but provide a mechanism for an author to explicitly flag "I've thought about this, and decided it doesn't need a changelog entry?)

poljar commented 1 month ago

I made some git-cliff changes for vodozemac: https://github.com/matrix-org/vodozemac/pull/165.

Adding trailers for common info about security issues was nice as well.

matrix-org / matrix-rust-sdk