search

hannobraun / fornjot

Early-stage b-rep CAD kernel, written in the Rust programming language.
https://www.fornjot.app/
Other
2.03k stars 115 forks source link

Automate release procedure #104

Closed hannobraun closed 1 year ago

hannobraun commented 2 years ago

It would be great to largely automate the release procedure.

There are basically just three parts that need to be manual:

  1. Choose the version number.
  2. Update the changelog.
  3. Write a release announcement.

It would be great, if somehow we could provide these ingredients in a commit (maybe by adding an entry to a releases.toml file that specifies the version number and links the release announcement), then have an automated action be triggered by that, that takes care of everything else.

I'm not sure how practical this is. For example, I don't know if such a trigger can be specified. But it would be great to have.

Labeling as https://github.com/hannobraun/Fornjot/labels/help%20wanted, as I'm no expert in GitHub Actions. If someone who knows more happens to drop by and could leave some insights about the feasibility of this, that would be very appreciated!

hannobraun commented 2 years ago

@jamwaffles suggested cargo-release. I haven't looked into it, but it's certainly worth checking out.

hendrikmaus commented 2 years ago

Hey @hannobraun, I'd like to support you with your release process (including #57).

I have built https://github.com/hendrikmaus/rust-workflows as a reference implementation to copy and hack on for one's own repositories. It also includes a CI workflow that might be of interest.

I am also looking to adopt cargo-release into the reference, since it takes the bulk of manual commands away into a single one.

rust-workflows includes:

As mentioned before, it is made to copy the parts one wants and hack on them. It should make it way faster and easier to setup a complex Rust release process on GitHub Actions.

I'd propose to make multiple smaller change-sets that slowly build up the release process.

hannobraun commented 2 years ago

Hey @hendrikmaus, that sounds awesome! I'm really busy with work on the CAD kernel, and this whole release business is not really my specialty anyway, so I'm really happy to receive help here.

I've taken a quick look at your repository, and it does look really promising. Where do you propose we start? Maybe address #57 first, then automate the release procedure starting with creating the release on GitHub?

I'd propose to make multiple smaller change-sets that slowly build up the release process.

Yes, I'm all for that :+1:

hendrikmaus commented 2 years ago

Yes, let's tackle #57 first. We should be able to store the binaries on each workflow run, using the artifacts feature. That way, no release is needed to host them.

hannobraun commented 2 years ago

Sounds good!

hannobraun commented 2 years ago

@jamwaffles made me aware of release-plz: https://github.com/MarcoIeni/release-plz

I haven't had the chance yet to check it out, but it seems worth looking into.

hendrikmaus commented 2 years ago

Nice, I was researching in the same direction in the last days. I adopted the "original" release please in this crate https://github.com/hendrikmaus/kube-leader-election but, at a glance, the new proposal is even better.

Are you also ok with adopting conventional commits? I am trying right now in every project and so far it is beneficial.

hannobraun commented 2 years ago

Are you also ok with adopting conventional commits? I am trying right now in every project and so far it is beneficial.

For the following comment, I'm assuming we're talking about this specification. I wasn't aware that existed, but I tried a similar technique a few years back, to generate changelogs automatically. Based on that experience, and a short reading of the specification, I'm against enforcing it (although I have no problem merging commits that follow that format).

The specification has a section titled Why Use Conventional Commits. I'm going to go through each of the items to explain my position.

Automatically generating CHANGELOGs.

I strongly believe that this is fundamentally the wrong approach. The resulting changelogs are too detailed and not readable. Making them even remotely readable puts too much of a burden on contributors (for creating compliant commits) and maintainers (for policing those commits, to make sure they're compliant).

Enforcing the commit format in CI takes care of some of the maintenance burden, but doesn't help make the changelog readable.

I see changelogs as documentation, and that's an art, not a science. I summarize and editorialize when writing them. (The current changelog is not the best example of what I'm aiming for, as the last release still covers a lot of ground from the time when I just pushed directly to main without a care in the world. The Weekly Dev Log format is more along the lines of what I'm aiming for, although I would omit a lot of that from a release changelog.)

Automatically determining a semantic version bump (based on the types of commits landed).

That would be great, but that automation is only going to be as reliable as the humans writing the commit messages. And breaking changes can be subtle, easy to miss, and recognizing them reliably would again put too much of a burden on contributors and maintainers.

I don't know, if the technology to do this is available, but I'd feel better about an automated scan of the API at release time, to determine if a release is breaking or not.

Communicating the nature of changes to teammates, the public, and other stakeholders. Making it easier for people to contribute to your projects, by allowing them to explore a more structured commit history.

Worthwhile goals, but there's a trade-off to be made here: Do we optimize for ease of writing, or ease of reading?

Personally, I put a lot of care into my commits and pull requests. I make sure my commits are not too big and not too small. I regularly split off commits that are self-contained and centered around a common topic from my WIP branch and submit them separately. While I do that, I modify my history all the time, to make it more clear. (Although I could put more effort into my commit messages. I'm working on it, but sometimes bad habits slip in).

If I were paying a highly skilled team to work on Fornjot, I would expect the same level of care from them. I do not expect that level of care from contributors. I don't even expect contributors to realize that putting in that level of care is a possibility.

Encouraging people to get involved is essential. Even (and especially!) inexperienced people, who are looking for a project to learn with. In that context, optimizing for ease of writing (within reason) is the right call, in my opinion.

Triggering build and publish processes.

Sounds like a good idea, especially in the context of this issue. We wouldn't need to adopt the whole spec to do this, though.

This has turned into a long comment, but I appreciated the opportunity to write down my thoughts on the matter. It's always nice to have something to refer back to, if similar suggestions pop up. Maybe I'll even turn this into a blog post for the Fornjot website :grin:

If you think I'm wrong, I'd be happy to be convinced otherwise. But I do feel strongly about this topic, and I think the arguments I've made are sound.

hendrikmaus commented 2 years ago

Thank you for explaining your position, many other maintainers would've simply posted "no" and be done with it. I appreciate you took the time. Also, I agree with you on a lot of points.

Sounds like a good idea, especially in the context of this issue. We wouldn't need to adopt the whole spec to do this, though.

Yes, that was also my thought after reading through your comment.

I am currently challenged by getting an overview of all the tools and their opinions. Composing them to create one's own workflow is not a trivial undertaking. Each tool makes certain assumptions about the overall process which might not be matching every single time. I will keep experimenting a bit on other projects as well. This is a highly iterative process for me.

Back to Fornjot, I think we should be able to build a neat pull-request based process. I will keep working on it.

hannobraun commented 2 years ago

Thanks, @hendrikmaus! I appreciate you taking the time to figure this stuff out. This is the kind of thing I simply wouldn't have the capacity to do.

Just a reminder, this doesn't need to be perfect. If one tool provides 80% of the solution, that might be worth adopting. Same for a homegrown alternative. I'm happy to merge anything that's an improvement over the current state. Further improvement later is always possible.

hendrikmaus commented 2 years ago

So I have been thinking and experimenting a bit. I'd like to offer my current conclusion as a proposal before starting to make any changes.

All holistic tools I've tried are too opinionated about the process and impose restrictions that do not fit Fornjot. I did a small case study to get a feel for writing a custom solution in the GitHub Actions context; I am quite pleased with the idea:

I want to write a small Rust command-line application that lives in .github/release-operator (name up for discussion, of course). This application will be called via cargo run in the Continuous Delivery workflow (dependencies will be cached) on every merge to main. It will examine the commit message of main's HEAD to determine if a release is supposed to happen. The algorithm would be:

If all of the above can be asserted, a release shall happen. E.g. Release v0.5.1, from a pull-request labeled with release, would trigger a Fornjot release of v0.5.1, while a commit like bump version of release operator to v0.1.0 would not, because it's pull-request wasn't labeled with release. I impose the requirement of a pull-request with a specific label to rule out these edge cases.

Now, since we're in the context of GitHub and GitHub Actions, we can fully embrace it on the the implementation details:

This keeps the manual steps in place, like maintaining the changelog and release announcements, while allowing the workflow to take over more and more steps in an automated fashion.

Looking at the current Release Procedure, here is a patch to it:

  1.  [manual] Create release branch: git switch -c v0.1.2
  2.  [manual] Update changelog
  3.  [manual] Write release announcement
  4.  [manual] Update versions
- 5.  [manual] Publish the release
+ 5.  [manual] Open a release pull-request
+     label it as `release`, merge it with a message like `release v0.1.2`
- 6.  [manual] Tag the release: git tag v0.1.2
- 7.  [manual] Create release on GitHub
+ 6.  [auto] Create release on GitHub (includes a new `tag`)
+ 6.1 [auto] Add relevant portion of the changelog to the release body
+ 7.  [auto] Compile binaries and attach them to release incl. checksums
+ 7.1 [auto] Publish to crates.io
+ 7.2 [auto] Build and push Linux container image
  8.  [manual] Finish publishing release announcement
  9.  [manual] Promote release announcement
  10. [manual] Improve release procedure

Steps 8. and 9. also call for automation potential at a later stage.

hannobraun commented 2 years ago

Thank you @hendrikmaus, this looks really good. I'm not surprised that the existing tools wouldn't work for us. Your assessment that they are too opinionated matches my own impression (which, to be honest, is based on years-old experience and little up-to-date research).

The concept you present here is fully in line with my own (rather superficial) thinking on the topic. Consider the concept approved, as far as I'm concerned :grin:

I've left some comments below. I don't feel strongly about either of them. Use or ignore them, as you prefer.

I want to write a small Rust command-line application that lives in .github/release-operator (name up for discussion, of course).

Is there a specific reason that it should live in .github? Having Rust code in there seems unexpected, and I have a weak preference for just having a directory for it in the repository root, where it would be part of any potential Cargo workspace we'd have in the future. It's Rust code, after all, and it makes sense to me to maintain them alongside the other Rust code in this repository.

(And release-operator seems like the perfect name, by the way.)

It will examine the commit message of main's HEAD to determine if a release is supposed to happen.

Can we skip that step and just check for pull requests? Unless there's a reason to keep the matching of commits to pull requests to a minimum (API rate limiting, or whatever), this step seems redundant.

Steps 8. and 9. also call for automation potential at a later stage.

Plus, we'll have downloads on the website at some point, so there will definitely be more opportunities for automation.

One additional note: https://github.com/hannobraun/Fornjot/issues/141 is going to start happening in the near-ish future, which would mean we'd have more crates to release. In contrast to the two crates we have currently, I wouldn't expect all of them to follow the same release lifecycle.

There's always a tradeoff between monorepos and splitting things up. It has been my experience that monorepos become cumbersome once you have crates in them that release at different cadences, and have different version numbers. I expect that the same problem would apply to this system. Ideally, we'd release all crates in this repository at the same time, with one shared version/changelog/tag/release/announcement/etc.

Any crates I'd like to move to separate repositories, are probably going to be lower-level dependencies that are on a slower release cadence than the main body of Fornjot. I don't think this is going to be a problem, as this repository (including the automated release system) can just treat them like any other dependency.

Even though I don't think this will affect the automated release system, I wanted to give you a heads up, in case you have thoughts (especially if they contradict my own).

hendrikmaus commented 2 years ago

Thanks for your feedback, I'll start to submit pull-requests now.

The concept you present here is fully in line with my own (rather superficial) thinking on the topic.

Couldn't have gone better then :)

Is there a specific reason that it should live in .github?

No, I was sitting on the fence between a dir in the root and in .github; root it is.

Can we skip that step and just check for pull requests?

Yes, there is no hard requirement for it.

In regards to #141: Either way, we can leverage the new release-operator. I would lean to a poly-repo approach as well. When time comes, we can move release-operator into its own repository and provide a little wrapper GitHub Action for it. Then, all Fornjot repositories can use the same process in individual cycles.

hannobraun commented 2 years ago

Thanks for your feedback, I'll start to submit pull-requests now.

Great, thank you! :+1:

The concept you present here is fully in line with my own (rather superficial) thinking on the topic.

Couldn't have gone better then :)

"Great minds think alike", or maybe "zwei Dumme, ein Gedanke". Choose the one you prefer :grin:

In regards to #141: Either way, we can leverage the new release-operator. I would lean to a poly-repo approach as well. When time comes, we can move release-operator into its own repository and provide a little wrapper GitHub Action for it. Then, all Fornjot repositories can use the same process in individual cycles.

Sounds great!

hendrikmaus commented 2 years ago

@hannobraun which part should we automate next?

Publishing releases to crates.io could also be done from within the workflow. It requires a token to be put in the repository secrets though.

Should we containerize anything? Should we publish anything via Homebrew?

hannobraun commented 2 years ago

@hannobraun which part should we automate next?

Publishing releases to crates.io could also be done from within the workflow. It requires a token to be put in the repository secrets though.

I think publishing to crates.io would be the logical next step, as it's adjacent to the part that's already automated, and should be relatively easy to do. After that, maybe updating all versions and path dependencies?

The token doesn't sound like a problem? Just tell me what you need and where you need it.

Should we containerize anything?

I don't know what that means, outside the context of deploying backend services.

Should we publish anything via Homebrew?

I have little experience with Homebrew, and no experience publishing to it. If it's easy, then I guess why not.

Long-term, I'd like the Fornjot app to be available anywhere people would expect to install their software from. I don't think there's any pressing need right now though. It might even be better to wait for another 2-3 releases. By that time, Fornjot has hopefully approached a minimally useful state for more use cases.

hendrikmaus commented 2 years ago

Ok, let's go with publishing to crates.io next then. When you log in to crates.io, you can create API tokens in your profile. Generate one and put it into the Actions Secrets of the Fornjot repository. The name of the secret, usually all uppercase, could be something like CRATESIO_API_TOKEN. Github Actions will then be able to access this secret.

I would make the workflow idempotent on the version - i.e. I'd query crates.io before publishing a version to determine if it is already there. Otherwise, the call to publish a crate will fail.

Could you provide a list of crates to publish? I would then build the workflow part to iterate over the list.

Would you like more bash or should we start to extend the release-operator? I would lean for the latter. We have the base in-place and could simply start adding sub-commands using clap. The first sub-command would be the one for detecting the release (i.e. what a simple cargo run does right now) and the second one would be the publish routine.

I reckon we can skip Linux containers and Homebrew for now. To cover everyone's preferred package manager and operating system will require community effort. I do not see that as a responsibility of the upstream project.

hannobraun commented 2 years ago

Ok, let's go with publishing to crates.io next then. When you log in to crates.io, you can create API tokens in your profile. Generate one and put it into the Actions Secrets of the Fornjot repository. The name of the secret, usually all uppercase, could be something like CRATESIO_API_TOKEN. Github Actions will then be able to access this secret.

I created a token, and added it as the repository secret CARGO_REGISTRY_TOKEN, as that seems to be the name that Cargo uses for it.

There are also "Environments" in the settings, which seem to be another way to add secrets. I don't quite grasp the difference, but it seemed like a repository secret would be the way to go.

I would make the workflow idempotent on the version - i.e. I'd query crates.io before publishing a version to determine if it is already there. Otherwise, the call to publish a crate will fail.

Sounds good :+1:

Could you provide a list of crates to publish? I would then build the workflow part to iterate over the list.

This is subject to change, as I'm still working on #141. For now, this is the list:

For now, a static list is good enough, but eventually, I'd like a more maintainable solution.

One option would be to use the Cargo workspace to generate the list (slap a publish = false on release-operators and everything in models/, filter by that), but that seems error-prone. Only a matter of time, before someone adds a model and forgets publish = false.

A better option might be to move all public crates into a crates/ folder, and just publish everything in there. What do you think?

Would you like more bash or should we start to extend the release-operator? I would lean for the latter. We have the base in-place and could simply start adding sub-commands using clap. The first sub-command would be the one for detecting the release (i.e. what a simple cargo run does right now) and the second one would be the publish routine.

I would also prefer to extend release-operator. It's already there, so I see no reason to build some of the infrastructure in Bash.

I reckon we can skip Linux containers and Homebrew for now. To cover everyone's preferred package manager and operating system will require community effort. I do not see that as a responsibility of the upstream project.

Well, since I'm trying to make a living off this project, my perspective is a bit different. If a broad packaging effort can help promote Fornjot in a meaningful way, then I might make it my responsibility.

But in any case, it's premature. For now, I wouldn't discourage anyone from packaging Fornjot, but I don't see a reason to encourage it either.

hendrikmaus commented 2 years ago

I created a token, and added it as the repository secret (...)

That's perfect. Aside: The "environments" feature is supposed to aid projects that have different release tiers, e.g. edge, stage, prod. And each of those have different secrets.

For now, a static list is good enough, but eventually, I'd like a more maintainable solution.

We could start with a repeatable cli option, provided to the release-operator call, e.g. --crate fj-app --crate fj-debug (...). Like that, the list of crates to publish would be in the GitHub Actions workflow. This would be an explicit opt-in to publish a crate.

Another way would be to use the metadata section that Cargo offers to be open for extension: https://doc.rust-lang.org/cargo/reference/manifest.html#the-metadata-table, e.g.:

[package.metadata.release-operator]
# ...

One could place any number of fields in each Cargo.toml. A combination of both approaches would yield a great amount of flexibility whenever the first edge cases start to arrive. By adding the --crate option, the release-operator gets an explicit list of crates to look at, and if it finds metadata in there, it can use that as well.

Well, since I'm trying to make a living off this project, my perspective is a bit different. If a broad packaging effort can help promote Fornjot in a meaningful way, then I might make it my responsibility.

Makes sense. I would then propose to start a list in a separate issue. A good start would be:

hannobraun commented 2 years ago

Aside: The "environments" feature is supposed to aid projects that have different release tiers, e.g. edge, stage, prod. And each of those have different secrets.

Ah, makes sense.

We could start with a repeatable cli option, provided to the release-operator call, e.g. --crate fj-app --crate fj-debug (...). Like that, the list of crates to publish would be in the GitHub Actions workflow. This would be an explicit opt-in to publish a crate.

Another way would be to use the metadata section that Cargo offers to be open for extension: https://doc.rust-lang.org/cargo/reference/manifest.html#the-metadata-table, e.g.:

[package.metadata.release-operator]
# ...

One could place any number of fields in each Cargo.toml. A combination of both approaches would yield a great amount of flexibility whenever the first edge cases start to arrive. By adding the --crate option, the release-operator gets an explicit list of crates to look at, and if it finds metadata in there, it can use that as well.

All of that makes sense. I just worry that we'll be adding small crates somewhat regularly (as functionality grows, and it makes sense to split something out), and then it'll be a pain doing a release each time, because I forgot to add something or other to the list.

But that's probably bullshit. Starting with an explicit list is definitely fine, and if problems show up, we can adjust.

Makes sense. I would then propose to start a list in a separate issue.

I think opening an issue would probably be premature, but I'll add it to the feature wishlist and link to your comment.

hannobraun commented 2 years ago

394 has some discussion on populating the GitHub release once it is created, specifically on extracting the release announcement from the website to add it there.

hannobraun commented 2 years ago

I just had a thought: The order in which we release crates is important, as they depend on each other. I don't think we take that into account right now.

For now, we would need to make sure that the GitHub Actions job passes the crates in the correct order. Eventually, it might be nice if release-operator could determine the correct order automatically, especially if we decide that we want to go with a convention-over-configutation approach, like putting all the fj-* crates in a single crates/ directory and publishing everything from there.

hannobraun commented 2 years ago

@hendrikmaus I'm starting the process of publishing the next release now. Looking forward to see how it goes!

hannobraun commented 2 years ago

Update: Still working on the release. First the changelog took forever. Then writing the announcement took forever (not completely done yet). Then other stuff got in the way (spent a bunch of time reviewing a pull request today).

Haven't gotten to the automated portion yet.

hannobraun commented 2 years ago

I attempted the release in https://github.com/hannobraun/Fornjot/pull/524. Didn't work, unfortunately. release-operator didn't detect a release: https://github.com/hannobraun/Fornjot/runs/6308629861?check_suite_focus=true#step:5:129

hannobraun commented 2 years ago

Ah, I think I messed up the commit message. The version number I included was missing the 'v' prefix.

hannobraun commented 2 years ago

Another attempt in https://github.com/hannobraun/Fornjot/pull/526. Same result: https://github.com/hannobraun/Fornjot/runs/6309044619?check_suite_focus=true#step:5:129

I think we need more logging in release-operator.

hendrikmaus commented 2 years ago

You can use RUST_LOG for more details. On 5. May 2022, 17:31 +0200, Hanno Braun @.***>, wrote:

Another attempt in #526. Same result: https://github.com/hannobraun/Fornjot/runs/6309044619?check_suite_focus=true#step:5:129 I think we need more logging in release-operator. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

hannobraun commented 2 years ago

You can use RUST_LOG for more details.

How so? It doesn't seem to emit any log messages during the release detection procedure. I've added log messages with more information here: https://github.com/hannobraun/Fornjot/pull/527

hendrikmaus commented 2 years ago

Alright, I didn’t have the code in mind any more. I am actually sick at the moment and cannot be of more help 😐 On 5. May 2022, 17:39 +0200, Hanno Braun @.***>, wrote:

You can use RUST_LOG for more details. How so? It doesn't seem to emit any log messages during the release detection procedure. I've added log messages with more information here: #527 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

hannobraun commented 2 years ago

No problem, hope you get well soon!

Okay, forgot to add the release tag this time. Retrying again...

hannobraun commented 2 years ago

Another attempt in https://github.com/hannobraun/Fornjot/pull/528. It's not finding the version number in the commit message: https://github.com/hannobraun/Fornjot/runs/6309497951?check_suite_focus=true#step:5:129

Seems like it's only looking at the first line, but the merge commit has the version in line 3.

Side note: Something's not working correctly with the caching of the Release job. It's re-downloading and re-compiling a bunch of dependencies each time, even though all I'm doing is pushing empty commits.

hendrikmaus commented 2 years ago

IIRC the version has to be in the first line per docs. On 5. May 2022, 18:02 +0200, Hanno Braun q @.>, wrote: Another attempt in #528. It's not finding the version number in the commit message: https://github.com/hannobraun/Fornjot/runs/6309497951?check_suite_focus=true#step:5:129 Seems like it's only looking at the first line, but the merge commit has the version in line 3. Side note: Something's not working correctly with the caching of the Release job. It's re-downloading and re-compiling a bunch of dependencies each time, even though all I'm doing is pushing empty commits. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.>

hannobraun commented 2 years ago

The docs don't specify that actually. But can I even influence the first line of the merge commit? I'm not sure, as I always click that text field away so quickly. I'll have to pay more attention next time.

But it doesn't matter, I think. I've modified release-operator to search the whole commit. Still no luck: https://github.com/hannobraun/Fornjot/runs/6309787693?check_suite_focus=true#step:5:129

hendrikmaus commented 2 years ago

Did you use the correct label on the release pull-request? Every release commit needs to belong to a pull-request labeled as defined in the GitHub actions step. On 5. May 2022, 18:22 +0200, Hanno Braun @.***>, wrote:

The docs don't specify that actually. But can I even influence the first line of the merge commit? I'm not sure, as I always click that text field away so quickly. I'll have to pay more attention next time. But it doesn't matter, I think. I've modified release-operator to search the whole commit. Still no luck: https://github.com/hannobraun/Fornjot/runs/6309787693?check_suite_focus=true#step:5:129 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

hannobraun commented 2 years ago

Did you use the correct label on the release pull-request? Every release commit needs to belong to a pull-request labeled as defined in the GitHub actions step.

Yes. It's getting past that point in the detect function.

Turns out you can control the whole commit message. I just didn't notice because I've trained myself for years to click it away immediately.

Added more logging to release-operator. Next attempt in https://github.com/hannobraun/Fornjot/pull/530.

hannobraun commented 2 years ago

Nice! This did it: https://github.com/hannobraun/Fornjot/pull/531/commits/0d847ebeba6c633230a448b0e27109a5aae3bc78

Also, this: https://github.com/hannobraun/Fornjot/pull/531/commits/cfc97439fcbf7a9143cdda63fe856b1d0a158ae5

That second change wasn't critical, I think, but it made a difference (before, after).

This got me a few steps further. New failure here: https://github.com/hannobraun/Fornjot/runs/6310273848?check_suite_focus=true#step:9:24

hannobraun commented 2 years ago

For the record, extracting the release announcement from the website, like we talked about, works. Here's what I did:

  1. Take contents of <main>
  2. Replace href="/ with href="https://www.fornjot.app/.
  3. Replace src="/ with src="https://www.fornjot.app/.
  4. Added note up top, linking to the release announcement on the website.

release-operator problems encountered:

It's getting late, so I'm doing the cargo publish manually now. Everything else works. I guess this went about as well as can be expected for the first try. Pretty happy with the end result!

hannobraun commented 2 years ago

I just got this while publishing locally:

error: failed to publish to registry at https://crates.io

Caused by:
  the remote server responded with an error (status 429 Too Many Requests): You have published too many crates in a short period of time. Please try again after Thu, 05 May 2022 18:00:29 GMT or email help@crates.io to have your limit increased.

Not sure, if this is due to the crates I actually published, or if it takes into account the recent failed attempts. If the former, this is something we'll have to deal with to get the automation working.

hannobraun commented 2 years ago

I've opened issues for the problems I encountered:

hannobraun commented 2 years ago

I've published a new release! Overall, the experience was still mixed, but we're making progress!

Previously we discussed further automating the release procedure, by downloading the release announcement from the website. The approach we discussed is to download the HTML and processing it.

I've run into new problems with doing that manually:

That whole approach of downloading and processing the HTML is probably flawed. It would be better to somehow download the original Markdown instead, which I don't think Zola can do without additional manual steps. And additional manual step render the automation moot.

Unless someone can figure out how to do this with Zola, it's probably best to shelve this part of the automation until we have something more flexible to generate the website (which I don't have concrete plans for right now).

Problems encountered this time:

Plus a few more that were unrelated to the release automation (repository settings I messed up, path dependency was missing version, stuff like that).

I plan to switch to a weekly release cadence starting the week after next, which should help flush out the remaining problems.

hannobraun commented 2 years ago

Notes from the latest release, which is the first weekly one:

788 seems to be the last problem that prevents a smooth release process.

I have an idea for how to automate the "Update versions" step. I think this is a prime candidate for automation, as it's right next to the already automated part.

How about this:

  1. Open release PR without having updated the versions before.
    • Title contains the version in the v0.1.2 format.
    • PR is tagged with release.
  2. release-operator runs as part of CI build and detects the release tag.
  3. release-operator parses version from the PR title and detects the mismatch.
  4. release-operator pushes a commit to the PR branch that updates the versions.
  5. release-operator fails the CI build, to prevent a race condition between the commit it pushes and the merge.
  6. CI re-runs because of the new commit. release-operator no longer detects a mismatch and passes.
  7. PR gets merged; release process continues as normal
hendrikmaus commented 2 years ago

In Github Actions, a pipeline cannot trigger other jobs[^1]. Hence, steps 5. and 6. wouldn't run.

I think it is a good idea. It is similar to what the Rust feature in Google's release-please does.

[^1]: there are ways, but they involve a personal access token or workflow templates

hannobraun commented 2 years ago

I just published the latest release. No problems this time! Just some good old-fashioned robotic efficiency. This leaves the lack of further automation as the most annoying aspect of the process :smile:

I personally plan to create some tooling over in the website repository to assist writing the release announcement (generating the announcement from the template, getting the list of pull request from GitHub, getting the list of sponsors, stuff like that). Meanwhile, it would be great to see further automation of the release procedure itself.

I've had some ideas for how to automatically provide the Markdown source of the release announcement from the website. If that pans out, this would also enable further automation here.

Thought: Should we run cargo publish with the --no-verify option? The last step of the process spends a lot of time re-compiling code that has already been compiled multiple times by that point.

hannobraun commented 2 years ago

I've published two more releases since last posting. I can definitely say that this has become routine now, and as far as I can tell, all the problems of the release automation have been flushed out and fixed.

I've started attacking the larger area of release automation from another direction, and have started automating the first steps of the release procedure, writing the release announcement. Not only is this the biggest chunk of manual work, where I see the most low-hanging fruit, but it also paves the way for extending the release automation in this repository. By providing more information about the release through the website, for example, which can then be used on this side.

hendrikmaus commented 2 years ago

Thanks for the update 🙏 glad to hear that everything continues to fall into place.

As to your question at the end of the previous comment: I guess it is OK to skip the verification. You correct in that the code has been compiled enough times by the time the publishing runs.

hannobraun commented 2 years ago

Thanks for the comment, @hendrikmaus! I already made the change to skip verification after last week's release (https://github.com/hannobraun/Fornjot/pull/903). This has accelerated this week's release nicely!

hannobraun commented 2 years ago

It's been a while since I posted here, so I figured it's time for an update.

The release automation has been trucking along without issues. I'm very happy with how everything works! I've also been automating more steps, using a command-line tool that primarily helps me with the release announcement, by creating it from a template, determining the release version, automatically adding all relevant pull requests and generating the list of sponsors.

Writing the release announcement is still the most labor-intensive step, but I think it's as automatic as it's going to get. The next logical step is to bridge the gap between the release announcement and the rest of the automation. Right now, there are three steps here:

  1. Update the changelog.
  2. Update version.
  3. Trigger the release automation by merging a pull request.

All of those should be mostly trivial to automate, in principle:

There are a few practical problems, however:

I'm fully confident that I can bend the website and the CLI tool to my will somehow, but I lack knowledge on the GitHub Actions side. I don't know if it can be triggered in a way we'd need it to, and what exactly it can do in regards to updating files in the repository.

hannobraun commented 1 year ago

I've decided to close this issue. Release automation has long been in place, and this issue has gotten stale. While the release process could be improved and automated further, I think we should track this in more specific issues.

In addition, with the end of the weekly release schedule and the new reduced scope (see A New Direction), I think it makes sense to take a step back and re-evaluate what is necessary. I expect to open those more specific issues in the future, as I get a better understanding of what's needed.