blueprint: release strategy and documentation

[theopolis]

There was recent discussion on Slack about a release strategy.

Here is my proposal / recommendation, please provide criticism and feedback. The goal is to flush out a strategy, document and communicate it (perhaps on README or otherwise), then implement.

• We set periodic GitHub milestones, perhaps on a 2-month cadence, with some significant issues attached and try to drive those to completion. These are labeled X.Y.0, for example 4.3.0.
• Issues that are not complete roll-over to the next milestone (Y+1) or the milestone due-date is pushed forward (be flexible!)
• When we have agreement the milestone is met, either via discussion in #core or during office hours we close the milestone and make a GitHub tag/release, marked pre-release in the GitHub UI.
• Tooling (or someone) builds milestone packages, shares them to #core, and requests testing.
• The master branch is semi-frozen meaning only documentation, small changes, and bug fixes are merged.
• After a period of feedback, for example 2 weeks, if changes were required from testing a new 0.0.Z release is tagged from the milestone release. Packages are published off of this release.

In this model we have two main benefits:

• We mark/record when we hit milestones and protect against scope creep.
• Milestone releases encourage beta testing.

One large downside:

• It's not clear that milestone releases are treated as traditional release-candidates.

[directionless]

ooo, I was also just thinking about this!

I agree with your highlevel points about how we should treat milestones.

One of the things I wanted to work through, was how the release candidates, git tags, github releases, and changelog work.

Some things I believe. There may be disagreement:

1. git tags are only loosely coupled to github releases
2. We should not re-tag things.
3. Our tests are probably never good enough pre-catch bugs, we're going to need to ship RC candidates for testing
4. Automate all the things. There should be minimal humans
5. release candidates don't need full release notes. They get super confusing if they had them.

This leads me to thinking that we might have a process along the lines of:

1. Starting when a github milestone is declared as met
2. tag X.Y.X-rc1
3. Trigger full CI/CD process to build and signs packages
4. Now the nightly package repo contains X.Y.Z-rc1 packages
5. Hopefully various people test the package
6. Time passes
7. If there are bugs found, we tag a new X.Y.Z-rc2 and go back to step 5
8. When we're happy, we make a a X.Y.Z release with full release notes
9. CI/CD builds and signs packages
10. Humans update the web site

An alternate process might look like editing and re-tagging the github release. Not sure

The thing I'm most leery of is what semi-freezing master means while an RC is baking. I'm not sure we'd want a new table to sneak into a release, but I'm less sure if we should semi-freeze, or do some other tag/branch shenanigans. Freezing is easiest, so we should probably start with that

[theopolis]

Re-tagging is out of the equation, everyone agrees on this.

I personally do not like “-rcN” tags, I’d rather call “X.Y.0” as ‘the milestone and initial release candidate’, and follow up with .1, .2, .3 as more stable versions until we decide to push one to repos/downloads. However, along the way, each tag receives a change log update.

[directionless]

One thing I like about -rcN (or whatever) is that it's easy to tell whether a release is a RC or a production release. I think the counter argument is that the difference is arbitrary. We should release X.Y.0, and if we need to fix it, we can fix it. I think in that sense, we're be adopting something closer to an everygreen style -- HEAD is always a release, and sometime we stick a nice label on it.

However, along the way, each tag receives a change log update

This ties into the above. If I view .0 as an RC, and then .1 as a release, I don't know what goes in the changelog.

But, if I think of .0 as a release, that may or may not be promoted to "stable", then I agree with you.

Said differently, I think you're proposing (and I'm willing to go with it) something like:

• osquery does not have release candidates. osquery has releases
• We aim for major/minor bumps to happen at a regular cadence, and be linked to milestones
• Patch releases are primarily to address stability issues
• orthogonal to an osquery release, is the "stable" designator
• After an osquery release has been suitable tested, it is deemed stable, and promoted as such
• We believe the github "prerelease" checkbox is primarily gloss, and may or may not be used to indicate something that has been deemed stable.

[directionless]

Downside to that approach is that I don't think one can easily tell from a package, binary, or version whether something was stable. I'm not sure it matters

[theopolis]

The GitHub release pages, and whatever is published to the website for downloads would be the source of truth for what is marked stable, seems reasonable, ya?

[directionless]

Yeah. To tell whether a release is stable, you need an outside resource (like github, or downloads). Else all you can tell is that it's version "X.Y.Z"