Auto-publish Adaptor Release Candidates

josephjclark commented 3 months ago

Overview

This PR enables adaptor release candidates to be published to npm with the @next tag and a pre-release version number.

It allows us to test adaptors in Lightning and the CLI without having to release to the public.

Support added to CLI and Worker in: https://github.com/OpenFn/kit/pull/716

Issue

Closes #585

tl;dr How it works

Run pnpm next:start to generate the first pre-release build
Whenever a new changeset is pushed to the branch, the pre-release is updated
An automated commit is made with the new version number (devs must pull this back down! No force pushing!)
The adaptor is published to npm with the @next tag, where it'll only be used when explicitly referenced
Push a new changeset to trigger a new pre-release build
Run pnpm next:end to close down the pre-releases and prepare the final build

Strategy

changesets has support for pre-releases - so to start this process off I'm just building changeset functionality into our workflow.

It's OK, but it's a bit more manual than I'd like it to be. This does at least mean we're being more intentional about posting release candidates to npm, and will result in fewer junk builds.

Maybe later we can build something more automated and tailored.

How it works

Developers will create new adaptors/features/fixes and open up a PR to track the changes.

When ready for testing, developers run pnpm next:start and check in the changes. This will generate the first prerelease build, which will auto deploy with a version number like 1.0.0-next.0, tagged as @next, and released to npm.

This next release will be available to users through the CLI and Lightning (with a little bit of work, I think)

Every time a new release is generated, an updated package.json will be committed back to the branch. This is a bit annoying because the user's branch is always outdated. Devs must resist the urge to force push back. Even during development I keep forgetting about this and getting stuck out of date.

New commits to the branch will NOT trigger new @next builds. When ready to build a new version, generate and push a new changeset to describe the changes.

Alternatively, you can manually bump the version number and push, which will trigger a new release (but not a commit).

When ready for the final release, run pnpm next:end to close off the pre-release version. You can now run pnpm changeset version to generate the final release, which obviously won't have the next tag or the -next.x suffix.

Note that when compiling the final release notes, we may want to remove some of the prerelease notes - they're not likely to be interesting to the public.

When a new @next tag is published for an adaptor, the old one is automatically deprecated. It's the best way I can see to tidy up a little bit.

TODO

[x] right now if pre.json is checked into main, the whole prerelease workflow will fail. I suppose it'll be a positive fail. But I basically want a more robust check than "if pre exists, assume there's a prerelease active"
[x] optimise the workflow. I don't want to install & build twice before publishing, but I do want Ci/test and prerelease/publish to be different workflows. I'd also ideally like prerelease to only appear if there's a .pre.json file - I'm not sure I can run job conditions against the file system like that? Done well enough for now.
[x] Make sure @next is supported by the CLI and worker. I need to ensure that if version is next, we always check that the latest next is available locally
[ ] We need a little support from Lightning to handle next nicely https://github.com/OpenFn/lightning/issues/2208
[x] Whenever we publish a new @next, we should deprecate the old one. This stops it showing up in Lightning.
[x] Set the test adaptor up with a better name, maybe exclude it from lightning, put @latest on 1.x and @next on 2.x
[x] Take Mutchi's face off the automated commits (change the env vars)
[x] restore CI back to a working state
[x] on publish, deprecate any outstanding next tags

josephjclark commented 3 months ago

The work so far uses pnpm changeset pre to handle pre-releases.

It's a very manual process because it requires commits to be made.

It's not going to work though, because when publishing the next version, we need to commit the changes. Which means your local branch is always out of date, which I think is an annoying dev experience.

I dunno, maybe I'm being dogmatic about that

josephjclark commented 3 months ago

This commit triggered an @next release of cartodb :tada: https://github.com/OpenFn/adaptors/actions/runs/9502748928/job/26191512475

Here it is: https://www.npmjs.com/package/@openfn/language-cartodb/v/0.4.0-next.0

Rather upsettingly it seems to have been tagged with latest, which is very much not what I was expecting.

So actually it failed, basically.

Also I didn't commit the changes made in CI, so I think my next commit will fail to update that as it'll just try and publish .next.0 again

Posting a screenshot because i need to remove that build now

josephjclark commented 3 months ago

I would love to add a commit hash to the build number, instead of this -0, -1 stuff (or as well as?)

I'd also love nice detailed commit messages

josephjclark commented 3 months ago

hmm. I'm really struggling to get the prerelease version to bump. I don't quite understand how it works.

What I want is: when I run pnpm changeset version in pre mode, any pre versions get their package.json bumped. The logic being: I'm testing a prerelease, I find a bug, I make a change and commit it, and now I want to bump the version number.

But it feels like the number will only be bumped if there's a new changeset to go with it.

As it is now, I can make a change and run changeset version and nothing gets bumped.

Do I REALLY have to go in manually and set the version when I want a release? Urgh, no thanks. That does at least work, but no thanks to changesets.

josephjclark commented 3 months ago

Ok, I think I need to accept that we'll only get a version bump with a changeset.

That means we'll have to clean up the release notes when merging the final release, because I anticipate that many of these prerelease changesets are bunk - not interesting to users.

josephjclark commented 3 months ago

I wonder if there's any way that docs can show next versions.

This isn't essential and originally I didn't think it was worth bothering.

But it would be good to test docs in context and make it easy to see docs changes. I've made fixes so that you can locally see the docs build, but that's hard for other people to review.

josephjclark commented 3 months ago

Deprecation is in and works fairly nicely

Every time we build a pre-release, we lookup the current @next version and deprecate it before publishing the next.

It's not totally foolproof - if for any reason the deprecation fails, there will be no other automated attempt to deprecate it. We get one shot. It should work 99% of the time though - it's reasonably robust, there's just no redundancy if it fails.

josephjclark commented 3 months ago

I'm a bit worried about how changeset bumps version numbers.

I just pushed a major version changeset, hoping it would bump the pre-release version to 2.0.0. But it didn't.

I think the pre-release version number is just @latest plus the incrementing build number, then the final version number is generated when the pre-release has been exited.

I suppose that solves problems like If I go from 1.0.0-next-x to 2.0.0-next-y, what is y? Is it x + 1 or is it 0? It should be 0 really, but you lose the consistency of the build version. But I'm not worried about that - I'd prefer that the next release version is the same as the actual release version, minus the suffix

josephjclark commented 3 months ago

Unfortunately despite all this effort I'm just not liking the changeset workflow very much.

josephjclark commented 3 months ago

The automated commit thing is really irritating.

Should we make this an entirely manual process?

Run pnpm next:start to go into pre mode
When you want to bump next, manually bump package.json or add a changeset and run pnpm changeset version
Either way, commit the bumped version to branch and push.

It's still not a nice process like that

josephjclark commented 3 months ago

One more little problem: when the changeset is exited, we need to deprecate and remove the final @next tag. Or maybe on publish. Because as soon as we publish, @next is no longer the @next.

Maybe we can add this to the publish pipeline actually...

josephjclark commented 3 months ago

Right, I'm closing this down.

The biggest problem with this approach is the remote commit. Every time we push, the automation will generate a new version number and commit it back to branch, which means your local branch is out of date. Which inevitably means you get a stupid conflict and confusion every time you try a subsequent push.

In addition, I have several problems with the changeset process:

It's a manual opt-in. You have to manually start AND end the "pre" process, which requires training and special processes
Version numbers don't bump correctly. The version number seems to be <latest>-next-<n>. But what I actually want is to do <next>-next-<n>. In other words, take my changesets, work out the next version number on npm, and publish THAT version number plus the next stuff
The incrementing build number at the end is what brings the requirement to commit every time.
The build number only actually increases when we submit a new changeset. But changesets are user facing - not all builds will have user facing changes! So this is a silly system.
It does nothing to help me tidying up old release candidates and not leave npm in a mess

OpenFn / adaptors