Support adaptors with version @next

[josephjclark]

I need help!

I am implementing a feature where adaptors may be pre-released with the version @next.

Background

Prerelease adaptors will be developed by an engineer and allow early-access to the adaptor. This lets us run QA inside lightning.

At the moment, we tend to release an adaptor, have implementation try it out in lightning, and then hive feedback which maybe results in a new PR. Not a good process.

Adaptors will be published to npm with the @next tag.

Not all adaptors will have @next tagged builds.

Implementation Notes

Worker support is added in https://github.com/OpenFn/kit/pull/716

Lightning needs to provide access to the next release, if appropriate, in the adaptors dropdown. Something like this:

In the select, the next version should be at index 0, the latest version should be at index 1, and all other versions from 2+.

You can literally send next to the worker like a full semver string. The worker knows what to do with it and will handle caching and autoinstalling elegantly.

Lightning's adaptor picker should ignore versions with the -next-<hash> suffix. This is a redundancy that means if a version somehow fails to be deprecated by automation, it won't appear in the drop-down list. You need a regex like /-next-(.+)/

[josephjclark]

hmm. It looks like deprecating old prereleases is gonna be kinda hard.

I'm still working on it, but maybe Lightning will have to ignore all versions of the pattern x.y.z-next.n, which are all legacy pre-release builds and shouldn't spam up the version list.

Update: Yes, Lightning should ignore /-next.\d/ suffixes. It should be easy to do and be a robust solution

[josephjclark]

We may also have to solve this, otherwise we'll have to manually restart version every time we release a next https://github.com/OpenFn/lightning/issues/2209

Actually, it's only the FIRST prerelease that's sensitive. Once next is in the drop down list, the worker will handle versioning. We just need lightning to add next to the list as soon as possible, and also remove it from the list once it's released

[josephjclark]

I am a little concerned about this use-case:

• I write my step with the @next adaptor version
• Everything goes great, the adaptor gets merged, the @next tag gets deprecated, a new @latest is released
• But the step is still locked to @next, which no longer exists, so the workflow will fail.

I am not sure what to do about this one. I can't think of any way to auto-migrate steps from next to latest on release. It's going to have to be on the user to go in and update the version - which is annoying because last time they looked at the workflow, it worked great!

We also can't not deprecate the old @next packages - even if we did, the adaptor would slide onto the NEXT @next when it's released, which can also cause problems.