Release process

[danshearer]

@kenmeacham I have committed the first version of release-process.md . Please can you review/fill in/etc. We will need a few rounds of this.

This is related to @scp93ch's comments in #68 .

My goal in this issue is to get a practical and lightweight process in place. We will certainly be revisiting this.

Things to note:

• There is nothing yet in the "run tests" section. What should be the minimum sanity check to run? I propose that if we make the version tag and update system-modeler-deployment, and then do a successful deploy, that this is sufficient for now. Should I have more confidence in the Java tests?
• I refer to a currently non-existing mailing list, but we are getting that sorted out right now. This is an example of why we need lists.
• After discussion with Ken, I dispensed with the need for third subversion number, as in "3.5.0", because it will always be 0. So the next release will be just 3.6.

[kenmeacham]

Just to comment initially on these points:

* There is nothing yet in the "run tests" section. What should be the minimum sanity check to run? I propose that if we make the version tag and update system-modeler-deployment, and then do a successful deploy, that this is sufficient for now. Should I have more confidence in the Java tests?

Normally when creating a release of SSM/system-modeller, there have already been several rounds of running Java unit tests (>600), to filter out any obvious problems due to recent code updates. Nothing should be released that has failing unit tests, and in fact the merge from dev to main would normally be disallowed due to this.

Of course, we assume that all code is being tested, i.e. there is good coverage, and with various conditions tested, but we know that this is never perfect.

Once unit tests are passing reliably, we normally run Spyderisk in bootTest mode to check basic functionality via the GUI (as we don't have any unit tests for this). This process is a bit ad-hoc of course. I would normally test all the common functions, e.g.

• create a model, add a few assets, relations
• validate model
• run risk calculation
• navigate through a few threats, consequences, etc.

Then, if there are known new features, test these (as they should have been during development), and any related features that might be affected by any changes.

If all of this seems OK, we can proceed to run similar tests on the release candidate, via system-modeller-deployment.

* After discussion with Ken, I dispensed with the need for third subversion number, as in "3.5.0", because it will always be 0. So the next release will be just 3.6.

I'm not sure this is what I said, when we last discussed, What I think I meant was that we wouldn't generally make very many incremental releases of Spyderisk (e.g. 3.6.0, 3.6.1, 3.6.2, etc), as the overheads are too great. We would normally make a new release containing a reasonable set of new features, bug fixes, etc. No fixed timescales - could be around monthly, but more normally every 3-months or so. In these cases, it makes sense to increment the 2nd subversion number (particularly if the REST API has changed). However, I would still keep the 3rd subversion number (@panositi agrees with this), as this is conventional.

If we make a release then soon after find a major bug that needs fixing in it, it makes total sense to then fix this and make a minor version release (e.g. 3.6.1).

After discussing with @panositi, we felt it would be good to document the general rules about when we increment the particular subversion numbers.

Hope that helps. I'll also look through the document and make comments, etc.

[danshearer]

I don't mind about version numbers, but I do think you raise a really important point about policy. I think the tricky bit is perhaps conflating "minor release version" with "minor amount of work?". And to be fair we don't know how much work, because we haven't written the release process yet :-)

@kenmeacham wrote:

[... x.y.z point releases instead of just x.y ] as this is conventional.

It is only conventional for projects that are a lot more mature than Spyderisk is today. Spyderisk is a long way from stable software, and we are not as well-resourced as we would like, and this has policy implications. I will make a policy proposal at the end, but first let me give an example of where I think your proposal leads:

• we release Spyderisk 3.6.0, correctly describing it as unstable and experimental software, with zero support guarantees (although our developers are truly friendly and responsive, and always glad to hear from users)
• over the next 2 months we merge features and bugfixes to main, and release Spyderisk 3.7.0. Hooray!
• oh no! We have a bad security bug, so we tell people "download 3.7.0, or, if you want to keep using 3.6.x series we have just released 3.6.1.

But that's a problem, because:

• 3.6.1 is a backported fix. Backport engineering might seem easy but it often isn't, and this particular skill gets harder as the .z releases increase to 3.6.2 and 3.6.3 etc. See the paper Backports: Change Types, Challenges and Strategies. Who will do this? We do not currently do any backporting.
• We now have to follow the Release Process twice, once for 3.7.0 and again for 3.6.1. We can't skip any testing, and we have to regenerate images and whatever else is involved in a release. That's a lot of effort, on top of the backporting.
• We just told the world that we are offering a support guarantee, in the sense that we will do our best to fix bad bugs in old release versions of Spyderisk. But we don't offer this, and in fact the opposite is true... we can only really promise to do our best to make sure the next release is an improvement on the current one.
• We just discouraged our own userbase! One of the most valuable things we can get is testing on our current release version, and we absolutely don't want our users hooked on old versions of software.
• We just made an implied security guarantee as well, because typically that is the main reason for a .z release

I propose the opposite policy, something like this:

Spyderisk is not stable software and we only have one supported version, which is the latest release version. If you experience a bug or security problem with Spyderisk we really want to hear from you, but also, do please check to see you are running the latest version. Our goal is to make each version better than the one before it, and if on occasion that is not true we will fix the bug or regression as quickly as we can and release a new version. But people using non-latest versions of Spyderisk are pretty much on their own.

What do you think of this? Perhaps there is some midway point between these two. I feel that we are in a special time for Spyderisk at the moment, where our userbase is small enough and keen enough that we can just expect them to be using our latest.

[scp93ch]

I don't agree that [sematic versioning] "is only conventional for projects that are a lot more mature than Spyderisk is today". If you want to put the most trivial of modules into NPM for instance, then you use semantic versioning.

Having 3 numbers in the version does not imply that you are going to go back and do bug fix releases on older versions. In your example above, if a security issue was found we would have to say "download 3.7.0 and don't use 3.6.0". If we then quickly found another bug we might decide to release 3.7.1, but it's likely that any bugs would just be rolled into the next feature release (3.8.0).

So I don't think the 5 problems in your list really exist. But I do agree that the paragraph which basically says "use the latest" is sensible.

[kenmeacham]

I don't agree that [sematic versioning] "is only conventional for projects that are a lot more mature than Spyderisk is today". If you want to put the most trivial of modules into NPM for instance, then you use semantic versioning.

Agreed.

Having 3 numbers in the version does not imply that you are going to go back and do bug fix releases on older versions. In your example above, if a security issue was found we would have to say "download 3.7.0 and don't use 3.6.0". If we then quickly found another bug we might decide to release 3.7.1, but it's likely that any bugs would just be rolled into the next feature release (3.8.0).

Agreed. I don't think we would generally be able to go back to older releases and patch them - only if the latest release had a significant issue, we'd create a patched release of this. But generally issues are collected into the next significant release, as Stephen said.

So I don't think the 5 problems in your list really exist. But I do agree that the paragraph which basically says "use the latest" is sensible.

Generally, use the latest, yes.

[kenmeacham]

Spyderisk is not stable software

This seems overly negative, implying that you shouldn't use it. Perhaps more "Spyderisk is not guaranteed to be stable". Bear in mind that we have fixed hundreds of bugs in the past, so it is actually far from unstable I would say.

[danshearer]

Ok I accept all the above - we will emphasise "Use the latest version", and, we will use x.y.z numbering.

There is a also a more precise way of discussing churn than the emotive term "stable", and I think that would be better than my use of the term "unstable".

I propose that it is more correct to say we don't guarantee compatability between successive versions of pretty much anything. There are many versioned components in Spyderisk (the software, the deployment script, the core model and its subcomponents, the domain model, other runtime envrionment state, even docs.) Very commendably, there are already big warnings on the domain model to say we expect breakage if the version of the domain model doesn't match particular other versions of things. We should continue this good practice elsewhere in the project.

Over time we would expect to move to a situation where our thousands of users get advance warning about breaking changes, but that is some long distance in the future. Presumably we would do our best to provide a migration path for people with system models if we break the ability to import old ones, but we can't promise that.

[danshearer]

... and our own API versions of the different APIs, and versions within the npm stack, etc etc.

[scp93ch]

I noticed there was a section in the release notes about setting the version number in the gradle build script. I didn't think this version string was used now or had been used for a while. We recently made it so that the "about" information available in the web UI, including the version string, was taken from the environment variables for instance.

[kenmeacham]

I noticed there was a section in the release notes about setting the version number in the gradle build script. I didn't think this version string was used now or had been used for a while. We recently made it so that the "about" information available in the web UI, including the version string, was taken from the environment variables for instance.

Yes, I was wondering about this section too. I wasn't sure if the version string was still used, so didn't want to just remove it. It is referenced in a couple of places in build.gradle:

version = systemModellerVersion  // picked up by sonarqube

and

jar {
    baseName = systemModellerName
    version = systemModellerVersion
    dependsOn(':src:main:webapp:bundleProd', 'extractWebFiles')
}

Is a versioned jar file actually created for system-modeller, because I can't seem to find one.

If we could remove the systemModellerVersion bits, it would save us some effort in the release process...

[scp93ch]

Yes, I was wondering about this section too. I wasn't sure if the version string was still used, so didn't want to just remove it. It is referenced in a couple of places in build.gradle:

version = systemModellerVersion  // picked up by sonarqube

and

jar {
    baseName = systemModellerName
    version = systemModellerVersion
    dependsOn(':src:main:webapp:bundleProd', 'extractWebFiles')
}

Is a versioned jar file actually created for system-modeller, because I can't seem to find one.

If we could remove the systemModellerVersion bits, it would save us some effort in the release process...

I hadn't realised it was there still. Raise a new issue on it or perhaps reopen #68 as it is very closely related (we should remove the old cruft when adding the new method).

If a system-modeller jar is built then I guess the version info will be in the metadata inside the jar. However, it's a WAR file we build so I am confused.