transition to versioned releases, instead of releasing on a branch

[edwardhartnett]

Currently the release system for rrfs-workflow is to have a branch of a known name, and whatever is on that branch is the release.

This is not a great process. Branches can change over time, and for this reason the software world has the versioned release.

For example when we use netcdf-c in the UFS, we don't use any branch of the netcdf-c repo, which may change over time, we use a versioned release (4.7.4 for example), and that stays the same forever. When netcdf-c has new features we need to use, we can upgrade to a newer version, which has a different version number. This is well-understood and managed by all build and install systems.

As we transition to the use of spack for WCOSS2 installs, rrfs-workflow needs to be released as versioned releases. We do not want to install a branch, which may be changing from time to time - we need to know exactly what we are installing everywhere, and we need that to be fixed.

The way to achieve this:

1. rename most active current branch to "main"
2. release current main as version 2.0.0.
3. set up CI and branch protection of main.
4. develop and maintain spack package file in the repo.
5. do quarterly releases off main.

Since the cost of release for an agile project is low, we can release whenever we need to. Quarterly should be considered a minimum cadence.

Versioned releases have many other advantages for software development. That's why literally every other software project in the world uses them. rrfs-workflow needs to use them as well.

[edwardhartnett]

In discussion with @MatthewPyle-NOAA we have decided on the following course of action:

1. Copy branch production/RRFS.v1 to new branch "main".
2. Set main as the default branch.
3. Set up branch protection for main: PRs only, successful builds.
4. Do a Version 2.0.0 release from main branch.
5. All new development to take place on main branch - existing branches left alone until no longer used.
6. Develop a spack package file.
7. At least quarterly releases thereafter.

This will happen early this week.

[MatthewPyle-NOAA]

Any chance you could delay it until late this week or early next week? I'd like to share these plans with the broader rrfs-workflow team at a Thursday meeting. Also, will existing (non-main branch) branches be locked with the unification around main?

[edwardhartnett]

Certainly we can wait until the end of the week.

[guoqing-noaa]

Please keep the rrfs-mpas-jedi branch which is a work parallel to the upcoming "main" branch.

[edwardhartnett]

No branches will be deleted in this operation.

[edwardhartnett]

The idea has been raised of installing rrfs-workflow from git, with other software as submodules.

Submodules are favored here at NOAA, but they are a poor choice. They cause software to be overly coupled and also untestable. (How do you test the submodule if it is not a stand-alone component?)

Installing as a git clone comes with many problems: for example, we get every file in the repo, although we certainly don't need every file in the repo. When installing with git, we can't control what's installed or where each file goes.

Finally, NOAA is moving towards an all-spack solution for NCO. In other words, we will no longer install software by cloning a git repo - spack will install everything. And spack works best with versioned packages.

So instead of using submodules, we will release rrfs-workflow as a stand-alone installable package. This is how all other software in the stack is released and managed. Installing as a package will also give us full control over what is installed.

The end result is that when we install rrfs-workflow on WCOSS2, we will be able to do so in a standard way, and then we can run the built-in unit tests to assure that everything is working OK on that platform. Then people and other packages can use rrfs-workflow in the usual way.

NOAA should not be originating new ways to distribute software. There are excellent reasons why software is distributed as versioned packages, and not as clones of a git branch. Instead of attempting to originate new methods of software distribution, let's just do what everyone else does, because that is easiest and cheapest. When you use netCDF, you do not use a branch cloned into a submodule - you use a stand-alone installed package with a well-defined version number. That makes it much easier for everyone to manage the installation of software.

The good news is that this will have very little impact on development (except to make it easier). Programmers do not have to care about releases or installations. Just work on the (soon to be created) main branch, and leave releases and deployment to Matt, the NCEPLIBS team, and NCO.

[edwardhartnett]

OK, we have successfully transition RRFS.v1 to versioned releases. I will close this issue and we will handle RRFS.v2 under another issue.