Find a better tool to manage versioning, changelogs and publishing

[jroebu14]

Is your feature request related to a problem? Please describe. Managing dependencies in Psammead can be time and energy consuming. We built Talos and bumpPackages and bumpChangelogs scripts to manage versioning, changelogs and publishing in Psammead. Talos does its job but finding time to improve it is difficult given the team's now limited resources.

Describe the solution you'd like We'd like to investigate if there are other popular open-source solutions that could work in Psammead to manage this whole process. A streamlined versioning, changelogs and publishing process could free up a lot of our time.

• Changesets by Atlassian looks interesting and would be a good starting point
• Also Semantic Release

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Testing notes [Tester to complete]

Dev insight: Will there be any potential regression? etc

• [ ] This feature is expected to need manual testing.

Additional context Add any other context or screenshots about the feature request here.

[jroebu14]

The current Psammead changes/release process

1. Make changes
2. Commit changes
3. Bump package version
4. Open PR
5. Update Changelog
6. Get Approvals
7. Merge changes then triggers release
8. If package is an inter-dependency, Talos creates PRs bumping inter-dependant Psammead packages affected by release
9. Manually approve and merge Talos PRs
10. Repeat steps 8 and 9 until inter-dependencies are fully up-to-date

NB

• No concept of tagging our releases
• No commit standards

Pros

• We are all used to this process and we've managed to ship features effectively with it thus far
• No commit conventions to follow

Cons

• Versioning and changelogs is a manual step and can become hard to manage when doing sweeping changes
• Releasing on merging of changes (and then bumping inter-dependencies) means we are releasing more new versions than we have to - is a burden for consumers trying to stay up to date
• We built our own tool to automate this process - Talos - that we don't have a lot of time to think about how to improve and work on it
• Inter-dependency bumping is difficult to manage e.g. bumping inter-dependencies after a new package version is released - can this not be done all in one go?
• We sometimes get major/minor/patch bumps wrong
• Bumping versions as part of the changes PR creates a lot of surface area for PR conflicts - can be a lot of effort to fix

What tools can we use to help?

Changesets

How it works

You open a Pull Request and include a changeset file (documentation of the changes and the desired semantic version bump e.g. major, minor, patch).

Merging the Pull Request does not trigger any release, instead a new Pull Request named Version Packages is created by the Changeset GitHub Action and includes the release plan. The release plan bumps the versions of all packages and dependent packages affected by the changeset files present in the repository, as well as the updated CHANGELOG.md files in each package.

When it’s time to trigger a release, you simply merge the Version Packages Pull Request. The Changesets GitHub Action takes care of doing the actual release, by publishing the packages to NPM, creating the git tags and the GitHub Release notes.

Pros

• Is focused on monorepos
• Built by Atlassian and used in a few popular OS projects
• Is open-source meaning it's free, Atlassian and the OS community will improve and evolve it, and we can also contribute to it
• Don't need to release on merging of changes
• Versioning PR is worked out automatically. Includes inter-dependency bumps
• I don't think it enforces any commit conventions. Debatable if this is actually a con.

Cons

• A slight change in our workflow with the addition of maintaining changeset files
• Adds another dependency to the project

References

• https://github.com/atlassian/changesets
• https://techblog.commercetools.com/adopting-changesets-for-release-and-changelog-automation-cbdc36bbdc10

Repurposing Talos

Talos exists to upgrade our internal dependencies (i.e. Psammead dependencies) after we bump and release new Psammead packages. Why can't we just upgrade internal dependencies at the same time as bumping/releasing new packages? Because when CD runs, all dependencies, including our internal ones are pulled from the NPM registry meaning the build will fail because the new packages haven't been released yet and of course aren't in the NPM registry. This is what Talos does - it creates the PRs with the internal dependency upgrades when they are available.

Now that we are using Yarn Workspaces, when you install, all internal dependencies are linked locally, meaning they are not fetched from NPM and builds will not fail. This means we can release new packages and upgrade internal packages at the same time and Talos doesn't need to run at all.

Based on this, we can repurpose Talos into a tool we can run locally before making a PR - Something like this CLI (that I completed on a learning day) https://github.com/bbc/psammead/pull/4423.

Pros

• Repurposing and reusing what we already have built. We get some things for free
• We can bump all packages in our PR and not have to manage follow-up PRs from Talos (described above in steps 8-10)
• No new dependencies, workflows or new tools to learn

Cons

• In-house built tooling comes with a maintenance cost. Will also need unit/integration tested before it can be used
• There will be edge cases we haven't thought about
• Algorithm for bumping might end up being complex and hard to understand/remember, especially after not working with it for a while - will need extensive documenting
• We end up maintaining our distance from the way the open-source community manages monorepos meaning we may miss out on important insights and improvements to our workflow
• It's specific to Psammead. No one else can use this tool