Improve development workflow for DX Hub plugins

[wasadigi]

Context

The initial version of this repo was not designed to release the individual plugins we will add to DX Hub over time. We want to make these plugins available to Backstage adopters and therefore need a way to publish them via the central npm registry.

This question fits into the broader context of managing this repo as a "mono repo". Looking at the evolution and the current state of these repos, we see that there is no single approach:

• Some organizations have migrated to yarn 3, while others are still using yarn 1
• Some organizations have dropped lerna while moving to yarn 3
• Some organizations have upgraded to a more recent version of lerna
• The pipelines, implemented as GitHub workflows, vary quite a lot

Here are (some of) the repos that we have studied:

• https://github.com/backstage/backstage
• https://github.com/RoadieHQ/roadie-backstage-plugins
• https://github.com/thefrontside/playhouse
• https://github.com/janus-idp/backstage-plugins

Approach

We will start with a simple solution and evolve it over time. The objectives of the first step are:

• stay on yarn 1 until it becomes the recommended version for Backstage (today, the documentation explains how to migrate to yarn 3, but the scaffolder still uses yarn 1)
• upgrade to the latest version of lerna
• use the independent mode for lerna, so that we can manage versions for our plugins independently
• implement a release pipeline that publishes plugins on npmjs.com

[wasadigi]

After digging into this topic, only using lerna does not seem to be enough. Running lerna version forces us to specify a new version for every package (even the packages that have not been modified).

We are going to use changesets and the companion GitHub action to help us. The main reason is that it is used both in the backstage/backstage and in the RoadieHQ/roadie-backstage-plugins repos (with different configurations...).

This article compares some alternatives to changesets: https://www.hamzak.xyz/blog-posts/release-management-for-nx-monorepos-semantic-release-vs-changesets-vs-release-it-

[wasadigi]

This is a great, short video about changesets and the companion GitHub action: https://www.youtube.com/watch?v=eh89VE3Mk5g

[wasadigi]

Another one, with more details about using the changesets cli: https://www.youtube.com/watch?v=vO80X5zM8_Y

[wasadigi]

See the origin of using changesets in Backstage: https://github.com/backstage/backstage/issues/1839

[wasadigi]

I have installed: https://github.com/apps/changeset-bot. Unfortunately, it does not work because our .changeset directory is not at the root of the repo. This is related to https://github.com/changesets/bot/issues/60.