Discovery: Publishing multiple packages from the component-library repo

[k80bowman]

Description

With the addition of the CSS library package, we would like to explore tooling options for easy and automated publishing of multiple packages from the component-library repository. The result of this discovery should be the recommendation of a particular tool along with an analysis of it and other tooling options.

Details

Things to consider:

• We want to make the process as easy as possible for engineers. There should be very little that we have to remember to do manually to make the publishing process happen correctly.
• Whatever we choose it should be well-supported and well-documented
• While for now we are only looking to publish two packages, in the future we may want to do something like publishing each of our components as an individual package. Whatever tool we choose should be able to handle that.

Tasks

• [ ] Put together a list of tooling options to consider
• [ ] Review each option and document your analysis (can be a separate Confluence doc or a comment on this ticket)
• [ ] Choose a tool to recommend and review with the DST engineering team
• [ ] Create a ticket for implementing the recommended tooling option

Acceptance Criteria

• [ ] Tooling options have been identified and compared and the comparison has been documented
• [ ] A tooling option has been selected and discussed with the DST engineering team
• [ ] A ticket for implementing the tooling option has been created

[bkjohnson]

The main "off the shelf" solution I found is semantic-release. There are other players in this ecosystem for managing publishing & release notes in a monorepo, but none were as large or had the same kind of documentation. semantic-release-plus has some documentation on monorepos, but this is a less popular fork of semantic-release.

The hard part about this as I see it isn't just the publishing - it's having descriptive release notes which correspond to each individual package as an artifact of publishing.

Using semantic-release-monorepo

Similar to the work done in https://github.com/department-of-veterans-affairs/component-library/pull/490

We can see the semantic-release config in VAFSC, and this combined with their CI will create a GitHub release and publish the package to npm. The releases and release note generation is based around Angular commit conventions (fix, feat, etc.). Version numbers still have to be updated manually due to a limitation with the bot token for CI.

With the semantic-release-monorepo addition, the documentation doesn't make any mention of commit formatting. It's unclear how release notes would be generated and I couldn't find an example online where I could see the artifacts of this kind of setup. The configuration exports a generateNotes const, but the main documentation doesn't say anything useful about what this does or how it works. This makes me somewhat nervous about choosing this option.

Custom solution

Another option is to build out the CI mostly ourselves using Github's API. Here's an outline of what that could look like:

1. We have release note config templates for each of our packages
   • The main differences would be that each config file would exclude labels for other packages
   • For this to work our PRs would have to have labels for the package that they correspond to. This may mean that we couldn't use the same PR to make changes to the component-library package and the css-library package if we wanted that PR to appear in the generated release notes
2. On merge to main, CI would check if there had been a version change for any of the public packages.
3. If so, begin the release step for each package with an incremented version
   • Use the endpoint for generating release notes. We would pass it the configuration_file_path of the package-specific release notes config file
   • Save the name and body of the response
   • Use those values to create a release where the tag has the format [PACKAGE_NAME]-v[VERSION]
4. Once the release has been created, a publish step will trigger for publishing the package to NPM

A nice bonus would be if each package had its own changelog in addition to having its releases in the repos list of all the other releases. Since it seems like our Github bot token doesn't have permissions to make commits, I think we would have to use a new bot/token to do this automatically or have a script that we run and commit the resulting CHANGELOG changes locally. This automation could essentially copy the release notes for a package version and paste them into a file - I don't think we would need to do anything additional like format our commits a certain way so that it can parse them.

[k80bowman]

I'm curious, what other options did you look into?

[bkjohnson]

I looked at the options mentioned in this article.

[k80bowman]

I see NX mentioned in the comments, did you look into that? I also have heard that it's possible to publish using Yarn workspaces, but I don't see that in this article. Have you looked into that at all?

[k80bowman]

A couple of good resources to take a look at:

• https://github.com/korfuri/awesome-monorepo
• https://monorepo.tools/#monorepo-tools

[bkjohnson]

We're already publishing using Yarn workspaces. The real challenge is having separate release notes for each of those published packages.

I didn't look closely at nx, I'll see if it would help solve our problems.

[k80bowman]

There may be other options in those resources I linked to above as well. I would recommend taking a look at at least a few of them.

[bkjohnson]

In the awesome-monorepo link I found a list of notable public monorepos, one of which is a good example of what we want to achieve as a result of this discovery and followup development.

• celo-monorepo release celocli-v1.6
• celo-monorepo release contract-v2.1

That repo has Github Releases (along with release notes) and git tags specific to a package and version. Even with our current setup, we are building and publishing multiple packages, but we don't have separate releases. As an example, 11.6.0 of the component-library had changes to the core package and the web-components package, and we can see both of them being published.

• successful component-library publish (the "core" package")
• successful web-components publish

We're already using yarn workspaces, and that gives us tools to do common things across all packages, like manage dependencies and build and publish. Most of the tooling I've seen is designed to support that kind of functionality - running various scripts on some or all of the packages or making code sharing easier.

I haven't found anything in nx that looks like it would have the functionality to create releases and release notes like we would want it to. It looks like the semantic-release-monorepo plugin would support this functionality, but the documentation doesn't look great. I dug into lerna some more and found a blog post showing that the lerna version command has a --create-release option which could be what we want. I still haven't found a good example of that in action though. Even lerna's own "Getting started" example doesn't have any releases to show what's possible.

[k80bowman]

Lerna used to be state-of-the-art for monorepo management, but last year the team that had been maintaining it announced that they would no longer be doing that. Another team subsequently picked it up, but that doesn't give me a great deal of confidence in the future of that project, so while Lerna probably has great tools, I'm not sure we want to rely on it. Unfortunately.

I'm glad you've found a good example, though, that's awesome. Thank you for digging in a little further.

[bkjohnson]

I've also had to deal with some pain & frustration with lerna (the tools repo is a lerna repo) so I'm also hesitant of jumping into lerna for that reason alone. I didn't know about the change in ownership, I can see how that doesn't inspire confidence.

[bkjohnson]

Next steps are to create a dummy repo setup with semantic-release-monorepo and see if I can make that a bit more friendly & forgiving.

[bkjohnson]

I've been doing some more discovery in this testing repo and found out a few things.

semantic-release tags a release even when the version in package.json hasn't been bumped

This seems like it could be problematic if we're trying to group a few different PRs in the same release like I tried to do here.

Squash commits have to have very specific formatting

We already knew this - semantic-release works by analyzing our commits and looking for specific prefixes. In order to ensure that we "follow the rules", we'd probably want to add CI like this that can fail a PR if the title doesn't have the right formatting. Additionally, we may want to set our repo up so that the squash commit defaults to the PR title:

So that if CI says that the title is ok, we know that the commit message will be ok as well:

Major/breaking changes follow a different pattern

With feat/minor and fix/patch changes, the kind of change is in the commit message. Major changes don't have a corresponding prefix tag - instead they are indicated with a BREAKING CHANGE: feature X has been removed bit of text in the commit body. Having to manually add this for any breaking change seems error prone, especially since semantic-release would publish the package as soon as we merge into main.

Bug with choosing version

There seems to be a bug where semantic-release-monorepo doesn't use the version defined in package.json and instead makes up its own version. This is similar to other incidents listed.

[bkjohnson]

Action item: Setup another dummy repo for experimenting with lerna.

[caw310]

@bkjohnson , closing this out. Please write up ticket(s) for any work going forward based on this discovery.