Hacky semantic-release for monorepos
Proof of concept that wraps semantic-release to work with monorepos.
This package should work well, but may not be fundamentally stable enough for important production use as it's pretty dependent on how semantic-release works (so it may break or get out-of-date in future versions of semantic-release).
One of the best things about semantic-release is forgetting about version numbers. In a monorepo though there's still
a lot of version number management required for local deps (packages in the same monorepo referenced in dependencies
or devDependencies
or peerDependencies
). However in multi-semantic-release the version numbers of local deps are
written into package.json
at release time. This means there's no need to hard-code versions any more
(we recommend just using *
asterisk instead in your repo code).
yarn add multi-semantic-release --dev
npm i multi-semantic-release -D
multi-semantic-release [options]
npx multi-semantic-release [options]
Configuration for releases is the same as semantic-release configuration, i.e. using a release
key under package.json
or in .releaserc
file of any type e.g. .yaml
, .json
.
But in multi-semantic-release this configuration can be done globally (in your top-level dir), or per-package (in that individual package's dir). If you set both then per-package settings will override global settings.
multi-semantic-release does not support any command line arguments (this wasn't possible without duplicating files from semantic-release, which I've tried to avoid).
multi-semantic-release automatically detects packages within workspaces for the following package-managers:
Make sure to have a workspaces
attribute inside your package.json
project file. In there, you can set a list of packages that you might want to process in the msr process, as well as ignore others. For example, let's say your project has 4 packages (i.e. a, b, c and d) and you want to process only a and d (ignore b and c). You can set the following structure in your package.json
file:
{
"name": "msr-test-yarn",
"author": "Dave Houlbrooke <dave@shax.com",
"version": "0.0.0-semantically-released",
"private": true,
"license": "0BSD",
"engines": {
"node": ">=8.3"
},
"workspaces": [
"packages/*",
"!packages/b/**",
"!packages/c/**"
],
"release": {
"plugins": [
"@semantic-release/commit-analyzer",
"@semantic-release/release-notes-generator"
],
"noCi": true
}
}
Make sure to have a packages
attribute inside your pnpm-workspace.yaml
in the root of your project.
In there, you can set a list of packages that you might want to process in the msr process, as well as ignore others.
For example, let's say your project has 4 packages (i.e. a, b, c and d) and you want to process only a and d (ignore b and c). You can set the following structure in your pnpm-workspace.yaml
file:
packages:
- 'packages/**'
- '!packages/b/**'
- '!packages/c/**'
Note, workspace:
prefix in pkg versions is not supported yet. issues/85
Make sure to have a bolt.workspaces
attribute inside your package.json
project file.
In there, you can set a list of packages that you might want to process in the msr process, as well as ignore others.
For example, let's say your project has 4 packages (i.e. a, b, c and d) and you want to process only a and d (ignore b and c). You can set the following structure in your package.json
file:
{
"name": "msr-test-bolt",
"author": "Dave Houlbrooke <dave@shax.com",
"version": "0.0.0-semantically-released",
"private": true,
"license": "0BSD",
"engines": {
"node": ">=8.3"
},
"bolt": {
"workspaces": [
"packages/*",
"!packages/b/**",
"!packages/c/**"
]
},
"release": {
"plugins": [
"@semantic-release/commit-analyzer",
"@semantic-release/release-notes-generator"
],
"noCi": true
}
}
There are several tweaks to adapt msr to some corner cases:
Flag | Type | Description | Default |
---|---|---|---|
--sequential-init |
bool | Avoid hypothetical concurrent initialization collisions | false |
--debug |
bool | Output debugging information | false |
--first-parent |
bool | Apply commit filtering to current branch only | false |
--deps.bump |
string | Define deps version update rule.
|
override |
--deps.release |
string | Define release type for dependent package if any of its deps changes.
|
patch |
--deps.prefix |
string | Optional prefix to be attached to the next version if --deps.bump set to override . Supported values: ^ | ~ | '' (empty string) |
'' (empty string) |
--dry-run |
bool | Dry run mode | false |
--ignore-packages |
string | Packages list to be ignored on bumping process (append to the ones that already exist at package.json workspaces) | null |
--ignore-private-packages |
bool | Private packages will be ignored | false |
Examples:
$ multi-semantic-release --debug
$ multi-semantic-release --deps.bump=satisfy --deps.release=patch
$ multi-semantic-release --ignore-packages=packages/a/**,packages/b/**
You can also combine the CLI --ignore-packages
options with the !
operator at each package inside package.json.workspaces
attribute. Even though you can use the CLI to ignore options, you can't use it to set which packages to be released – i.e. you still need to set the workspaces
attribute inside the package.json
.
⚠️ Keep in mind, that allowUnknownFlags
is enabled, so the rest of flags will be passed to internal semrel
call as options
argument for all packages.
multi-semantic-release default exports a multirelease()
method which takes the following arguments:
packages
An array containing string paths to package.json
filesoptions
An object containing default semantic-release configuration optionsmultirelease()
returns an array of objects describing the result of the multirelease (corresponding to the packages
array that is passed in).
const multirelease = require("multi-semantic-release");
multirelease([
`${__dirname}/packages/my-pkg-1/package.json`,
`${__dirname}/packages/my-pkg-2/package.json`,
]);
Multi-semantic release seems to be compatible with many CI/CD systems. At least we are sure about three of them, here are examples of configurations:
When releasing a monorepo you may get a npm ERR! code ETARGET
error. This is caused by npm version
creating a reify update on packages with future dependency versions MSR has not updated yet.
The simplest work around is to set workspaces-update to false either in your .npmrc or manually by running npm config set workspaces-update false
When releasing a monorepos you may get EINVALIDNPMTOKEN
error. The more packages, the more chance of error, unfortunately.
INVALIDNPMTOKEN Invalid npm token.
The npm token (https://github.com/semantic-release/npm/blob/master/README.md#npm-registry-authentication) configured in the NPM_TOKEN environment variable must be a valid token (https://docs.npmjs.com/getting-started/working_with_tokens) allowing to publish to the registry https://registry.npmjs.org/.
Do not rush to change your token. Perhaps this is related to npm whoami
request throttling on your registry (just a hypothesis: https://github.com/semantic-release/npm/pull/416). At this point you can:
This error seems to be related to concurrent git invocations (issues/24). Or maybe not.
Anyway we've added a special --sequental-init
flag to queue up these calls.
Automatically finds packages as long as workspaces are configured as-per the workspace-feature of one of the support package managers.
I'm aware Lerna is the best-known tool right now, but in future it seems clear it will be replaced by functionality in Yarn and NPM directly. If you use Yarn workspaces today (January 2019), then publishing is the only remaining feature Lerna is really required for (though it'd be lovely if Yarn added parallel script execution). Thus using multi-semantic-release means you can probably remove Lerna entirely from your project.
Other packages that enable semantic-release for monorepos work by iterating into each package and running the semantic-release
command. This is conceptually simple but unfortunately not viable because:
A key requirement is handling local dep version numbers elegantly. multi-semantic-release does the following:
patch
bump on that package toopackage.json
file (overwriting any existing value)The above means that, possibly, if someone upgrades dependencies and pulls down a package from NPM during the multirelease (before all its deps have been published at their next versions), then their npm install
will fail (it will work if they try again in a few minutes). On balance I thought it was more important to be atomically correct (this situation should be fairly rare assuming projects commit their lockfiles).
This is the jankiest part of multi-semantic-release and most likely part to break relies. I expect this to cause maintenance issues down the line. In an ideal world semantic-release will bake-in support for monorepos (making this package unnecessary).
The way I ended up integrating is to create a custom "inline plugin" for semantic-release, and passing that in to semanticRelease()
as the only plugin. This then calls any other configured plugins to retrieve and potentially modify the response.
The plugin starts all release at once, then pauses them (using Promises) at various points to allow other packages in the multirelease to catch up. This is mainly needed so the version number of all packages can be established before any package is released. This allows us to do a patch
bump on releases whose local deps have bumped, and to accurately write in the version of local deps in each package.json
The inline plugin does the following:
context.commits
with a list of commits filtered to the folder onlyplugins.analyzeCommits()
to get the next release type (e.g. from @semantic-release/commit-analyzer)patch
if that's trueplugins.generateNotes()
to get the notes (e.g. from @semantic-release/release-notes-generator)dependencies
, devDependencies
, peerDependencies
in package.json
git push
asynchronously, multiple releases at once fail because Git refs aren't locked — semantic-release should use execa.sync()
so Git operations are atomic)The integration with semantic release is pretty janky — this is a quick summary of the reasons this package will be hard to maintain:
context.commits
object before it was used by @semantic-release/commit-analyzer
(so it only lists commits for the corresponding directory).
context.commits
was very difficult! I did it eventually creating an inline plugin and passing it into semanticRelease()
via options.plugins
plugins.analyzeCommits()
with an overridden context.commits
— see createInlinePluginCreator.jsdependencies
in package.json points to an internal package) this step also does a patch
bump if any of them did a bump..releaserc
and then per-directory overrides for individual packages).
git tag
is done asynchronously
execa()
in semantic release should be replaced with execa.sync()
to ensure Git's internal state is atomic. Synchronizer
is the neat part. It is critical to make the tag and commit publishing phases strictly sequential. Event emitter allows:Releases always use a tagFormat
of my-pkg-1@1.0.1
for Git tags, and always overrides any gitTag
set in semantic-release configuration.
I can personally see the potential for this option in coordinating a semantic-release (e.g. so two packages with the same tag always bump and release simultaneously). Unfortunately with the points of integration available in semantic-release, it was effectively impossible when releasing to stop a second package creating a duplicate tag (causing an error).
To make the tagFormat
option work as intended the following would need to happen:
v1.0.0
will have the same effect as Lerna's fixed mode (all changed monorepo packages released at same time)Feel free to open any kind of issues: bugs, feature requests or questions. You're always welcome to suggest a PR. Just fork this repo, write some code, add a pinch of tests and push your changes. Any feedback is appreciated.