[rush] Design: Publishing 2.0

[elliot-nelson]

Introduction

This ticket is a design tracking ticket for the next iteration of Rush publishing. There are probably many other tickets to link to and/or consolidate into this one, which we can do as needed.

Requirements

Although change tracking, versioning, and publishing are distinct features, they combine to form the "publishing story" of a monorepo, and we will consider them all as a whole.

These requirements are intentionally as sparse as possible, and could be made even sparser: we want to include only things we think are bare minimum features of the publishing story for a monorepo, which might mean things that are a staple of Rush today are not really "requirements", they are just existing design decisions.

• Change tracking

  • Changes in the monorepo should be tracked with "change files" that are checked in along with the corresponding changes.
  • Projects should be able to opt in or out of change tracking.

• Versioning

  • There should be a mechanism to "bump" projects, which involves: consuming change files, calculating new versions, validating the new version set, and updating project versions and changelogs.
  • The versioning process should allow standing configuration for projects, that controls how project versions interact with each other (such as: version groups, project rollups, linked projects, fixed projects, and etc.)
  • At the time of versioning, user should be able to restrict versioning to a specific "version group" (this is not standing configuration, but a choice made at version time).
  • At the time of versioning, user should be able to produce prerelease versions instead of regular versions.

• Publishing

  • There should be a mechanism to publish all (or a single group) of projects, uploading them to the registry.
  • Projects should be able to opt in or out of publishing (separately from change tracking).
  • At the time of publishing, user should be able to restrict publishing to a specific "version group".
  • Publishing should work for projects even if change tracking and versioning features are not used.

Goals

Goals for Publishing 2.0 go above and beyond the requirements, and can refer to specific features we think are missing from Rush today or changes we'd like to make. This wishlist intentionally does not include design or implementation suggestions.

• Goal: Cleanly and totally separate versioning and publishing, so that there is no overlap between these two functions. This would make these features easier to maintain and document, and easier for users to understand.
• Goal: Support some new versioning features, similar to changesets, which allow project versions to be linked together. Unlike Rush's existing version policies, these rules can be overlapping and link projects together in a many-to-many manner.
• Goal: The versioning task should be able to reason about a set of calculated versions, detect if it is impossible to version all of the changed packages in a way that a user would expect, and produce an error instead of continuing. (This may include a distinction between a set of versions that are merely "unsafe", and a set of versions that is "impossible".)
• Goal: Include specific documentation and advice about how to use the Rush publishing tools. These tools don't live in a vacuum: they are run by specific CI/CD pipelines, in specific circumstances, on specific branches, and there are other steps in those pipelines that may happen before and after running the Rush tools. Ideally, Rush doesn't just provide the tools, but specific advice on how to use them to achieve the goals of the publisher.
• Goal: Provide a clear story for how Rush supports publishing without bot-driven commits to trunk. (In other words: ensure that Rush's publishing story works even if human beings must approve every change to trunk.) There are pros and cons to bot-driven and human-approved change file consumption, and Rush documentation should be able to outline them.

Use cases

Use cases are specific scenarios we want to support with the Rush publishing story.

A simple tool

As a developer in the monorepo, I've created a simple tool or library, and I want it to be available for users outside of the monorepo, by publishing it to my company's private NPM registry. By adding just a couple lines of configuration, I would like to get automatic version bumping, changelog generation, and publishing "for free". I have no opinions on when my library is published, and am happy to release along with other changed libraries (hourly, nightly, weekly, etc.).

A collection of libraries

As a developer in the monorepo, I started with a simple tool or library, but it's grown into a related collection of several libraries. I still don't want to worry about publishing, and would like everything to be taken care of for me automatically, but now I have more opinions about the way the versions should be generated. I would like:

• To be able to force my libraries to share the same version when they are updated, but keep dedicated changelogs.
• Optionally, I'd like to "consolidate" (or "delegate") all changes in my linked projects to one main project.

A dedicated API team

As a team that owns a complex library (or network of libraries), I would like to publish our API to the private NPM registry, but it has to be done on our terms -- our quality bar is much higher than just "has merged to main". This "library group" (our API) belongs to us, and we would like:

• No one should be able to publish release versions of our API without my team's explicit sign off.
• We need to be able to publish prerelease versions of our API on demand, for human testing in other repos.
• We need to be able to publish release versions of our API, from an explicit commit (one we human tested), and then update main branch with the consumed change files and updated changelogs.
• We need to be able to publish hotfix versions of our API, out of band from main, with cherry-picked bug fixes, and release updated libraries.
• The API should have the option, but not the requirement, for linked versions. In fact, we may need multiple groups of linked libraries within our API.
• Although we like the option of automatic version bumps and changelog generation, we will sometimes need to explicitly set our version numbers.

GitHub Actions

As a developer at a company, I've ended up building out several custom GitHub Actions, and they make use of libraries inside the monorepo. It would be useful if I could "publish all" from my monorepo, and it would publish my libraries to NPM and my GitHub Actions to target repositories in the company org.

• There's a relatively straightforward way to bundle, push, and tag branches "the GitHub way".
• It would be nice if this was as simple as specifying publishMethod: "pnpm" or "publishMethod": "github-action", and all the versioning and publishing and pushing happens all at once without any extra effort. Note that this GH Action publish requires some configuration -- you need to know the target repo name, at least -- so we'd need to decide how such configuration is provided.

Docusaurus / GitHub Pages / Static Websites

As a developer at a company, I've built out some static websites, and similar to GH Actions above, I'd like to "publish" them automatically and have Rush manage it for me. This involves bundling and pushing code to branches in other repositories.

"Preview and Publish"

As an SDK maintainer in the monorepo, I would like support from Rush for a standardized "preview and publish" workflow... from my perspective:

• I run a workflow "Prepare Release":
  • Normal rush version / rush pack behavior
  • Publish all packages with a consistent prerelease tag (-rc.169820394)
  • Pack all packages with final versions, but don't publish them. (Save tarballs in a big 169820394.zip).
  • Open a pull request from a temp branch to main, with all change log and version edits.
• My team will test the prerelease libraries to see if they meet quality gates.
• If they do, I will merge the opened pull request.
  • Merging the pull request into main will automatically trigger the Publish Release workflow.
  • All tarballs in 169820394.zip will be pushed to NPM.

Rush doesn't have to handle all of the above -- workflow definitions, triggering jobs, inventing prerelease tag, that can happen outside Rush. But it would need to: allow packing with and without prerelease tags, allow publishing a pile of tarballs.

The design

Here we will lay out the basics of the new design, detail exactly how each proposed Rush command will work, what it will and won't do, including implementation notes if appropriate.

Configuration: The "Publishing Plan"

The publishingPlan ("plan") is the spiritual successor to versionPolicy.

Highlights:

• Publishing plans live in a new, dedicated config file in the common file.
• Rush comes pre-baked with one plan, titled "default", but the user can create as many as they want, named whatever they want.
• A publishing plan includes the following:
  • A selector formula, drawing a line around exactly what packages can be published by the plan. It is impossible to version or publish any package outside this selection while using this plan.
    • An example of a selector formula might be --to @microsoft/rush. Or --only tag:global-sdk.
    • This would be a particularly good time to finally support selector queries as outlined in #3327 (because this is a JSON file, we could tackle the JSON format first and leave the command-line syntax until later, if desired).
  • A set of versioning rules, modifications to the way Rush normally versions projects.
    • Example: projects [A, B, C] must bump to the same version
    • Example: projects [A, B, C] roll up to project A (note that it's not a requirement that every package in the plan be included!)
  • Additional flags and settings related to publishing
  • Example: there might be a "safety check" feature that can be toggled on or off, aborting the publish if it would release packages that depend on packages with unreleased changes (essentially, a combination of code that has never passed CI).
  • Example: version requirement (spiritual successor to lockedMajor: a more robust version regex or multi-field configuration object that can require a particular major, minor, patch, semver string, or require presence of a particular prerelease tag in order to complete publishing).

Key considerations:

• The rush.json file does not link projects directly to publishing plans; instead, a plan directly lists its projects, or delegates this job to other selectors like tag.
• This scheme allows for declarative configuration of things like "a library tagged sdk will never be published by the default plan" (safety: teams can ask that their own stuff will never accidentally be released).
• Even for such a team, they might acknowledge the usefulness of being included in a repo-wide "canary" plan, which publishes all projects in the repo (including sdk), but enforces an -alpha.* prerelease tag -- this is possible because projects can be published by more than own publishing plan, and have different versioning rules in each plan.
• The sdk team might have their own plan, sdk, which includes only their own tag, and is run by their own CI job.

Change file generation ("rush change")

• The "rush change" command, like today, prompts the user for change information for each changed project with change tracking enabled.
• Comparing Rush and Changesets: Rush's model, where the tool informs the user what projects require change messages, is different than Changeset's, where the user tells the tool what projects they'd like to enter messages for. Because Rush is designed for large, multi-tenant monorepos, we expect that users will often touch projects they have no context for (they might not even have heard of the team that owns the project, let alone the project itself), so keeping this model is important.
• I propose that rush change no longer respects version policy / publishing plan when prompting the user: it should treat all projects the same while gathering information. Decisions about whether or not to discard minor/patch/major information, or whether to move the gathered messages to a "main" project, are decisions made in the context of a publishing plan, and one project can belong to multiple publishing plans. So, we should avoid a design that requires "rush change" to know the context in which the project will be published.

Versioning ("rush version")

• TBD

Publishing ("rush publish")

• TBD

Next steps

• https://github.com/microsoft/rushstack/issues/3957
  • This is Part 1 to support Asynchronous Publishing workflows
• https://github.com/microsoft/rushstack/issues/4000
  • This is Part 2 to support Asynchronous Publishing workflows
• https://github.com/microsoft/rushstack/issues/3327
  • Need to nail down a robust and flexible "package selection" schema before we can use it to build Publishing Groups (the successor to Version Policies)
• https://github.com/microsoft/rushstack/issues/4045
  • These publishing scenarios can be designed and implemented side-by-side with the existing versioning (and won't be used until we add this functionality into the command)

[Faithfinder]

Ooh. Should "Deployment" be included as well? Isn't "Publishing", in a sense, just a variant of a larger "Deployment" scenario? Apps are deployed, libraries are published.

[elliot-nelson]

Ooh. Should "Deployment" be included as well? Isn't "Publishing", in a sense, just a variant of a larger "Deployment" scenario? Apps are deployed, libraries are published.

@Faithfinder Great question... I guess you could argue my "github action" and "docusaurus" use cases are really cases of Deployment, not Publishing. But at the extreme end of Deployment, where you are running terraform or aws cdk, baking and pushing docker images etc., the promise of rush-managed deployment without opportunity for "customization" (of the CI pipeline, of your environmental controls, etc.) gets very dicey.

Maybe this is an argument that instead, we should allow Rush to do all the versioning for you, and then produce what I've been tentatively calling a "publishing plan" (changesets calls it a "release plan"). You could do all the versioning and bumping and commit the results, and then the output (the "release plan") could be used by custom processes. "If we bumped @example/complicated-service, trigger the dedicated CI pipeline for that service." (This custom process may need to run on a totally separate job, for example a Windows or Mac self-hosted runner, so we can't assume Rush can even run a script inline... you need a way for your CI pipeline to take control back and make a decision based on the output.)

[Faithfinder]

Sounds about right. Versioning is a common concern, but different ways to pack up a code and send it where it needs to be is really a by-package thing?

Sounds like rush publish and rush deploy should be offloaded to plugins instead? Maybe even heft plugins, rather than rush?

[dmichon-msft]

One of the bigger gaps in current publishing is that fundamentally, pack and publish are and should be separate operations. This is because pack is a local file system operation, and publish is purely a network operation. For CI reliability it is generally beneficial to perform the two as separate jobs, so that the publish task can be retried in the event of transient network issues, without needing to redo all of the work for pack (which usually involves, building, running test suites and validation, etc.).

Creating Git release tags associated with packages is also its own thing, since the optimal way to do so varies depending on your environment (e.g. for Azure DevOps there is a single REST API call to create a full set of Git tags for every package in the monorepo, whereas using local Git commands is significantly slower and more complicated).

Regarding automated vs. human-approved commits to trunk, I've been considering the model that the packed package.json is at least partially a build artifact; the actual version information is tracked by a separate file, and changes to that file are the trigger for the publishing CI pipeline.

The rushstack repository has demonstrated the need to be able to mutate package.json at pack time for various reasons: adding/removing dependencies, trimming built-time only sections in package.json, etc. Currently we can do this with the npm prepack script, but we might want to integrate some Rush-level configurability.

[elliot-nelson]

One of the bigger gaps in current publishing is that fundamentally, pack and publish are and should be separate operations... Creating Git release tags associated with packages is also its own thing...

:+1: Makes sense! Where possible, sounds like we want different pieces of the process to be retryable: updating version, push tags, pack the tarballs, publish the tarballs.

Regarding automated vs. human-approved commits to trunk, I've been considering the model that the packed package.json is at least partially a build artifact; the actual version information is tracked by a separate file, and changes to that file are the trigger for the publishing CI pipeline.

What do you think about my idea of a "release plan" file of sorts?

One of the reasons that rush publish today encroaches on the territory of rush version, is that certain actions require knowledge of what will be consumed -- for example if you want to publish a set of prerelease packages, you use rush publish because it can take parameters that inline consumes versions and uses that to know what packages to publish.

If rush version produced a "release plan", a JSON file containing the set of all packages that were updated and what versions to release, it could be an input to all the later steps (especially if the later steps are 3-4 different rush actions).

The rushstack repository has demonstrated the need to be able to mutate package.json at pack time for various reasons: adding/removing dependencies, trimming built-time only sections in package.json, etc. Currently we can do this with the npm prepack script, but we might want to integrate some Rush-level configurability.

Could you elaborate on the "mutating package.json at pack time" -- is there an example live today in the repo? I'm not sure what an example would be.

[StarFishing]

Maybe can add a beta or a specified non-stable version to support incremental version number example: I need record beta in my repo run publish first

rush publish --type beta 
package1@1.1.1-beta.0

run publish again

rush publish --type beta 
package1@1.1.1-beta.1

Correspondingly, this version needs to be taged and pushed remotely.

[elliot-nelson]

@StarFishing Yes, good call on prerelease, although I'm not sure on the incremental version number requirement.

I think what you want (and what I would want) is that package1's package.json contains 1.1.1, and does not change, but that each rush publish --beta produces a new version.

That is pretty easy, if it's OK to use a datetime or similar number that can be generated without context:

$ rush publish --prerelease beta.$(date +%y%m%d%H%M%S)

-> Publishing package1@2.0.0-beta.230223080931
-> Publishing package2@1.3.1-beta.230223080931
-> Publishing package3@1.4.0-beta.230223080931

This approach is nice for many related packages, as they all get a similar-looking prerelease suffix. The disadvantage is that the version number is a bit long and cumbersome. You can get by with smaller numbers if you have some context available to your pipeline; in GitHub Actions for example, you could substitute this:

$ rush publish --prerelease beta.${{ github.run_number }}${{ github.run_attempt }}

-> Publishing package1@2.0.0-beta.2311
-> Publishing package2@1.3.1-beta.2311
-> Publishing package3@1.4.0-beta.2311

What I think we can't do is have Rush itself generate per-package incremental prerelease suffixes, because to do so we would need a way for the caller to inject their own custom "version generating" script (which I think would be more confusing than helpful).

[StarFishing]

The existing prerelease-name of rush publish can be satisfied, but I really need an ability to retain the incremental version number here. Like lerna, since there will be a lot of users for the beta, I need the stable version number to keep track of the tests. Acquisition here can assume a fixed representation, such as --should-increase. Then I can calculate the version that needs to be increased from my current specified prereleae-type

example: current package is package1@0.1.1

add change type patch

$ rush publish --prerelease-type beta  --should-increase
-> publishing package1@0.1.2-beta.0

add change type minor

$ rush publish --prerelease-type beta  --should-increase
-> publishing package1@0.2.0-beta.0

if add change have sample type ,like patch

$ rush publish --prerelease-type beta  --should-increase
-> publishing package1@0.1.2-beta.1

for more case

    const case1 = genneratePrereleaseVersion({
      type: PublishType.BETA,
      currentVersion: '1.1.2-beta.0',
      changeType: ChangeType.patch,
      newVersion: '1.1.2'
    });
    expect(case1).toEqual('1.1.2-beta.1');

    const case2 = genneratePrereleaseVersion({
      type: PublishType.BETA,
      currentVersion: '1.1.2-beta.3',
      changeType: ChangeType.minor,
      newVersion: '1.2.0'
    });
    expect(case2).toEqual('1.2.0-beta.0');

    const case3 = genneratePrereleaseVersion({
      type: PublishType.BETA,
      currentVersion: '1.2.0-beta.3',
      changeType: ChangeType.minor,
      newVersion: '1.2.0'
    });
    expect(case3).toEqual('1.2.0-beta.4');

    const case4 = genneratePrereleaseVersion({
      type: PublishType.BETA,
      currentVersion: '1.2.0-beta.xx',
      changeType: ChangeType.minor,
      newVersion: '1.2.0'
    });
    expect(case4).toEqual('1.2.0-beta.0');

[StarFishing]

In addition, whether --include-all is specified or not, the default check whether the version number already exists in registry. This will be very useful.

[elliot-nelson]

@StarFishing I see what you mean, but one of our goals for the new publishing system is to increase the separation between the different actions:

• rush version consumes change files and generates new versions
• rush pack turns those projects into packed tarballs
• rush publish takes a pile of tarballs and pushes them to NPM

If you are checking in the current version "1.2.0-beta.3" to trunk, then rush version will work just fine for your use case I think (although, I think I am generally opposed to checking in a prerelease tag).

If you aren't checking in the current version, but expect the committed version 1.2.0 to be able to produce 1.2.1-beta.1 and 1.2.1-beta.2 (a sequential ordering of prerelease tags), then you're right, rush version would need a special flag that means "please ask the NPM registry for a list of all the published versions, and consult that list while generating a new tag".

(That could be a useful feature to explore.)

[StarFishing]

Recently, I encountered a case in which I can get the list to be released before release, but some of this releases may be successful, and some of them may not be released due to the existence of relevant versions on npm. I can't know the actual release of the package. I can only look at it from the log. If there is a way to get this result, I think it's very useful.

[StarFishing]

Regarding the rush version, it does not seem to mention whether the version field of package.json can only be changed to those items that declare shouldPulish:true, rather than together with those that do not need to be published.

[elliot-nelson]

Yes, "rush version" should be able to version projects that aren't intended for publishing - for example a website you deploy via docusaurus and just want to maintain a changelog for.

In that case (let's say it's "trackChanges"), there'd be no context to look up from npm for that project, so decisions have to be made without any outside context.

[elliot-nelson]

One of the "Goals" I reference in OP is the ability to publish without bot-driven commits to trunk.

I think one way to capture this is to lay out stories for "synchronous publishing" and "asynchronous publishing". Synchronous publishing is what the Rush CI workflows do today; Asynchronous publishing is more like Changeset's model (you use PRs to represent a process that can be approved or delayed by humans, and "merging" the PR represents approval of the publishing event).

The key to understanding asynchronous publishing is that upon the trigger (P1 merging into trunk and becoming M1), you publish tarballs built from P1. This ensures that trunk and the published binaries agree on what versions are published, but the published binaries don't contain unapproved commits (like C3) that don't match the changefiles that were consumed by P1.

[smolinari]

I'd like to add that I agree with the beginning concerns/ discussion about the differences between publishing and deploying (which aren't much) and would venture to further say, that is where the issue of "muddiness" comes from in the current system and why even this thread/ issue seems "difficult" too. I believe Rush should only concern itself with versioning and changeset management. And thus, the publishing process should be considered completely separate and not even a part of this change.

Even though I'm not the most experienced with software development directly i.e. I'm not a professional developer, I do have a lot of experience with process automation and it is very clear to me the products of Rush's version and changeset management should be something like meta-information ( the process product) for the publishing and deployment processes. Getting to these "results" of the two processes (version management and changeset management) should be flexible enough that it can feed any pipelines for publishing and/ or deployment.

In the end, publishing and/ or deploying anything is the process of creating artifacts from anything that may have changed in the monorepo. So, going out on a limb now, I believe this shouldn't be a "Publishing 2.0" update, but rather a "Version and Change Management 2.0 and Publishing moved to another place" update. :blush:

In other words, stick to only the Change Management part of the process and it will simplify the objectives of this "new way" of doing the work to generate the meta-information for the next process steps - deployment/ publishing. Move the publishing work to the deployment side of Rushstack and make any extra publishing tools/ plugins available there.

I know that if you go with this change of direction, both processes will become much more flexible and less convoluted for the user overall, because it will allow them to have an easier understanding of the processes with the side effect of allowing more control over them too. And more importantly, it will be easier to develop these new Versioning/Change Management and Publishing tools too. :grin:

Hope I could help. :grin:

Scott

[StefanoPastore]

Hi @elliot-nelson adding here my ticket about change and publish commands #4223

I didn't know this task but it seems exactly what i'm asking for, tracking changes even if the project isn't a NPM package.

For projects this kind of project it would be helpful have something like publish command that instead of publish package on NPM registry run a command on the project that will build the docker image or what the project need to be deployed. So it would be great track changes even if shouldPublish is false.

Another interesting feature, as said by @smolinari, could be have a way to deploy changed projects instead of only publish them. Maybe with a command that works like publish but instead of publish project to NPM registry call a command into the project.

In this way the deploy process is in the developer hands but the process is in charge of rush.

For example, imagine have projects changed that must be published on docker registry and then in kubernetes, in CI pipeline call this new command that after pack the project (like deploy command already do) could call a command configured for the project that will build the docker image with the deploy folder of the project, publish it on docker registry and then apply new image on kubernetes.

It could make sense?

[jasongornall-dd]

I'm bumping this as I'd love to see a path forward with Publishing 2.0. Any traction on pursuing this?