[rush] Merge conflicts with "rush publish" in parallel CI runs

[CarltonHowell]

Summary

We are running into an issue where a merge conflict arises for "CHANGELOGS.{json,md}" when multiple CI pipelines running "rush publish" kick off at one time. Wondering if there is a recommended approach for dealing with parallel or concurrent "rush publish" CI pipelines?

Details

• Every commit to the main branch triggers a CI run
• Error only arises on parallel CI pipeline runs

Repro steps

1. Person A makes Commit A which changes one application and produces a changelog:

   common/changes/application-A.json

2. Commit A is merged into the main branch
3. The CI pipeline (PIPELINE A) is kicked off which runs rush build, tests and finally rush publish on "application-A"
4. An autocommit (COMMIT C) is made by rush publish into the main branch

but before the PIPELINE A gets to the autocommit stage the following happens:

1. Person B makes Commit B which changes one application and produces a changelog:

   common/changes/application-B.json

2. Commit B is merged into the main branch
3. The main branch now contains the following changelogs:

   common/changes/application-A.json
   common/changes/application-B.json

4. The CI pipeline (PIPELINE B) is kicked off which runs rush build, tests and finally rush publish on both "application-A" and "application-B"
5. While PIPELINE B is running PIPELINE A completes and COMMIT C is added to the main branch
6. PIPELINE B attempts to run rush publish but creates a merge conflict due to the CHANGELOG.{json,md} files having already been changed and committed by COMMIT C

Screenshot

^^ note in this scenario the conflict is just the date. However if COMMIT A and COMMIT B both change the same application then this conflict will also include changelog comments, author, etc

Details

The problem is removed if we disable parallel CI runs and force "synchronous" CI runs. However as monorepo activity increases this causes longer and longer build/deploy timelines.

Standard questions

Question	Answer
@microsoft/rush globally installed version?	5.55.1
rushVersion from rush.json?	5.55.1
useWorkspaces from rush.json?	true
Operating system?	Mac/Linux
Would you consider contributing a PR?	Yes
Node.js version (node -v)?	16.13.12

[dmichon-msft]

Since the version bump part of publishing commits back to main, it really shouldn't be run in parallel with itself. So for starters I'd recommend configuring your publish pipeline to run in batched CI mode.

The way my team addresses the scaling issue in house is that we do the version bump, the build, and the actual publishing to the npm feed as three separate jobs; only the first one commits back to main and if there is a conflict it aborts the entire pipeline.

Version bump job

1. Checkout with depth 1
2. Install Rush (if needed); note, this is not rush install, just installing Rush itself to be able to run step 4
3. Pull main
4. rush version --bump
5. Commit changes
6. Push

Build job

1. Checkout with depth 1 the result of the version bump job
2. rush install
3. rush build
4. For each package, pnpm pack
5. Upload pipeline artifact oontaining all tarballs
6. Upload pipeline artifact containing the map of package versions to the current commit (for use when creating Git tags)

Publish job

1. Download pipeline artifacts (no checkout needed)
2. Run npm publish for each tarball
3. Make an API call to the REST API of the chosen Git provider to batch set tags for all published packages; if any tag already exists it will be skipped

Making changes to Rush to better support this workflow is one of the goals for Rush 6 with respect to reworking publishing.