[rush] Publish fails with ENOBUFS when there are too many versions in the registry

[patryk-bartoszek]

Summary

When there are too many versions in the registry, the rush publish --apply command fails. With the:

Failed to get NPM information about @scope-name/package-name.
ERROR: spawnSync /bin/sh ENOBUFS

The failure happens when checking if the new version already exists in the registry. The PublishAction._packageExists method gets the full version lists, and then it looks for a specific version.

Repro steps

1. Have a private registry with an insane number of versions for a package.
2. Attempt to publish a new version of the package with the rush publish command

Expected result: The package version is bumped and published.

Actual result:

Failed to get NPM information about @scope-name/package-name.
ERROR: spawnSync /bin/sh ENOBUFS

Details

Retrieving the version list is where the error happens. https://github.com/microsoft/rushstack/blob/99bec06b4dc50c30f1255600626eedb5b1fa5b9f/apps/rush-lib/src/utilities/Npm.ts#L8

This method runs npm CLI to get the full list of package versions. Unfourunetly, the command output doesn't fit in the default buffer size. (In the project I'm working on, we got to the point of having over 1MB of versions returned in JSON format).

I tried increasing the buffer size by setting the maxBuffer in the: https://github.com/microsoft/rushstack/blob/99bec06b4dc50c30f1255600626eedb5b1fa5b9f/apps/rush-lib/src/utilities/Utilities.ts#L771

This appears to solve the problem.

However, I wonder if getting the package list is the best approach here.

I think that running the: npm view @scope-name/package-name@<version-number> version and then checking if the command output matches should work fine to determine if the version exists. I don't know if there is any nuance to that.

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
@microsoft/rush globally installed version?	5.51.1
rushVersion from rush.json?	5.51.1
useWorkspaces from rush.json?	false
Operating system?	Windows, Linux
Would you consider contributing a PR?	Yes
Node.js version (node -v)?	14.17.4

[iclanton]

I think that running the: npm view @scope-name/package-name@<version-number> version and then checking if the command output matches should work fine to determine if the version exists.

Yeah that's probably a better approach. IIRC all we're checking for is if that particular version is already published.

If you'd be willing to contribute a fix, we'd certainly appreciate it.

[renoirb]

I could reproduce this one by "luck". I had a CI job build 50*5 versions because of a CI Je nkinsfile mistake, as I said "luck".

while debugging, it looks like there's a place where we have an AsyncIterator.

[renoirb]

I was wondering.

If we already git tag versions. Instead of having to make good number of HTTP requests to an npm HTTP host, couldn't we use git tags instead. They're local.

[lianghx-319]

ERROR: spawnSync /bin/sh ENOBUFS Error: Command failed with exit code 1: rush version --ignore-git-hooks --bump -b release/rush-version

I got a same ENOBUFS issue while executing rush version

It happen on my CI machine, even thought maxBuffer set to 200Mb

[lianghx-319]

ERROR: spawnSync /bin/sh ENOBUFS Error: Command failed with exit code 1: rush version --ignore-git-hooks --bump -b release/rush-version

I got a same ENOBUFS issue while executing rush version

It happen on my CI machine, even thought maxBuffer set to 200Mb

CI cache download large of files into woking dir, which become untracked files that make rush version out of buffer while calling getUntrackChanges. Just add cache dir into .gitignore can solve my issue

[renoirb]

Oh, makes sense. Error says about filesystem, not HTTP.

You have an idea what would be the path pattern?

Maybe, assuming max depth 2, in say */*/.cache/?