nx format fails with Error: spawnSync /bin/sh E2BIG

stijnvn commented 1 year ago

Current Behavior

Running nx format on GitHub Actions fails with:

<ref *1> Error: spawnSync /bin/sh E2BIG
    at Object.spawnSync (node:internal/child_process:1110:20)
    at spawnSync (node:child_process:8[7](https://github.com/flirits/flux-web/actions/runs/5276442833/jobs/9543158719#step:5:8)1:24)
    at execSync (node:child_process:952:15)
    at write (/home/runner/work/flux-web/flux-web/node_modules/nx/src/command-line/format/format.js:110:3[8](https://github.com/flirits/flux-web/actions/runs/5276442833/jobs/9543158719#step:5:9))
    at /home/runner/work/flux-web/flux-web/node_modules/nx/src/command-line/format/format.js:32:46
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/runner/work/flux-web/flux-web/node_modules/nx/src/command-line/format/format.js:32:27)
    at Generator.next (<anonymous>)
    at fulfilled (/home/runner/work/flux-web/flux-web/node_modules/tslib/tslib.js:[16](https://github.com/flirits/flux-web/actions/runs/5276442833/jobs/9543158719#step:5:17)4:62) {
  errno: -7,
  code: 'E2BIG',
  syscall: 'spawnSync /bin/sh',
  path: '/bin/sh',

Running nx format:check fails with a less clear error message:

Error: Process completed with exit code 1.

It seems that there are too many files to be checked.

Expected Behavior

nx format should work with many files.

Steps to Reproduce

run nx format on a branch with many changed files

Nx Report

Node   : 18.16.0
   OS     : linux x64
   npm    : 9.5.1
   Hasher : Native

   nx                 : 16.3.2
   @nx/js             : 16.3.2
   @nx/jest           : 16.3.2
   @nx/linter         : 16.3.2
   @nx/workspace      : 16.3.2
   @nx/angular        : 16.3.2
   @nx/cypress        : 16.3.2
   @nx/devkit         : 16.3.2
   @nx/eslint-plugin  : 16.3.2
   @nrwl/tao          : 16.3.2
   @nx/webpack        : 16.3.2
   nx-cloud           : 16.0.5
   typescript         : 5.0.4
   ---------------------------------------
   Community plugins:
   @ngneat/spectator        : 14.0.0
   @ngneat/transloco        : 4.2.7
   @ngrx/component          : 16.0.0
   @ngrx/component-store    : 16.0.0
   @ngrx/effects            : 16.0.0
   @ngrx/entity             : 16.0.0
   @ngrx/router-store       : 16.0.0
   @ngrx/store              : 16.0.0
   @ngrx/store-devtools     : 16.0.0
   @testing-library/angular : 14.1.0
   ng-mocks                 : 14.10.1
   ngx-toastr               : 17.0.2

Operating System

[ ] macOS
[X] Linux
[ ] Windows
[ ] Other (Please specify)

FrozenPandaz commented 1 year ago

Which particular distro of Linux are you on?

What is the result of:

getconf ARG_MAX

I'm on Pop!_OS and it's 2097152 for me.

That is the value we use to chunk the changed files into sets which fit into the command.

I could see it failing is if there is a file path that is so long that it alone, does not fit in a command but I think it would have to be really long which your paths do not seem to be that long. So maybe the command length limit is low?

stijnvn commented 1 year ago

I saw this issue on GitHub actions running ubuntu_latest. I could also reproduce it on a test server running Rocky Linux 9. On that server I also get 2097152 as result of getconf ARG_MAX.

However I can run nx format on my iMac without error. The result of getconf ARG_MAX is 1048576.

stijnvn commented 1 year ago

I did a more debugging. If I monkey patch getUnixTerminalSize() in node_modules/nx/src/utils/chunkify.js the command runs properly with a size smaller or equal than 131541.

rek commented 10 months ago

For me this works with node 16, but anything higher than that I get this issue.

de-don commented 7 months ago

The same issue on Gitlab CI.

Command nx format:check --base=XXX sometimes fails without any output (just code 1), but nx format:check --all always work (but slower).


getconf ARG_MAX
> 2097152

gautier-lefebvre commented 3 months ago

After investigating a bit, I found that while ARG_MAX is indeed 2097152 on my laptop, it actually only works for chunks < than about 131k. This is because while 2097152 is the maximum length of an entire command, the length of a single argument is hardcoded in the kernel (in our case it's 131072).

In the nx/src/command-line/format.js file, the command is executed via child_process.execSync, which is turns will exec /bin/sh -c "${THE_COMMAND}" so the command is considered as a single argument instead of having one argument per file.

I didn't find a way to get this value programmatically, so there are 2 ways to fix this:

Use child_process.spawnSync instead of execSync, it will use one arg per file instead of wrapping the command in a single arg.
- I tried by monkeypatching and it seems to work, I'm just not sure of the impact here.
Or don't try to get ARG_MAX and just hardcode it in chunkify.js (which already falls back to 100k if getconf ARG_MAX fails).

MaxKless commented 2 months ago

Fixed with https://github.com/nrwl/nx/pull/21074

github-actions[bot] commented 1 month ago

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

nrwl / nx