Reduce number of system calls made during yarn install

[trivikr]

Discussed in https://github.com/yarnpkg/berry/discussions/4023

^{Originally posted by **trivikr** January 26, 2022} The blog post on [In-depth of tnpm rapid mode](https://dev.to/atian25/in-depth-of-tnpm-rapid-mode-how-could-we-fast-10s-than-pnpm-3bpp). In this blog post the author sees yarn@3.1.1 making ~4X syscalls as compared to pnpm/tnpm as shown in the screenshot below.

Screenshot


Related twitter thread: https://twitter.com/arcanis/status/1486059029235388418

• [ ] I'd be willing to implement this feature (contributing guide)
• [x] This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

As a customer of yarn, I do not want yarn to make significantly more number of system calls as compared to other package managers.

When tested with node_modules + central store mode, yarn@3.1.1 makes ~3X syscalls as compared to pnpm. This is discussed prior in https://github.com/yarnpkg/berry/discussions/4023, and on Discord thread.

System information

```console $ hostnamectl ... Operating System: Amazon Linux 2 CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2:-:internal Kernel: Linux 5.4.172-100.336.amzn2int.x86_64 Architecture: x86-64 $ node -v v16.13.2 $ npm init vite@2.7.2 ✔ Project name: … vite-project ✔ Select a framework: › vue ✔ Select a variant: › vue-ts $ cd vite-project && git init && git add. && git commit -am "Initial commit" ```

yarn v3.1.1: cold - 221068, warm - 37702

```console $ git clean -dfx && git checkout . $ yarn set version stable ➤ YN0000: Retrieving https://repo.yarnpkg.com/3.1.1/packages/yarnpkg-cli/bin/yarn.js ➤ YN0000: Saving the new release in .yarn/releases/yarn-3.1.1.cjs ➤ YN0000: Done in 0s 353ms $ sed -i -e 's/^/compressionLevel: 0\nenableGlobalCache: true\nnodeLinker: node-modules\nnmMode: hardlinks-global\n\n/' .yarnrc.yml $ yarn --version 3.1.1 $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 35.051901 221068 13692 total strace -c -f yarn 4.33s user 3.58s system 147% cpu 5.354 total $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 5.470758 37702 3699 total strace -c -f yarn 1.26s user 0.62s system 152% cpu 1.235 total ```

pnpm v6.29.1: cold - 74734, warm - 28430

```console $ git clean -dfx && git checkout . $ echo 'node-linker=hoisted' > .npmrc $ pnpm --version 6.29.1 $ time strace -c -f pnpm install ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 4.355469 74734 8987 total strace -c -f pnpm install 1.75s user 1.03s system 117% cpu 2.368 total $ time strace -c -f pnpm install ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 1.895735 28430 1936 total strace -c -f pnpm install 1.08s user 0.44s system 129% cpu 1.173 total ```

Describe the solution you'd like

yarn to make less number of system calls while running install.

Describe the drawbacks of your solution

N/A

Describe alternatives you've considered

N/A

[trivikr]

Refs: https://github.com/yarnpkg/berry/pull/4036

[trivikr]

Verifying https://github.com/yarnpkg/berry/pull/4036, and several other perf fixes in v3.2.0-rc.14

yarn v3.2.0-rc.14: cold - 283933, warm - 37379

```console $ git clean -dfx && git checkout . $ yarn set version 3.2.0-rc.14 ➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.0-rc.14/packages/yarnpkg-cli/bin/yarn.js ➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.0-rc.14.cjs ➤ YN0000: Done in 0s 424ms $ sed -i -e 's/^/compressionLevel: 0\nenableGlobalCache: true\nnodeLinker: node-modules\nnmMode: hardlinks-global\n\n/' .yarnrc.yml $ yarn --version 3.2.0-rc.14 $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 85.094174 283933 23559 total strace -c -f yarn 7.56s user 5.74s system 149% cpu 8.912 total $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 5.029950 37379 3765 total strace -c -f yarn 1.26s user 0.61s system 153% cpu 1.218 total ```

[merceyz]

4036 shouldn't affect the node-modules linker since it doesn't use it.

[trivikr]

Verifying https://github.com/yarnpkg/berry/pull/4101, published in v3.2.0-rc.16

yarn v3.2.0-rc.16: cold - 211006, warm - 35565

```console $ git clean -dfx && git checkout . $ yarn set version 3.2.0-rc.16 ➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.0-rc.16/packages/yarnpkg-cli/bin/yarn.js ➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.0-rc.16.cjs ➤ YN0000: Done in 0s 790ms $ sed -i -e 's/^/compressionLevel: 0\nenableGlobalCache: true\nnodeLinker: node-modules\nnmMode: hardlinks-global\n\n/' .yarnrc.yml $ yarn --version 3.2.0-rc.16 $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 35.467709 211006 17482 total strace -c -f yarn 4.06s user 3.57s system 147% cpu 5.176 total $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- ... ------ ----------- ----------- --------- --------- ---------------- 100.00 7.148630 35565 4057 total strace -c -f yarn 1.11s user 0.71s system 152% cpu 1.191 total ```

[trivikr]

Numbers for v4.0.0

System Information

```console $ hostnamectl ... Operating System: Ubuntu 22.04.3 LTS ... $ node -v v20.9.0 $ npm create vite@4.4.1 ✔ Project name: … vite-project ✔ Select a framework: › Vue ✔ Select a variant: › TypeScript $ cd vite-project && git init && git add . && git commit -am "Initial commit" ```

yarn v4.0.0, cold - 271701, warm - 39303

```console $ git clean -dfx && git checkout . $ yarn set version stable $ yarn --version 4.0.0 $ echo -e 'nodeLinker: node-modules\n\nnmMode: hardlinks-global' > .yarnrc.yml $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ ... ------ ----------- ----------- --------- --------- ------------------ 100.00 80.624686 296 271701 1942 total $ time strace -c -f yarn ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ ... ------ ----------- ----------- --------- --------- ------------------ 100.00 8.671119 220 39303 335 total ```

pnpm v8.9.2, cold - 99650, warm - 63626

```console $ git clean -dfx && git checkout . $ corepack prepare pnpm@latest --activate $ pnpm --version 8.9.2 $ echo 'node-linker=hoisted' > .npmrc $ time strace -c -f pnpm install ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ ... ------ ----------- ----------- --------- --------- ------------------ 100.00 21.771456 218 99650 1592 total $ time strace -c -f pnpm install ... % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------------ ... ------ ----------- ----------- --------- --------- ------------------ 100.00 8.281274 130 63626 255 total ```