search

modrinth / code

The Modrinth monorepo containing all code which powers Modrinth!
https://modrinth.com
Other
921 stars 171 forks source link

Reconsider the decision to use yarn in favour of modern alternatives like pnpm #1604

Closed brawaru closed 1 year ago

brawaru commented 1 year ago

Is your suggested enhancement related to a problem? Please describe.

Not so long ago Modrinth repositories were switched to use legacy yarn v1 while previously using npm. I think this decision was shortsighted, misinformed and suboptimal.

One of the huge problems with legacy Yarn is that it's now frozen and not actively maintained, it has not been maintained since 2021-2022 and they recommend to migrate to Berry (v2+). And it already broke on my pull request that does not do anything unusual, which can be confirmed by simply using npm and seeing that it works with no issue, and I don't even know how to mitigate this!

Another problem I have encountered is Yarn's terrible caching strategy. Back in the days Yarn was much more space efficient than npm, but this has since changed. During active usage of npm my its cache store only accumulated 2 GB, whereas solely by working with Knossos my Yarn cache directory accumulated 4.3 GB.

The culprit? Yarn ignores the system requirements for packages and installs them even though they're clearly marked as incompatible with current OS. Therefore you end up with binaries of esbuild for all operating systems (including freaking Android!) for several versions of esbuild.

Another culprit is that Yarn doesn't cache individual files but rather whole packages. This is very space inefficient since you end up with lots of duplications of multiple versions of the package that haven't changed much between these versions.

Other issues I had with Yarn is it getting completely stuck on certain steps and not showing anything even with verbose logs, but even I find the reason, this will never get fixed anymore, as the yarn is frozen (effectively archived).

I don't think Yarn Berry is the right choice for Modrinth repos either as we rely one some of npm's quirks such as dependency hoisting. I'm saying ‘we’, but not like its our choice, it's a problem with Nuxt, really.

In my short testing Yarn Berry has failed to work with Knossos because it expects strict dependencies, another thing that Nuxt requires to be set to disabled, but I haven't found option to do this with Yarn Berry.

Describe the solution you'd like

So given we want to preserve some of the npm's quirks, but don't want to use npm due to how slow and space inefficient it is, I would propose switching to pnpm, which is gaining popularity among JavaScript developers.

pnpm is a fast and disk space-efficient package manager that serves as a drop-in replacement for npm. It centralizes library code storage, enabling shared use of dependencies across multiple projects, which leads to better performance and reduced disk space usage. pnpm installs packages into a single .pnpm-store subdirectory in your home directory and uses symlinks to link packages into the projects where they are needed. This also results in faster project initialization times compared to npm, even if npm has cached the package. flaviocopes.com, blog.logrocket.com

Note Answers by AI may not always be accurate.

Previously Geo dismissed pnpm calling it annoying, but I'm yet to see the reasons why it's annoying. Perhaps it was misconfigured and caused trouble with Nuxt, but there's a .npmrc config now which includes stuff like shamefully-hoist, which will be respected as long as pnpm is ran from the root of the project.

pnpm also has great DX shortcuts like Yarn:

My personal experience with pnpm is incredible and I can not recommend it to people enough!

Note This issue is not sponsored by pnpm 😄

Describe alternatives you've considered

Switching back to npm npm is very inefficient and slow. It received a lot of improvements and very close to legacy Yarn compared what it was, but I still think it's far far from great. However, it's still a better option than legacy Yarn, because it's continuously maintained, so if pnpm does not work, it can be an option.

Solving the problem and migrating to Yarn Berry Yarn currently promotes completely different model of Plug n' Play dependencies that aren't widely supported (and especially not supported by Nuxt), so it has to be configured to use npm linker, which is okay, but then there's the issue with peer dependencies. Yarn Berry is also pretty different from Yarn.

I honestly don't think it's worth the hassle to get it working, especially with pnpm and npm working right now without additional configuration (asides from what we have).

Additional context

I'm publishing this in Knossos repo because Omorphia and Theseus are out of our concern right now, and such switch might be disruptive to productivity there anyway, whereas the progress here for now is pretty chill allowing for experimentation like this.

Thanks for coming to my TED talk!

IMB11 commented 1 year ago

I've been using pnpm when contributing, never encountered any issues using it at all.

polarathene commented 1 year ago

Therefore you end up with binaries of esbuild for all operating systems (including freaking Android!) for several versions of esbuild.

I couldn't reproduce this? (I am aware of this screenshot)

# Starting with a reproducible environment:
$ docker run --rm -it --workdir /reproduction node:20-bookworm-slim bash

# Requirement (user should already have applied via reading yarn install docs):
$ corepack enable

# Temporary workaround until `package.json:packageManager` is implemented in `nuxi init`:
$ corepack prepare yarn@stable --activate
$ yarn -v
3.6.3

# Temporary linker workaround with ENV (`yarn config` not usable until `package.json` exists):
$ YARN_NODE_LINKER=node-modules yarn dlx nuxi init --packageManager yarn --gitInit false .

# esbuild only installed appropriate platform:
$ ls node_modules/@esbuild
linux-x64

$ ls node_modules/esbuild/node_modules/@esbuild
linux-x64

$ ls -l /root/.yarn/berry/cache/ | grep esbuild
-rw-r--r-- 1 root root    4706 Sep 22 21:54 @esbuild-kit-cjs-loader-npm-2.4.4-24b1896f49-8.zip
-rw-r--r-- 1 root root   25285 Sep 22 21:54 @esbuild-kit-core-utils-npm-3.3.2-f27be25172-8.zip
-rw-r--r-- 1 root root    6248 Sep 22 21:54 @esbuild-kit-esm-loader-npm-2.6.5-431b2c6e16-8.zip
-rw-r--r-- 1 root root 3929603 Sep 22 21:54 @esbuild-linux-x64-npm-0.18.20-de8e99b449-8.zip
-rw-r--r-- 1 root root 3967319 Sep 22 21:54 @esbuild-linux-x64-npm-0.19.3-55823289f9-8.zip
-rw-r--r-- 1 root root   33040 Sep 22 21:54 esbuild-npm-0.18.20-004a76d281-8.zip
-rw-r--r-- 1 root root   33106 Sep 22 21:54 esbuild-npm-0.19.3-32eb4a916f-8.zip

EDIT: Ah ok, it's specifically a yarn 1.x issue:

$ docker run --rm -it --workdir /reproduction node:20-bookworm-slim bash
$ yarn -v
1.22.19

$ yarn init --yes
$ yarn add esbuild
$ du -sh $(yarn cache dir)
279M    /usr/local/share/.cache/yarn/v6

$ ls -l /usr/local/share/.cache/yarn/v6
total 92
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-android-arm-0.19.3-08bd09f2ebc312422f4e94ae954821f9cf37b39e-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-android-arm64-0.19.3-91a3b1b4a68c01ffdd5d8ffffb0a83178a366ae0-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-android-x64-0.19.3-b1dffec99ed5505fc57561e8758b449dba4924fe-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-darwin-arm64-0.19.3-2e0db5ad26313c7f420f2cd76d9d263fc49cb549-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-darwin-x64-0.19.3-ebe99f35049180023bb37999bddbe306b076a484-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-freebsd-arm64-0.19.3-cf8b58ba5173440ea6124a3d0278bfe4ce181c20-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-freebsd-x64-0.19.3-3f283099810ef1b8468cd1a9400c042e3f12e2a7-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-arm-0.19.3-ff6a2f68d4fc3ab46f614bca667a1a81ed6eea26-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-arm64-0.19.3-a8b3aa69653ac504a51aa73739fb06de3a04d1ff-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-ia32-0.19.3-5813baf70e406304e8931b200e39d0293b488073-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-loong64-0.19.3-21110f29b5e31dc865c7253fde8a2003f7e8b6fd-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-mips64el-0.19.3-4530fc416651eadeb1acc27003c00eac769eb8fd-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-ppc64-0.19.3-facf910b0d397e391b37b01a1b4f6e363b04e56b-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-riscv64-0.19.3-4a67abe97a495430d5867340982f5424a64f2aac-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-s390x-0.19.3-c5fb47474b9f816d81876c119dbccadf671cc5f6-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-x64-0.19.3-f22d659969ab78dc422f1df8d9a79bc1e7b12ee3-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-netbsd-x64-0.19.3-e9b046934996991f46b8c1cadac815aa45f84fd4-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-openbsd-x64-0.19.3-b287ef4841fc1067bbbd9a60549e8f9cf1b7ee3a-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-sunos-x64-0.19.3-b2b8ba7d27907c7245f6e57dc62f3b88693f84b0-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-win32-arm64-0.19.3-1974c8c180c9add4962235662c569fcc4c8f43dd-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-win32-ia32-0.19.3-b02cc2dd8b6aed042069680f01f45fdfd3de5bc4-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-win32-x64-0.19.3-e5036be529f757e58d9a7771f2f1b14782986a74-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-esbuild-0.19.3-d9268cd23358eef9d76146f184e0c55ff8da7bb6-integrity

In my short testing Yarn Berry has failed to work with Knossos because it expects strict dependencies, another thing that Nuxt requires to be set to disabled, but I haven't found option to do this with Yarn Berry.

# Compatible `node_modules` folder approach:
yarn config set nodeLinker node-modules

# Use hardlinks with a global store like PNPM:
yarn config set nmMode hardlinks-global

I honestly don't think it's worth the hassle to get it working, especially with pnpm and npm working right now without additional configuration (asides from what we have).

PNPM isn't compatible without configuring non-defaults either. nuxi init creates a .npmrc to accommodate that, lacking similar functionality with yarn though.

This isn't a response to persuade you to switch back to Yarn btw, just providing insights to the walls you hit.

Yarn's terrible caching strategy. Back in the days Yarn was much more space efficient than npm, but this has since changed. During active usage of npm my its cache store only accumulated 2 GB, whereas solely by working with Knossos my Yarn cache directory accumulated 4.3 GB.

If compressionLevel is not 0:

If you opt-out of the global cache with enableGlobalCache: false:

With the default linker nodeLinker: pnp:

Another culprit is that Yarn doesn't cache individual files but rather whole packages. This is very space inefficient since you end up with lots of duplications of multiple versions of the package that haven't changed much between these versions.

I don't think that's different with npm or pnpm?

Yarn does have a nmMode: hardlinks-local feature or nodeLinker: pnpm (hardlinks + symlinks) which should mimic whatever optimization you're thinking of? I don't think any of the package managers have a global cache where they're using hardlinks between package versions though?

Your experience might have been with the compressionLevel setting which creates those .zip archives. It seems yarn v4 is not using compression by default (EDIT: the yarn cache dir still uses .zip pacakges regardless). Some filesystems can be more efficient with duplicate file content and optionally provide their own transparent compression.

EDIT: Looked into this and it's the global store

they recommend to migrate to Berry (v2+). And it already broke on my pull request that does not do anything unusual, which can be confirmed by simply using npm and seeing that it works with no issue, and I don't even know how to mitigate this!

It could be that I misunderstood, and all your complaints were regarding yarn v1 after this statement.

As mentioned newer versions of yarn changed the linker, but while it can usually install packages fine, it doesn't always work well at resolving them. It's usually due to projects with implicit dependencies, relying on accessing packages that aren't declared explicitly in your package.json (like Nuxt will access the Vue package) and this sort of behaviour IIRC is considered bad practice (as you noted with "npm quirks").

There's a similar compatibility issue with PNPM, but Nuxt now configures pnpm (via nuxi init at least) to fallback to hoisting packages, which yarn can do as well, point is neither of these are defaults.

You'd have been able to use nodeLinker: node-modules and everything would have gone smoothly.

In my short testing Yarn Berry has failed to work with Knossos because it expects strict dependencies, another thing that Nuxt requires to be set to disabled, but I haven't found option to do this with Yarn Berry.

If you don't want to use nodeLinker: node-modules there is a setting for this IIRC, just visit the yarn docs for the .yarnrc.yml config and search for strict / loose, that may be what you were after.

EDIT: The setting pnpMode: loose doesn't seem to help for nuxi init.

polarathene commented 1 year ago

Store comparison between PNPM and Yarn (nodeLinker: node-modules + nmMode: hardlinks-global) on a project (created with nuxi init):

PNPM

# Lookup a package from the lock file, the SHA-512 hash is a store key:
$ grep -A2 '/util-deprecate' pnpm-lock.yaml
  /util-deprecate@1.0.2:
    resolution: {integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==}
    dev: true

# The hash key was base64 encoded to reduce ASCII representation size,
# convert to hexadecimal which the store uses:
$ base64 -d <<< 'EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==' | xxd -ps -c 0
10f0f9ab5b97c85c49a42acb9c27359c79eade039ae83641a1c008888d93692080ed5089d5424331a802cc891736c5187c3d5d68afff2d3110f318886eb1ed73

# The store groups files under the first byte (two hex chars) of the hash key, with the rest of the hash as the filename,
# The lockfile is referencing the package json, so there is an additional `-index.json` suffix:
$ cat /root/.local/share/pnpm/store/v3/files/10/f0f9ab5b97c85c49a42acb9c27359c79eade039ae83641a1c008888d93692080ed5089d5424331a802cc891736c5187c3d5d68afff2d3110f318886eb1ed73-index.json

# JSON content output identifies each file from the package, each with their own hash that can find their content in the store:
{"name":"util-deprecate","version":"1.0.2","files":{"package.json":{"checkedAt":1695429623187,"integrity":"sha512-t8JHXKSuqDTJvzONFc6YAbMKMwRsaL539wb4WVOyeswdTSLpdY+tELBK8moq94CIMMhXSL+LfbzV7FiMLCkQ/g==","mode":420,"size":694},"README.md":{"checkedAt":1695429623187,"integrity":"sha512-gnBkJ+KrW26XBjy0Bo6pbA/s859Tkt4qQlbdoKTgxfAipxn+1WpMGzLfgKGNOxwU79KDGncXTpWrH7dRgq61+A==","mode":420,"size":1666},"LICENSE":{"checkedAt":1695429623187,"integrity":"sha512-hElreSqhgIRnqBHxtPWF+70iv+3/gk9uLTHUdCjnckMF7c7+wGiuFhYUXP5rWupepd6bGK2MIUW5u2aY2cVXRg==","mode":420,"size":1102},"browser.js":{"checkedAt":1695429623187,"integrity":"sha512-ZGsJKmbXf2naHjO9wg8QN4ZaAzaMfo6DmU3cfmz4rCRncoFaE9QFoRUaVSNyEtCBuWaZo4kLTfVVB9LMDJdj4Q==","mode":420,"size":1614},"node.js":{"checkedAt":1695429623187,"integrity":"sha512-6GDUjKSud32WOrZmqumfNxm98za/IYsoK3aioPAmjKa3KDv4yCVUSg7P29u/8863yYZJ2J+VZl06Xisvba7cDg==","mode":420,"size":123},"History.md":{"checkedAt":1695429623187,"integrity":"sha512-TUTHdDTG8WIwOSVQ0ChcJo/5P1FeZCZJ/fMRV5/JFBJ18zqVtBuTl9/pj5aGVw8ncL6ptnGGBfNopxG/76Ej3g==","mode":420,"size":282}}}

# A related JSON file is used by PNPM for package cache of the registry.
# PNPM does not cache the registry zip locally, so if the store lacks the extracted content, it'll fetch from the registry:
$ cat /root/.cache/pnpm/metadata/registry.npmjs.org/util-deprecate.json

{"name":"util-deprecate","dist-tags":{"latest":"1.0.2"},"versions":{"1.0.0":{"name":"util-deprecate","version":"1.0.0","dist":{"shasum":"3007af012c140eae26de05576ec22785cac3abf2","tarball":"https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.0.tgz","integrity":"sha512-sTmIWz2UtUfg8kaf6qlicnMn6ghnpMboyWJAv+kgorwAmCHY78TcfaRWfMD8OECWkqVFwyat+r1VJxA4dDfGSA==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQCri2jrXgd4KNJxbVAHav4n3MwZ3UrEqrKZ5X54Efk2/wIgI85j6D2+98v25iZu+ygxEUNvgmL3J1aehucOqaCD7hg="}]}},"1.0.1":{"name":"util-deprecate","version":"1.0.1","dist":{"shasum":"3556a3d13c4c6aa7983d7e2425478197199b7881","tarball":"https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.1.tgz","integrity":"sha512-/f+A7C3gucLtZ6F6z33sFBFxIrry4KPiO4S1r9KrwNv6ABp/T+IHJzzYGRFCzs2RfgTIm8cA3TJuTdc8INlkNQ==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIFClzWQOpZ1wLeTBejlQ3EjmoTEqyiS3w8drAjflDL11AiEArVdXKBnfnVWx8KOhvLopIg51cETWWrVnd4GqPiWVvCU="}]}},"1.0.2":{"name":"util-deprecate","version":"1.0.2","dist":{"shasum":"450d4dc9fa70de732762fbd2d4a28981419a0ccf","tarball":"https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz","integrity":"sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQD8kbTkCOi5K8s9SSUowetM2i82Yn0Fh8ksO5yTwvPaOAIgLZuKcelfvJTrdYwZOm4QxW2K6ilsaa/SRexFQHyGHvI="}]}}},"modified":"2022-06-28T07:10:45.145Z","cachedAt":1695429623143}

NOTE: If the PNPM store and the project have a boundary that'd prevent using hardlinks, I have observed PNPM relocating / recreating the store (on linux this moved from /root/.local/share/pnpm/store to /.pnpm-store). This can happen with different disks (or volume mounts in containers), where PNPM is trying to be helpful to avoid duplicating storage between store and node_modules.

Not sure if /root/.cache/pnpm/metadata is providing any benefit when you have a lockfile. I possibly need to flush file cache, as time pnpm install (_after clearing the store and node_modules_) was varying between 5-11s regardless of the PNPM cache dir being kept or removed.

Yarn (v3.6.3)

# Lookup a package from the lock file, the SHA-512 hash (checksum field) is a store key:
$ grep -A5 '^"util-deprecate' yarn.lock
"util-deprecate@npm:^1.0.1, util-deprecate@npm:^1.0.2, util-deprecate@npm:~1.0.1":
  version: 1.0.2
  resolution: "util-deprecate@npm:1.0.2"
  checksum: 474acf1146cb2701fe3b074892217553dfcf9a031280919ba1b8d651a068c9b15d863b7303cb15bd00a862b498e6cf4ad7b4a08fb134edd5a6f7641681cb54a2
  languageName: node
  linkType: hard

# The store is very similar to PNPM, but instead of `-index.json` there is just `.json` suffix,
# while other file content has a `.dat` suffix and only uses a SHA-256 hash.
$ cat /root/.yarn/berry/store/v1/47/4acf1146cb2701fe3b074892217553dfcf9a031280919ba1b8d651a068c9b15d863b7303cb15bd00a862b498e6cf4ad7b4a08fb134edd5a6f7641681cb54a2.json
{"package.json":{"kind":"file","mode":33188,"digest":"2e69081e7bab6e09d3dcfd680716fdeea577431d","mtimeMs":1695429933850},"README.md":{"kind":"file","mode":33188,"digest":"8e590b24df7b037031e571b7b2b9600217b83ae0","mtimeMs":1695429933851},"LICENSE":{"kind":"file","mode":33188,"digest":"cbde92577cb69b3b45dd33f8dc600732cf9e14cb","mtimeMs":1695429933852},"browser.js":{"kind":"file","mode":33188,"digest":"335965112d6117af8926dce4497a1fb9fca022eb","mtimeMs":1695429933853},"node.js":{"kind":"file","mode":33188,"digest":"26bb9fcabaf57f0bb50e5e026c13de394bc0c478","mtimeMs":1695429933853},"History.md":{"kind":"file","mode":33188,"digest":"1fc5b9cf603a0b6abeb852b35bec607a411e5b9b","mtimeMs":1695429933854}}

# `package.json` is also stored in the `.zip` file of the package in yarns cache:
$ cat /root/.yarn/berry/store/v1/2e/69081e7bab6e09d3dcfd680716fdeea577431d.dat
$ unzip /root/.yarn/berry/cache/util-deprecate-npm-1.0.2-e3fe1a219c-8.zip

NOTE: Related to the PNPM note, yarn behaves differently and falls back to nmMode: hardlinks-local instead. It would not attempt to create a store somewhere else and avoid hardlinks to a global store.

brawaru commented 1 year ago

I'm not sure why you're reviving this issue. All of the complaints were indeed for v1 which we used at the time, except for v3 migration attempt report. Given that you have to go lengths to explain us the little details of Yarn v3 workings and configuration, this is why we've chosen pnpm. It's just much easier: all we needed to do (which nuxi does, as you say) is to add shamefully-hoist=true to .npmrc, change workflow commands and do pnpm install :))

Nothing against Yarn 3+, I've heard people have good experiences with it when it's working for them, but for Knossos using something that is used across the industry now makes a lot more sense. We are not target audience of Yarn, which focuses more on pnp, project level isolation, etc. It's overcomplicated for our case, even if we can get it to work how we want, and has no apparent benefit.

Meanwhile pnpm is dead simple almost drop-in replacement for npm, the commands are almost all similar, and global cache helps when you have a bunch of similar projects (I regularly spin up Nuxt and Vite instances to test stuff). You're not losing much (anything?) by switching to it, but get huge performance and productivity benefits as a result.

polarathene commented 1 year ago

I'm not sure why you're reviving this issue.

Sorry I came across it from the nuxt yarn issue you referenced. Saw the rant about yarn caching being terrible + the esbuild concern and just wanted to clear that up, but it became a lot more than I originally intended.

Not just for your benefit, but anyone else that would stumble upon this 😅

Given that you have to go lengths to explain us the little details of Yarn v3 workings and configuration, this is why we've chosen pnpm.

It could have been said with less words and clarity, but I figured it was better to be verbose than vague.

It's just much easier: all we needed to do (which nuxi does, as you say) is to add shamefully-hoist=true to .npmrc, change workflow commands and do pnpm install :))

Yeah I totally understand but that's specifically for Nuxt. I would assume not all projects do that, and you'd have to become aware of that configuration elsewhere, just like you would with the equivalent for Yarn.

I've already raised an issue for nuxi to do the same for their yarn support.

We are not target audience of Yarn, which focuses more on pnp, project level isolation, etc. It's overcomplicated for our case, even if we can get it to work how we want, and has no apparent benefit.

It doesn't have to be complicated. Like with shamefully-hoist=true, you'd just use nodeLinker: node-modules + nmMode: hardlinks-global.

Meanwhile pnpm is dead simple almost drop-in replacement for npm, the commands are almost all similar, and global cache helps when you have a bunch of similar projects

I actually found pnpm to have more similarities with yarn, and as mentioned yarn also has a global cache (and optionally a store like pnpm when using the two config settings I mentioned).

I've not observed any major performance difference between the two with global stores (with nuxi init).

Yarn was a bit more efficient storage wise. PNPM was creating two 9MB esbuild binaries (different versions, yarn only has these in the disposable .zip cache), with 21MB node_modules vs 77KB for yarn. Store size was roughly the same (yarn a 2MB smaller).

You're not losing much (anything?) by switching to it, but get huge performance and productivity benefits as a result.

A project I inherited recently is Nuxt based with yarn being used. I haven't worked with NodeJS for a while and was just getting clued up with what the options are out there, and how to go about proper caching for the CI and docker images.

I'm not sure what the huge performance and productivity benefits are from PNPM vs Yarn, both seem capable. bun is supposedly more capable but not mature enough yet. I've found each to have their pros/cons, but I like where PNPM is going with pacquet.

If you know of anything that PNPM actually does better than Yarn, I'd be interested to know 👍

My main takeaway here is that PNPM was just better supported by Nuxt out of the box currently, and it works well for you so no need to switch.