Closed brawaru closed 1 year ago
I've been using pnpm when contributing, never encountered any issues using it at all.
Therefore you end up with binaries of esbuild for all operating systems (including freaking Android!) for several versions of esbuild.
I couldn't reproduce this? (I am aware of this screenshot)
# Starting with a reproducible environment:
$ docker run --rm -it --workdir /reproduction node:20-bookworm-slim bash
# Requirement (user should already have applied via reading yarn install docs):
$ corepack enable
# Temporary workaround until `package.json:packageManager` is implemented in `nuxi init`:
$ corepack prepare yarn@stable --activate
$ yarn -v
3.6.3
# Temporary linker workaround with ENV (`yarn config` not usable until `package.json` exists):
$ YARN_NODE_LINKER=node-modules yarn dlx nuxi init --packageManager yarn --gitInit false .
# esbuild only installed appropriate platform:
$ ls node_modules/@esbuild
linux-x64
$ ls node_modules/esbuild/node_modules/@esbuild
linux-x64
$ ls -l /root/.yarn/berry/cache/ | grep esbuild
-rw-r--r-- 1 root root 4706 Sep 22 21:54 @esbuild-kit-cjs-loader-npm-2.4.4-24b1896f49-8.zip
-rw-r--r-- 1 root root 25285 Sep 22 21:54 @esbuild-kit-core-utils-npm-3.3.2-f27be25172-8.zip
-rw-r--r-- 1 root root 6248 Sep 22 21:54 @esbuild-kit-esm-loader-npm-2.6.5-431b2c6e16-8.zip
-rw-r--r-- 1 root root 3929603 Sep 22 21:54 @esbuild-linux-x64-npm-0.18.20-de8e99b449-8.zip
-rw-r--r-- 1 root root 3967319 Sep 22 21:54 @esbuild-linux-x64-npm-0.19.3-55823289f9-8.zip
-rw-r--r-- 1 root root 33040 Sep 22 21:54 esbuild-npm-0.18.20-004a76d281-8.zip
-rw-r--r-- 1 root root 33106 Sep 22 21:54 esbuild-npm-0.19.3-32eb4a916f-8.zip
EDIT: Ah ok, it's specifically a yarn 1.x issue:
$ docker run --rm -it --workdir /reproduction node:20-bookworm-slim bash
$ yarn -v
1.22.19
$ yarn init --yes
$ yarn add esbuild
$ du -sh $(yarn cache dir)
279M /usr/local/share/.cache/yarn/v6
$ ls -l /usr/local/share/.cache/yarn/v6
total 92
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-android-arm-0.19.3-08bd09f2ebc312422f4e94ae954821f9cf37b39e-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-android-arm64-0.19.3-91a3b1b4a68c01ffdd5d8ffffb0a83178a366ae0-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-android-x64-0.19.3-b1dffec99ed5505fc57561e8758b449dba4924fe-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-darwin-arm64-0.19.3-2e0db5ad26313c7f420f2cd76d9d263fc49cb549-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-darwin-x64-0.19.3-ebe99f35049180023bb37999bddbe306b076a484-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-freebsd-arm64-0.19.3-cf8b58ba5173440ea6124a3d0278bfe4ce181c20-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-freebsd-x64-0.19.3-3f283099810ef1b8468cd1a9400c042e3f12e2a7-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-arm-0.19.3-ff6a2f68d4fc3ab46f614bca667a1a81ed6eea26-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-arm64-0.19.3-a8b3aa69653ac504a51aa73739fb06de3a04d1ff-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-ia32-0.19.3-5813baf70e406304e8931b200e39d0293b488073-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-loong64-0.19.3-21110f29b5e31dc865c7253fde8a2003f7e8b6fd-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-mips64el-0.19.3-4530fc416651eadeb1acc27003c00eac769eb8fd-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-ppc64-0.19.3-facf910b0d397e391b37b01a1b4f6e363b04e56b-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-riscv64-0.19.3-4a67abe97a495430d5867340982f5424a64f2aac-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-s390x-0.19.3-c5fb47474b9f816d81876c119dbccadf671cc5f6-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-linux-x64-0.19.3-f22d659969ab78dc422f1df8d9a79bc1e7b12ee3-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-netbsd-x64-0.19.3-e9b046934996991f46b8c1cadac815aa45f84fd4-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-openbsd-x64-0.19.3-b287ef4841fc1067bbbd9a60549e8f9cf1b7ee3a-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-sunos-x64-0.19.3-b2b8ba7d27907c7245f6e57dc62f3b88693f84b0-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-win32-arm64-0.19.3-1974c8c180c9add4962235662c569fcc4c8f43dd-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-win32-ia32-0.19.3-b02cc2dd8b6aed042069680f01f45fdfd3de5bc4-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-@esbuild-win32-x64-0.19.3-e5036be529f757e58d9a7771f2f1b14782986a74-integrity
drwxr-xr-x 3 root root 4096 Sep 23 04:20 npm-esbuild-0.19.3-d9268cd23358eef9d76146f184e0c55ff8da7bb6-integrity
In my short testing Yarn Berry has failed to work with Knossos because it expects strict dependencies, another thing that Nuxt requires to be set to disabled, but I haven't found option to do this with Yarn Berry.
# Compatible `node_modules` folder approach:
yarn config set nodeLinker node-modules
# Use hardlinks with a global store like PNPM:
yarn config set nmMode hardlinks-global
I honestly don't think it's worth the hassle to get it working, especially with pnpm and npm working right now without additional configuration (asides from what we have).
PNPM isn't compatible without configuring non-defaults either. nuxi init
creates a .npmrc
to accommodate that, lacking similar functionality with yarn though.
This isn't a response to persuade you to switch back to Yarn btw, just providing insights to the walls you hit.
Yarn's terrible caching strategy. Back in the days Yarn was much more space efficient than npm, but this has since changed. During active usage of npm my its cache store only accumulated 2 GB, whereas solely by working with Knossos my Yarn cache directory accumulated 4.3 GB.
nodeLinker: pnp
by default.compressionLevel: mixed
(gzip 9), while RC releases of v4 are using compressionLevel: 0
(none).enableGlobalCache: false
+ enableMirror: true
, while v4 uses enableGlobalCache: true
(which makes the mirror setting irrelevant).If compressionLevel
is not 0:
yarn cache clean
(--mirror
to clean the global cache instead, --all
to clean both).If you opt-out of the global cache with enableGlobalCache: false
:
enableMirror: false
.yarn.lock
. This is very space efficient for a single project.pnpm store prune
(but should also configure a store that is not global).With the default linker nodeLinker: pnp
:
node_modules
is needed and the global cache can be re-used efficiently across many projects. NPM and PNPM (if it cannot create hardlinks to global store) will make copies.nodeLinker: node-modules
.
nmMode: hardlinks-global
.node_modules
by creating a new .pnpm-store/
directory at the filesystem root, unless one was configured explicitly_). Another culprit is that Yarn doesn't cache individual files but rather whole packages. This is very space inefficient since you end up with lots of duplications of multiple versions of the package that haven't changed much between these versions.
I don't think that's different with npm or pnpm?
Yarn does have a nmMode: hardlinks-local
feature or nodeLinker: pnpm
(hardlinks + symlinks) which should mimic whatever optimization you're thinking of? I don't think any of the package managers have a global cache where they're using hardlinks between package versions though?
Your experience might have been with the compressionLevel
setting which creates those .zip
archives. It seems yarn v4 is not using compression by default (EDIT: the yarn cache dir still uses .zip
pacakges regardless). Some filesystems can be more efficient with duplicate file content and optionally provide their own transparent compression.
EDIT: Looked into this and it's the global store
nodeLinker: pnpm
with yarn only seems to have a local store.nodeLinker: node-modules
+ nmMode: hardlinks-global
will use a global store like PNPM..zip
packages, while PNPM cache contains .json
files that provide metadata for each package, roughly the same for nuxi init
(45M vs 49M cache, 155M vs 157M store))they recommend to migrate to Berry (v2+). And it already broke on my pull request that does not do anything unusual, which can be confirmed by simply using npm and seeing that it works with no issue, and I don't even know how to mitigate this!
It could be that I misunderstood, and all your complaints were regarding yarn v1 after this statement.
As mentioned newer versions of yarn changed the linker, but while it can usually install packages fine, it doesn't always work well at resolving them. It's usually due to projects with implicit dependencies, relying on accessing packages that aren't declared explicitly in your package.json
(like Nuxt will access the Vue package) and this sort of behaviour IIRC is considered bad practice (as you noted with "npm quirks").
There's a similar compatibility issue with PNPM, but Nuxt now configures pnpm (via nuxi init
at least) to fallback to hoisting packages, which yarn can do as well, point is neither of these are defaults.
You'd have been able to use nodeLinker: node-modules
and everything would have gone smoothly.
In my short testing Yarn Berry has failed to work with Knossos because it expects strict dependencies, another thing that Nuxt requires to be set to disabled, but I haven't found option to do this with Yarn Berry.
If you don't want to use nodeLinker: node-modules
there is a setting for this IIRC, just visit the yarn docs for the .yarnrc.yml
config and search for strict
/ loose
, that may be what you were after.
EDIT: The setting pnpMode: loose
doesn't seem to help for nuxi init
.
Store comparison between PNPM and Yarn (nodeLinker: node-modules
+ nmMode: hardlinks-global
) on a project (created with nuxi init
):
# Lookup a package from the lock file, the SHA-512 hash is a store key:
$ grep -A2 '/util-deprecate' pnpm-lock.yaml
/util-deprecate@1.0.2:
resolution: {integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==}
dev: true
# The hash key was base64 encoded to reduce ASCII representation size,
# convert to hexadecimal which the store uses:
$ base64 -d <<< 'EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==' | xxd -ps -c 0
10f0f9ab5b97c85c49a42acb9c27359c79eade039ae83641a1c008888d93692080ed5089d5424331a802cc891736c5187c3d5d68afff2d3110f318886eb1ed73
# The store groups files under the first byte (two hex chars) of the hash key, with the rest of the hash as the filename,
# The lockfile is referencing the package json, so there is an additional `-index.json` suffix:
$ cat /root/.local/share/pnpm/store/v3/files/10/f0f9ab5b97c85c49a42acb9c27359c79eade039ae83641a1c008888d93692080ed5089d5424331a802cc891736c5187c3d5d68afff2d3110f318886eb1ed73-index.json
# JSON content output identifies each file from the package, each with their own hash that can find their content in the store:
{"name":"util-deprecate","version":"1.0.2","files":{"package.json":{"checkedAt":1695429623187,"integrity":"sha512-t8JHXKSuqDTJvzONFc6YAbMKMwRsaL539wb4WVOyeswdTSLpdY+tELBK8moq94CIMMhXSL+LfbzV7FiMLCkQ/g==","mode":420,"size":694},"README.md":{"checkedAt":1695429623187,"integrity":"sha512-gnBkJ+KrW26XBjy0Bo6pbA/s859Tkt4qQlbdoKTgxfAipxn+1WpMGzLfgKGNOxwU79KDGncXTpWrH7dRgq61+A==","mode":420,"size":1666},"LICENSE":{"checkedAt":1695429623187,"integrity":"sha512-hElreSqhgIRnqBHxtPWF+70iv+3/gk9uLTHUdCjnckMF7c7+wGiuFhYUXP5rWupepd6bGK2MIUW5u2aY2cVXRg==","mode":420,"size":1102},"browser.js":{"checkedAt":1695429623187,"integrity":"sha512-ZGsJKmbXf2naHjO9wg8QN4ZaAzaMfo6DmU3cfmz4rCRncoFaE9QFoRUaVSNyEtCBuWaZo4kLTfVVB9LMDJdj4Q==","mode":420,"size":1614},"node.js":{"checkedAt":1695429623187,"integrity":"sha512-6GDUjKSud32WOrZmqumfNxm98za/IYsoK3aioPAmjKa3KDv4yCVUSg7P29u/8863yYZJ2J+VZl06Xisvba7cDg==","mode":420,"size":123},"History.md":{"checkedAt":1695429623187,"integrity":"sha512-TUTHdDTG8WIwOSVQ0ChcJo/5P1FeZCZJ/fMRV5/JFBJ18zqVtBuTl9/pj5aGVw8ncL6ptnGGBfNopxG/76Ej3g==","mode":420,"size":282}}}
# A related JSON file is used by PNPM for package cache of the registry.
# PNPM does not cache the registry zip locally, so if the store lacks the extracted content, it'll fetch from the registry:
$ cat /root/.cache/pnpm/metadata/registry.npmjs.org/util-deprecate.json
{"name":"util-deprecate","dist-tags":{"latest":"1.0.2"},"versions":{"1.0.0":{"name":"util-deprecate","version":"1.0.0","dist":{"shasum":"3007af012c140eae26de05576ec22785cac3abf2","tarball":"https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.0.tgz","integrity":"sha512-sTmIWz2UtUfg8kaf6qlicnMn6ghnpMboyWJAv+kgorwAmCHY78TcfaRWfMD8OECWkqVFwyat+r1VJxA4dDfGSA==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQCri2jrXgd4KNJxbVAHav4n3MwZ3UrEqrKZ5X54Efk2/wIgI85j6D2+98v25iZu+ygxEUNvgmL3J1aehucOqaCD7hg="}]}},"1.0.1":{"name":"util-deprecate","version":"1.0.1","dist":{"shasum":"3556a3d13c4c6aa7983d7e2425478197199b7881","tarball":"https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.1.tgz","integrity":"sha512-/f+A7C3gucLtZ6F6z33sFBFxIrry4KPiO4S1r9KrwNv6ABp/T+IHJzzYGRFCzs2RfgTIm8cA3TJuTdc8INlkNQ==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIFClzWQOpZ1wLeTBejlQ3EjmoTEqyiS3w8drAjflDL11AiEArVdXKBnfnVWx8KOhvLopIg51cETWWrVnd4GqPiWVvCU="}]}},"1.0.2":{"name":"util-deprecate","version":"1.0.2","dist":{"shasum":"450d4dc9fa70de732762fbd2d4a28981419a0ccf","tarball":"https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz","integrity":"sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==","signatures":[{"keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA","sig":"MEUCIQD8kbTkCOi5K8s9SSUowetM2i82Yn0Fh8ksO5yTwvPaOAIgLZuKcelfvJTrdYwZOm4QxW2K6ilsaa/SRexFQHyGHvI="}]}}},"modified":"2022-06-28T07:10:45.145Z","cachedAt":1695429623143}
NOTE: If the PNPM store and the project have a boundary that'd prevent using hardlinks, I have observed PNPM relocating / recreating the store (on linux this moved from /root/.local/share/pnpm/store
to /.pnpm-store
). This can happen with different disks (or volume mounts in containers), where PNPM is trying to be helpful to avoid duplicating storage between store and node_modules
.
Not sure if /root/.cache/pnpm/metadata
is providing any benefit when you have a lockfile. I possibly need to flush file cache, as time pnpm install
(_after clearing the store and node_modules
_) was varying between 5-11s regardless of the PNPM cache dir being kept or removed.
# Lookup a package from the lock file, the SHA-512 hash (checksum field) is a store key:
$ grep -A5 '^"util-deprecate' yarn.lock
"util-deprecate@npm:^1.0.1, util-deprecate@npm:^1.0.2, util-deprecate@npm:~1.0.1":
version: 1.0.2
resolution: "util-deprecate@npm:1.0.2"
checksum: 474acf1146cb2701fe3b074892217553dfcf9a031280919ba1b8d651a068c9b15d863b7303cb15bd00a862b498e6cf4ad7b4a08fb134edd5a6f7641681cb54a2
languageName: node
linkType: hard
# The store is very similar to PNPM, but instead of `-index.json` there is just `.json` suffix,
# while other file content has a `.dat` suffix and only uses a SHA-256 hash.
$ cat /root/.yarn/berry/store/v1/47/4acf1146cb2701fe3b074892217553dfcf9a031280919ba1b8d651a068c9b15d863b7303cb15bd00a862b498e6cf4ad7b4a08fb134edd5a6f7641681cb54a2.json
{"package.json":{"kind":"file","mode":33188,"digest":"2e69081e7bab6e09d3dcfd680716fdeea577431d","mtimeMs":1695429933850},"README.md":{"kind":"file","mode":33188,"digest":"8e590b24df7b037031e571b7b2b9600217b83ae0","mtimeMs":1695429933851},"LICENSE":{"kind":"file","mode":33188,"digest":"cbde92577cb69b3b45dd33f8dc600732cf9e14cb","mtimeMs":1695429933852},"browser.js":{"kind":"file","mode":33188,"digest":"335965112d6117af8926dce4497a1fb9fca022eb","mtimeMs":1695429933853},"node.js":{"kind":"file","mode":33188,"digest":"26bb9fcabaf57f0bb50e5e026c13de394bc0c478","mtimeMs":1695429933853},"History.md":{"kind":"file","mode":33188,"digest":"1fc5b9cf603a0b6abeb852b35bec607a411e5b9b","mtimeMs":1695429933854}}
# `package.json` is also stored in the `.zip` file of the package in yarns cache:
$ cat /root/.yarn/berry/store/v1/2e/69081e7bab6e09d3dcfd680716fdeea577431d.dat
$ unzip /root/.yarn/berry/cache/util-deprecate-npm-1.0.2-e3fe1a219c-8.zip
NOTE: Related to the PNPM note, yarn behaves differently and falls back to nmMode: hardlinks-local
instead. It would not attempt to create a store somewhere else and avoid hardlinks to a global store.
I'm not sure why you're reviving this issue. All of the complaints were indeed for v1 which we used at the time, except for v3 migration attempt report. Given that you have to go lengths to explain us the little details of Yarn v3 workings and configuration, this is why we've chosen pnpm. It's just much easier: all we needed to do (which nuxi
does, as you say) is to add shamefully-hoist=true
to .npmrc
, change workflow commands and do pnpm install
:))
Nothing against Yarn 3+, I've heard people have good experiences with it when it's working for them, but for Knossos using something that is used across the industry now makes a lot more sense. We are not target audience of Yarn, which focuses more on pnp, project level isolation, etc. It's overcomplicated for our case, even if we can get it to work how we want, and has no apparent benefit.
Meanwhile pnpm is dead simple almost drop-in replacement for npm, the commands are almost all similar, and global cache helps when you have a bunch of similar projects (I regularly spin up Nuxt and Vite instances to test stuff). You're not losing much (anything?) by switching to it, but get huge performance and productivity benefits as a result.
I'm not sure why you're reviving this issue.
Sorry I came across it from the nuxt yarn issue you referenced. Saw the rant about yarn caching being terrible + the esbuild concern and just wanted to clear that up, but it became a lot more than I originally intended.
Not just for your benefit, but anyone else that would stumble upon this 😅
Given that you have to go lengths to explain us the little details of Yarn v3 workings and configuration, this is why we've chosen pnpm.
It could have been said with less words and clarity, but I figured it was better to be verbose than vague.
It's just much easier: all we needed to do (which
nuxi
does, as you say) is to addshamefully-hoist=true
to.npmrc
, change workflow commands and dopnpm install
:))
Yeah I totally understand but that's specifically for Nuxt. I would assume not all projects do that, and you'd have to become aware of that configuration elsewhere, just like you would with the equivalent for Yarn.
I've already raised an issue for nuxi
to do the same for their yarn support.
We are not target audience of Yarn, which focuses more on pnp, project level isolation, etc. It's overcomplicated for our case, even if we can get it to work how we want, and has no apparent benefit.
It doesn't have to be complicated. Like with shamefully-hoist=true
, you'd just use nodeLinker: node-modules
+ nmMode: hardlinks-global
.
Meanwhile pnpm is dead simple almost drop-in replacement for npm, the commands are almost all similar, and global cache helps when you have a bunch of similar projects
I actually found pnpm to have more similarities with yarn, and as mentioned yarn also has a global cache (and optionally a store like pnpm when using the two config settings I mentioned).
I've not observed any major performance difference between the two with global stores (with nuxi init
).
Yarn was a bit more efficient storage wise. PNPM was creating two 9MB esbuild binaries (different versions, yarn only has these in the disposable .zip
cache), with 21MB node_modules
vs 77KB for yarn. Store size was roughly the same (yarn a 2MB smaller).
You're not losing much (anything?) by switching to it, but get huge performance and productivity benefits as a result.
A project I inherited recently is Nuxt based with yarn being used. I haven't worked with NodeJS for a while and was just getting clued up with what the options are out there, and how to go about proper caching for the CI and docker images.
I'm not sure what the huge performance and productivity benefits are from PNPM vs Yarn, both seem capable. bun
is supposedly more capable but not mature enough yet. I've found each to have their pros/cons, but I like where PNPM is going with pacquet
.
If you know of anything that PNPM actually does better than Yarn, I'd be interested to know 👍
My main takeaway here is that PNPM was just better supported by Nuxt out of the box currently, and it works well for you so no need to switch.
nuxi
addressed it. This isn't the only issue I've seen PNPM users struggle with either, thus good timing for you and your requirements? :)nuxi
also seems to add .npmrc
at the root of the disk as a fallback presumably, but if you're not aware of that any project that would benefit from not hoisting is implicitly configured to do that for the convenience of Nuxt 🤷♂️
Is your suggested enhancement related to a problem? Please describe.
Not so long ago Modrinth repositories were switched to use legacy
yarn
v1 while previously usingnpm
. I think this decision was shortsighted, misinformed and suboptimal.One of the huge problems with legacy Yarn is that it's now frozen and not actively maintained, it has not been maintained since 2021-2022 and they recommend to migrate to Berry (v2+). And it already broke on my pull request that does not do anything unusual, which can be confirmed by simply using npm and seeing that it works with no issue, and I don't even know how to mitigate this!
Another problem I have encountered is Yarn's terrible caching strategy. Back in the days Yarn was much more space efficient than npm, but this has since changed. During active usage of npm my its cache store only accumulated 2 GB, whereas solely by working with Knossos my Yarn cache directory accumulated 4.3 GB.
The culprit? Yarn ignores the system requirements for packages and installs them even though they're clearly marked as incompatible with current OS. Therefore you end up with binaries of esbuild for all operating systems (including freaking Android!) for several versions of esbuild.
Another culprit is that Yarn doesn't cache individual files but rather whole packages. This is very space inefficient since you end up with lots of duplications of multiple versions of the package that haven't changed much between these versions.
Other issues I had with Yarn is it getting completely stuck on certain steps and not showing anything even with verbose logs, but even I find the reason, this will never get fixed anymore, as the yarn is frozen (effectively archived).
I don't think Yarn Berry is the right choice for Modrinth repos either as we rely one some of npm's quirks such as dependency hoisting. I'm saying ‘we’, but not like its our choice, it's a problem with Nuxt, really.
In my short testing Yarn Berry has failed to work with Knossos because it expects strict dependencies, another thing that Nuxt requires to be set to disabled, but I haven't found option to do this with Yarn Berry.
Describe the solution you'd like
So given we want to preserve some of the npm's quirks, but don't want to use npm due to how slow and space inefficient it is, I would propose switching to pnpm, which is gaining popularity among JavaScript developers.
Previously Geo dismissed pnpm calling it annoying, but I'm yet to see the reasons why it's annoying. Perhaps it was misconfigured and caused trouble with Nuxt, but there's a
.npmrc
config now which includes stuff likeshamefully-hoist
, which will be respected as long as pnpm is ran from the root of the project.pnpm also has great DX shortcuts like Yarn:
pnpm [script / bin]
to quickly run commands likedev
(pnpm dev
) or binaries from project's dependencies likenuxi
(pnpm nuxi prepare
)pnpm i
(pnpm install
) orpnpm b
(pnpm run-script build
), and alsopnpm add [pkg]
(pnpm install [pkg]
)pnpm update -i
, or to see why certain dependency is presentpnpm why [dep]
pnpm patch [dep]
.My personal experience with pnpm is incredible and I can not recommend it to people enough!
Describe alternatives you've considered
Switching back to npm npm is very inefficient and slow. It received a lot of improvements and very close to legacy Yarn compared what it was, but I still think it's far far from great. However, it's still a better option than legacy Yarn, because it's continuously maintained, so if pnpm does not work, it can be an option.
Solving the problem and migrating to Yarn Berry Yarn currently promotes completely different model of Plug n' Play dependencies that aren't widely supported (and especially not supported by Nuxt), so it has to be configured to use npm linker, which is okay, but then there's the issue with peer dependencies. Yarn Berry is also pretty different from Yarn.
I honestly don't think it's worth the hassle to get it working, especially with pnpm and npm working right now without additional configuration (asides from what we have).
Additional context
I'm publishing this in Knossos repo because Omorphia and Theseus are out of our concern right now, and such switch might be disruptive to productivity there anyway, whereas the progress here for now is pretty chill allowing for experimentation like this.
Thanks for coming to my TED talk!