[Feature]: package manager switch

Hotell commented 1 year ago

Describe the feature that you would like added

ATM we use yarn midgard (msft fork of yarn v1 ) - it seems it doesn't have much traction from both feature additions/supporting the package.

"outside of fluent" family of packages maintained by core team uses yarn v3. It would be nice to unify on what package manager is used across our scope of projects.

"on paper" yarn 3 / pnpm should be faster and provide more richer feature set for our needs.

Features we are looking for:

fastest possible installs
- cold cache
- switching branches
proper dependency deduplication
lowest disk size usage
ability to patch installed packages
workspaces support
stricter dependency enforcement to avoid ghost dependencies and potential bugs/security issues
compatibility with single version policy approach

Have you discussed this feature with our team

v-build

Additional context

No response

Validations

[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Priority

None

Hotell commented 1 year ago

I've done some research/experimentation during FHL and tried to migrate our repo to following package managers:

Benchmarks

run on macos M1 MAX

package manager	install (no cache)	install (cache)	node_modules size
yarn 1 - midgard (Baseline)	114s	31s	1.3 GB
yarn 3 (nodeLinker: node-modules)	74s 🚅 (35% speed increase)	44s 🚨 (42% speed decrease)	1.7 GB 🚨 (31% size increase)
yarn 4 rc (nodeLinker: node-modules)	94s 🚅 (18% speed increase)	45s 🚨 (45% speed decrease)	1.7 GB 🚨 (31% size increase)
pnpm	88s 🚅 (23% speed increase)	28s 🚅 (10% speed increase)	1.1 GB 🚅 (15% size decrease)
bun	36s 🚅🚅 (68% speed increase)	14s 🚅🚅 (54% speed increase)	3.1 GB 🚨 (138% size increase)

yarn v1 - midgard

pros:

works the same as normal yarn
faster than v1

cons:

ghost dependencies / potential security issues
nohoist doesn't properly work in some use-cases (react 18 and 17 types)
no native proper dedupe mechanism
no native package patching
no workspace:* protocol

yarn 3

pros:

relatively convenient switch for contributors
nodeLinker node-modules is relatively easy for migration
workspace:* protocol
native package patching

cons:

packages (and their binaries) installed only in root package.json don't work when executing from within workspace scope. They need to be explicitly defined per workspace "scripts" via yarn run -T <binary-name>
- this should work with PNP mode though
confusing behaviours when workspace packages are used within workspace configuration source - things work although they should not as nor root nor workspace package.json doesn't have specified those as dependencies
nodeLinker pnpm triggers lots of errors (in comparison with native pnpm pm)

pnpm (v8)

pros:

excellent node_modules size reduction ( thanks to shared storage )
zero ghost dependencies within workspaces
workspace:* protocol
nicely complements single version policy approach
- packages (and their binaries) installed only in root package.json just work when executing from within workspace
- pnpm --filter @fluentui/react-text exec tsc -v works
- ```
"@fluentui/scripts-prettier": "workspace:*",
"@fluentui/scripts-storybook": "workspace:*",
"@fluentui/scripts-babel": "workspace:*",
```
  adding workspace packages to root /package.json makes them available globally within any workspaces
patching packages is build-in
lerna support

cons:

inconvenient switch for contributors ?

bun (v1)

pros:

install speed is superior to any existing pm
one can use yarn v1 "as is" for task execution (incl workspaces) etc

cons:

having 2 "pm" in monorepo/ci is confusing
huge node_modules size delta
implications how to declare dependencies

more details:

every workspace dependency needs to use workspace:* protocol to enable hoisting
every non workspace dependency needs to have exactly same version specified as in root package.json to properly enable hoisting
one of reasons of significant node_modules size increase
react-northstar doesn't uses workspaces and lerna is not supported via bun ( additional work needed to migrate N* to bun/yarn workspaces)
- one of reasons of significant node_modules size increase

Summary

Based on initial benchmarks and our use cases testing it looks like we have 3 options:

continue using yarn midgard
switch to pnpm
once bun ships full package manager functionality for monorepos we might consider switching
- https://github.com/oven-sh/bun/issues/533
- https://github.com/oven-sh/bun/issues/2450

All of those option will nicely complement to our goal to adopt single version policy.

Bessonov commented 1 year ago

I am wondering how you have checked the size for bun. bun uses hardlinks, which can lead to incorrect size measurements.

Personally, I think pnpm is a great choice, and bun is far from being stable.

Hotell commented 1 year ago

I am wondering how you have checked the size for bun.

great question ! I used simple find . -name "node_modules" -type d -prune -print | xargs du -chs

yeah adding bun to the metrics was just pure experiment.

I run yarn 3 with various nmMode as well do get further metrics:

nmMode	size
classic	1.7 GB
hardlinks-local	1.1 GB (same as pnpm)
hardlins-global	1.1 GB

Bessonov commented 1 year ago

Ok, this should work. Don't forget to look at ~/.bun/install/cache/ and ~/.cache/pnpm/metadata/ :+1:

microsoft-github-policy-service[bot] commented 8 months ago

Because this issue has not had activity for over 150 days, we're automatically closing it for house-keeping purposes.

Still require assistance? Please, create a new issue with up-to date details.

microsoft / fluentui