Monorepo support - Githubissues

wbcs commented 2 years ago

Updated issue

Bun supports "workspaces" in package.json, "workspace:* (protocol) in dependencies, and many more features related to workspaces. See the docs page on workspaces.

You can also use bun run --filter="*/*lib" to run multiple package.json scripts from different packages within a workspace simultaneously and view tail the logs - see the docs.

We're keeping this issue open for now, but it's not entirely clear what's left to close it.

Original issue below

What is the problem this feature will solve?

bun not support monorepo(root package.json workspaces field, workspace:* schema, etc.)

What is the feature you are proposing to solve the problem?

The download speed of monorepo will be faster

What alternatives have you considered?

No response

Jarred-Sumner commented 2 years ago

yes, but probably not for a little while

FinnRG commented 2 years ago

Related: The workspace: field progress is tracked in #83.

davidmeirlevy commented 2 years ago

can the implementation of workspaces include:

the ability to have separated node_modulues on each workspace.
the equivalent of lerna exec to execute any command inside the workspaces, even if it's not inside the package.json. for example, I want to run bun dev on each workspace, but the dev itself probably won't be a script inside the package.json of each workspace. the same for tests...

factoidforrest commented 2 years ago

I just want to say that I haven't missed lerna exec at all after ditching lerna for pure yarn workspaces. Things like that are certainly nice-to-haves but I don't think going any further than the support provided by yarn or NPM is necessary as a first-pass.

davidmeirlevy commented 2 years ago

@factoidforrest yarn has yarn exec that can work on workspaces, so it's equivalent.

unional commented 2 years ago

Maybe consider monorepo support similar to turbo and rush. They are both very efficient.

wbcs commented 2 years ago

nsider monorepo support similar to turbo and rush. They are both very efficie

I prefer pnpm

unional commented 2 years ago

I prefer pnpm

npm/yarn/pnpm are orthogonal to lerna/turbo/rush. One are package managers, the others are monorepo tools.

davidmeirlevy commented 2 years ago

I prefer pnpm

npm/yarn/pnpm are orthogonal to lerna/turbo/rush. One are package managers, the others are monorepo tools.

He probably meant to pnpm workspaces. I also prefer this way, because of the full scope isolation for each package.

ozyman42 commented 2 years ago

I prefer pnpm

npm/yarn/pnpm are orthogonal to lerna/turbo/rush. One are package managers, the others are monorepo tools.

He probably meant to pnpm workspaces. I also prefer this way, because of the full scope isolation for each package.

Yarn 2+ Plug n Play also has full scope isolation for each package. A main reason the pnpm resolution strategy might be preferable to Plug n Play is because notable projects such as React Native are currently incompatible with Plug n Play. On the flip side, Plug n Play might be considered preferable because it may solve the doppleganger problem differently than pnpm (someone ought to investigate this comparison in detail). Personally I find the "non-single singletons" doppleganger consequence the most important one because it's the root cause behind monorepo bugs involving popular libraries such as graphql-js.

Overall though I think Bun should make an authoritative choice for workspaces/monorepo support rather than letting this be highly configurable. Too much configurability leads to too much project complexity both in implementation and from a client's point of view w/ too many options to choose from & too much boilerplate. Bun needs to have a clear story for workspace/monorepos before it can be seriously considered for adoption by medium to large projects.

unional commented 2 years ago

Agree. IMO pnpm doppleganger solution is also the way to go.

The point I want to make by bring yo turbo and rush is the ability to efficiently running multiple scripts with dependencies.

For example: https://turborepo.org/docs/core-concepts/pipelines https://rushjs.io/pages/advanced/incremental_builds/ https://rushjs.io/pages/developer/selecting_subsets

wbcs commented 2 years ago

Agree. IMO pnpm doppleganger solution is also the way to go.

The point I want to make by bring yo turbo and rush is the ability to efficiently running multiple scripts with dependencies.

For example: https://turborepo.org/docs/core-concepts/pipelines https://rushjs.io/pages/advanced/incremental_builds/ https://rushjs.io/pages/developer/selecting_subsets

turborepo is greate. but how rush run custom npm scripts in subsets? It run reserved keyword instruction only.

unional commented 2 years ago

but how rush run custom npm scripts in subsets? It run reserved keyword instruction only.

It does not. rush has it's own way to add commands to the system. (to run custom npm scripts in one package, use rushx run ...) I personally like turbo better too. The rush system sounds too heavy and rigid to me (I tried both). Also mentioning rush here for completeness and also may able to draw an idea or two from it's selective script execution (those from/to/since things).

melMass commented 2 years ago

Hi,

IMHO the only responsibility from Bun regarding workspaces is dependency resolution, the rest should be handled by a monorepo tool like Turbo or NX.

o-alexandrov commented 1 year ago

Is npm run SCRIPT_NAME --workspaces part of this issue, or does it need a separate one?

evelant commented 1 year ago

Per https://github.com/oven-sh/bun/issues/83 it looks like some support for workspace: protocol was added but it doesn't appear to work with pnpm workspaces. Did I miss something or are there still additional changes needed to allow bun to work with a pnpm monorepo?

Bessonov commented 1 year ago

Probably, related: https://github.com/oven-sh/bun/issues/2517

jtlapp commented 1 year ago

Speed and disk space are the reasons pnpm is best for monorepos. Whenever multiple workspaces within a monorepo depend on the same npm package, npm and yarn download and install a separate copy of the npm package for each workspace, whereas pnpm symbolically links them all to the same download of the package. Yarn's PnP does muddy the waters a bit, but its problems keep me from using it. See the benchmarks.

scinos commented 1 year ago

Whenever multiple workspaces within a monorepo depend on the same npm package, npm and yarn download and install a separate copy of the npm package for each workspace

For yarn, it's configurable: https://yarnpkg.com/configuration/yarnrc#nmMode

jakeleventhal commented 1 year ago

Will migrate to bun when this works

Bessonov commented 1 year ago

I am not sure now, it isn't supported and described here? https://bun.sh/docs/install/workspaces Do I miss something or can this issue be closed as completed?

EDIT: I see it miss support for script execution inside monorepo, ex. pnpm run --filter mypackage start. Probably, should be tracked as a separate issue.

Hebilicious commented 1 year ago

@Bessonov I'm not sure it should be considered completed so soon, there's several workspace related bugs and missing features (like filtering that you mentioned)

I went through the open issues, it looks like there's a bunch of duplicate, but we could use this issue as a tracking one.

Bessonov commented 1 year ago

@Hebilicious Thank you very much for the excellent overview! I am still of the opinion that documented basic use cases should work. So, the binary answer to this issue is "yes, a monorepo is supported". I am not opposed to leaving this issue open, but, in my opinion, it distracts from addressing actual problems. Alternatively, @wbcs could add your list to the top of this issue, making it a meta-issue for outstanding work.

james-elicx commented 1 year ago

I feel it's relevant to share this tweet that I saw from Jarred a couple of months ago. He mentioned wanting to have another go at implementing workspaces as he didn't think the support was that good, although I'm not sure if they ever got around to redoing it.

https://twitter.com/jarredsumner/status/1682860099528978432

jakeleventhal commented 1 year ago

Additionally plugging this issue: https://github.com/oven-sh/bun/issues/4844

itpropro commented 1 year ago

@Hebilicious Thank you very much for the excellent overview! I am still of the opinion that documented basic use cases should work. So, the binary answer to this issue is "yes, a monorepo is supported". I am not opposed to leaving this issue open, but, in my opinion, it distracts from addressing actual problems. Alternatively, @wbcs could add your list to the top of this issue, making it a meta-issue for outstanding work.

It's hard to argue that bun has monorepo support in terms of a drop in replacement for pnpm. Replacing pnpm works for the most part, but as soon as you come to monorepos (which is very common in OSS and other projects these days), you have to redesign your whole CI/CD pipelines or have challenges that can be very hard to solve. You can't just run bun run --filter "@scope/package1" build instead of pnpm run --filter "@scope/package1" build etc.. There is still a long way to go and I am willing to go it with bun, but it should be communicated transparently that there is no real monorepo support as people expect when coming from pnpm or yarn.

jakeboone02 commented 1 year ago

@itpropro I think (1) "doesn't fail when used in a monorepo", (2) "supports monorepos", and (3) "includes tools that make working with monorepos easy" are all valid but very different definitions of "monorepo support". Bun currently meets the criteria for (1), sort of meets the criteria for (2), but definitely does not meet the criteria for (3).

As evidenced by @Hebilicious's list (https://github.com/oven-sh/bun/issues/533#issuecomment-1712865027) and other comments here, there's a lot of work to do before that last criteria is met. Documentation should probably reflect that.

danieldunderfelt commented 1 year ago

@itpropro I think (1) "doesn't fail when used in a monorepo", (2) "supports monorepos", and (3) "includes tools that make working with monorepos easy" are all valid but very different definitions of "monorepo support". Bun currently meets the criteria for (1), sort of meets the criteria for (2), but definitely does not meet the criteria for (3).

As evidenced by @Hebilicious's list (#533 (comment)) and other comments here, there's a lot of work to do before that last criteria is met. Documentation should probably reflect that.

Criteria 1 is not met yet, Bun 1.0 still fails for me when adding packages as described in #3502.

devdumpling commented 1 year ago

Not sure if it's been flagged yet, but is anyone else having issues running (bun run <script>) package.json scripts from outside of the root of the monorepo? I get a "FileNotFound" with even the simplest scripts.

idrakimuhamad commented 1 year ago

Not sure if it's been flagged yet, but is anyone else having issues running (bun run <script>) package.json scripts from outside of the root of the monorepo? I get a "FileNotFound" with even the simplest scripts.

I'm in the same boat. Also, i'm not sure how to run a specific workspace script eg with yarn something like yarn workspace my/ui run build

pmrotule commented 1 year ago

Also, i'm not sure how to run a specific workspace script eg with yarn something like yarn workspace my/ui run build

@idrakimuhamad As a workaround, I'm able to run a script of a specific package from the root by passing the path of the package as cwd:

bun run --cwd "packages/my-ui" build

Otherwise, if you have yarn or any other monorepo cli as a local dependency, you can always prefix the command with bun:

bun yarn workspace my/ui run build

or using nx:

bun nx run-many --target=build

I'm not sure yet if you have to pass the --bun flag as well (Force a script or package to use Bun's runtime instead of Node.js (via symlinking node)). I assume bun execute the bin file of yarn or nx, but if they use child processes running node cli, then --bun will be necessary.

pratiqdev commented 1 year ago

Hi,

IMHO the only responsibility from Bun regarding workspaces is dependency resolution, the rest should be handled by a monorepo tool like Turbo or NX.

Seems like the Bun team is already trying to accomplish a lot, why not squeeze in

2 parts incremental-builds and remote-caching (Turbo)
1 part automatic version / changelog management (changesets)
a dash of git-lifecycle integration (Husky)

ysfaran commented 1 year ago

What about dependency hoisting configuration?

I am thinking about migrating from yarn to bun as my package manager.

To make each workspaces really independent I use following confugration in yarn:

nmHoistingLimits: workspaces
nmMode: hardlinks-local

This basically means "Don't hoist dependencies above the workspace level, in case two workspaces use the same dependency@version use hardlinks". With this approach I am able to package every workspace independently in my pipeline and truly separate the packages file-wise from each other.

That's a must for any bigger monorepo IMO. Otherwise dependency management gets really messy and cumbersome.

websorokinweb commented 1 year ago

+1 dependency hoisting is the only thing that stops me from leaving yarn

Bessonov commented 1 year ago

@ysfaran @websorokinweb

What about dependency hoisting configuration?

I am thinking about migrating from yarn to bun as my package manager.

To make each workspaces really independent I use following confugration in yarn:
nmHoistingLimits: workspaces
nmMode: hardlinks-local
This basically means "Don't hoist dependencies above the workspace level, in case two workspaces use the same dependency@version use hardlinks". With this approach I am able to package every workspace independently in my pipeline and truly separate the packages file-wise from each other.

That's a must for any bigger monorepo IMO. Otherwise dependency management gets really messy and cumbersome.

Can you please explain the use case? I have been using monorepos for years for library, frontend, and backend development, and I have never experienced this need.

And I am not sure what you mean by "making each workspace really independent". Typically, monorepos are used for the opposite purpose, to create dependencies between projects.

websorokinweb commented 1 year ago

@Bessonov I don't know @ysfaran use case in details, but what I miss in bun right now is dependency hoisting.

You can see project structure there: https://github.com/developerway/example-react-project

I want to type bun install or whatever other command in the root of my project, which will install all my local packages inside /packages folder. As the result, all my dependencies are in the root package.json and I can use dependecies from packages by typing ONLY one command

Bessonov commented 1 year ago

@Bessonov I don't know @ysfaran use case in details, but what I miss in bun right now is dependency hoisting.

You can see project structure there: https://github.com/developerway/example-react-project

I want to type bun install or whatever other command in the root of my project, which will install all my local packages inside /packages folder. As the result, all my dependencies are in the root package.json and I can use dependecies from packages by typing ONLY one command

I haven't used Yarn for many years, but what you probably mean is that you specify all your dependencies in the root package.json instead of specifying them in the place where they are really used. More reliable dependency managers, such as PNPM, disallow this. I would suggest that you consider stopping this bad practice. What specific issue are you trying to address by doing this? Perhaps you are trying to keep versions in sync?

unional commented 1 year ago

@Bessonov I don't know @ysfaran use case in details, but what I miss in bun right now is dependency hoisting. You can see project structure there: https://github.com/developerway/example-react-project I want to type bun install or whatever other command in the root of my project, which will install all my local packages inside /packages folder. As the result, all my dependencies are in the root package.json and I can use dependecies from packages by typing ONLY one command

I haven't used Yarn for many years, but what you probably mean is that you specify all your dependencies in the root package.json instead of specifying them in the place where they are really used. More reliable dependency managers, such as PNPM, disallow this. I would suggest that you consider stopping this bad practice. What specific issue are you trying to address by doing this? Perhaps you are trying to keep versions in sync?

While I agree each package's dependencies should be independent, there are times that hoisting of devDependencies are desirable.

Here are some specific guidelines I can think of:

each package's dependencies should be independent and reside in each package. If they use the same dependencies, they should be using the same version. Especially when TypeScript is involved, or the dependency is not pure. If needed, use override to make sure they are using the same version.
Repo-related dependencies should be at the root for obvious reasons, e.g. commitlint, husky, turbo, nx, etc.
Most dev dependencies should be independent. e.g. one package can use jest, while another uses vitest.
Some dev dependencies can be considered as package dev dependencies or repo dependencies. These can/should be hoisted, or simply added at the root, especially when it has complicated resolution problems. As you might know, a prime example of this is eslint, and its plugins. There are other examples such as @types/* or any dev dependencies that is not pure.

by the way, pnpm resolution is not perfect and there are still problems causing duplicate packages (thus causing problems to tsc or webpack), especially around peerDependencies.

As for myself, I also try to avoid hoisting as much as possible. But I do encounter cases in my company around @types/* and they need to be hoisted. e.g. when a dependency has some @types/* in their dependencies and reuses those types.

evelant commented 1 year ago

pnpm will soon have a feature to keep dependencies in sync across multiple packages in a monorepo (and more) https://github.com/pnpm/rfcs/pull/1. A lot of thought and discussion has gone into the design and I think it's really solid so maybe something for bun to emulate.

As far as package management in bun goes, I think emulating pnpm in general would be a good choice. The patterns used by pnpm (content addressable store, "safe" no hoisting node_modules, etc) is overall safer, more correct, and eliminates a lot of problems common to yarn and npm. IMO keeping the old hoisted/flattened node_modules pattern around would be a step back when pnpm offers thoroughly proven solutions that could be emulated.

Bessonov commented 1 year ago

@unional I am undecided about devDependencies, but for tooling it seems to work: root/package.json:

...
    "devDependencies": {
        "typescript": "4.7.2"
    },
...

cd root/packages/mymodule
bun run tsc -v
Version 4.7.2

Even worse, bun put all modules into root/node_modules and there is no root/packages/mymodule/node_modules at all. So, it is always (dirty-)hoisted?

@evelant I agree that pnpm is superior in many areas, and I would love it if bun could adopt some of the great concepts from it.

unional commented 1 year ago

pnpm will soon have a feature to keep dependencies in sync across multiple packages in a monorepo (and more) pnpm/rfcs#1. A lot of thought and discussion has gone into the design and I think it's really solid so maybe something for bun to emulate.

Yeah, don't get me wrong, pnpm is great and I'm using it for all of my monorepos.

Even worse, bun put all modules into root/node_modules and there is no root/packages/mymodule/node_modules at all. So, it is always (dirty-)hoisted?

Um, that doesn't look right. e.g. you can have one package using react@16 and one with react@18. If everything are hoisted, for sure that will be broken.

unional commented 1 year ago

@unional I am undecided about devDependencies, but for tooling it seems to work: root/package.json:
...
  "devDependencies": {
      "typescript": "4.7.2"
  },
...
cd root/packages/mymodule
bun run tsc -v
Version 4.7.2
Even worse, bun put all modules into root/node_modules and there is no root/packages/mymodule/node_modules at all. So, it is always (dirty-)hoisted?

@evelant I agree that pnpm is superior in many areas, and I would love it if bun could adopt some of the great concepts from it.

A obvious one for me is that depcheck will fail when you use the tools in your scripts. Then you have to exclude them one by one.

o-alexandrov commented 1 year ago

Bun, please do NOT consider to emulate pnpm. npm is the most popular has been safer for me to use in tens of different projects in various requirements.

If Bun starts to emulate pnpm, there'll be just another magnitude of bugs and tasks to resolve. There's no stable feature in bun atm, so stability in at least one feature should be the priority.

Bessonov commented 1 year ago

@unional

Um, that doesn't look right. e.g. you can have one package using react@16 and one with react@18. If everything are hoisted, for sure that will be broken.

I always use conflict-free versions. I probably still don't fully understand the issue with hoisting, but never mind. (EDIT: of course, I am aware of the react@16 and react@18 conflict. I mean the issue that bun should allow hoisting above).

Not really a bun user yet, just tried it for the first time for https://github.com/Bessonov/monorepo-starter .

unional commented 1 year ago

Bun, please do NOT consider to emulate pnpm. npm is the most popular has been safer for me to use in tens of different projects in various requirements.

I would disagree. Not saying bun SHOULD emulate pnpm, but it should learn and adopt the implementation that makes the most sense.

The doppelganers issue from npm and yarn is a deal breaker for many packages, especially those using TypeScript.

o-alexandrov commented 1 year ago

It’s much easier to develop an identical to npm DX. No matter how you two call the non-existent in npm features (emulation, learning, adopting):

in the end, you suggest to complicate the solution with your premature optimization

Any complication brings more bugs to the already unstable bun. That’s why I strongly suggest to focus on replicating npm exactly. Once bun is stable, only then optimization and edge cases should be revisited.

websorokinweb commented 1 year ago

That's all fun and cool but I and other people with similar projects structure just will not use bun. No one is going to rewrite project to adapt package manager

johncarmack1984 commented 1 year ago

That's all fun and cool but I and other people with similar projects structure just will not use bun. No one is going to rewrite project to adapt package manager

hehe, yeah someone would have to be out of their mind to do that... haha... hehe... quietly makes repository private

jakeleventhal commented 1 year ago

pnpm will soon have a feature to keep dependencies in sync across multiple packages in a monorepo (and more) pnpm/rfcs#1. A lot of thought and discussion has gone into the design and I think it's really solid so maybe something for bun to emulate.

As far as package management in bun goes, I think emulating pnpm in general would be a good choice. The patterns used by pnpm (content addressable store, "safe" no hoisting node_modules, etc) is overall safer, more correct, and eliminates a lot of problems common to yarn and npm. IMO keeping the old hoisted/flattened node_modules pattern around would be a step back when pnpm offers thoroughly proven solutions that could be emulated.

https://github.com/oven-sh/bun/issues/4844

ysfaran commented 1 year ago

@Bessonov

@ysfaran @websorokinweb
What about dependency hoisting configuration? I am thinking about migrating from yarn to bun as my package manager. To make each workspaces really independent I use following confugration in yarn:
nmHoistingLimits: workspaces
nmMode: hardlinks-local
This basically means "Don't hoist dependencies above the workspace level, in case two workspaces use the same dependency@version use hardlinks". With this approach I am able to package every workspace independently in my pipeline and truly separate the packages file-wise from each other. That's a must for any bigger monorepo IMO. Otherwise dependency management gets really messy and cumbersome.
Can you please explain the use case? I have been using monorepos for years for library, frontend, and backend development, and I have never experienced this need.

And I am not sure what you mean by "making each workspace really independent". Typically, monorepos are used for the opposite purpose, to create dependencies between projects.

What I mean with independent is if you for example have two packages package-a and package-b and both use axios as a dependency. What you can do now is to only install axios for package-a (meaning it will only be part of package-a's package.jons) and still do import axios from "axios" in package-b without any error. This happens because by default every package manager is hoisting dependencies to the root of a project and the node resolution algorithm will also look in every parent node_modules folder until it reaches the root.

IMO this is pure madness and should not be possible at all. Only if the root package.json specifies those dependencies it should be possible (but even that is a bad practice IMO and should only be valid for binaries or when you write a library and want to be sure to use just on version of a package for a smaller bundle size). By setting a hoisting limit you are able to specifiy how far up dependencies are allowed to be hoisted.

But hoisting limits are not enough because it is very often the case that multiple packages use the same dependencies with the exact same version. To prevent data duplication and multiple downloads you can use links (soft- or hardlinks) to either the root node_modules of your project (local) or to a node_modules folder shared accross projects (global).

The main difference between softlink (symbolic links) and hardlinks resides in the fact that hardlinks point to an actual address on your hardrive while softlinks point to a file path. In other words: if you zip a hardlinked file it will actually zip the file contents, but if you zip a softlinked file it will just zip the path to a file, which is (if you didn't also zip it) not part of the resulting zip file. Now if you unzip that file on another machine (or other location) your softlink is most likely invalid.

This is why I previously said:

That's a must for any bigger monorepo

But you may ask why?

Some libraries/frameworks which have commands like nuxt build (SSR for Vue.js) require that dependencies are directly listed in the node_modules of your package. If you write a frontend with nuxt and want to integrate into a monorepo workspace, you are required to prevent hoisting. If you then want to package your build artifact in a zip or docker image your also required to use hardlinks, unless you want to unnecessarily also zip the root node_modules which contain more than you need.
If your monorepo consists of a lot microservices, frontends and shared packages, you need to care about pipeline performance and separation. So what we do in our pipeline is to separate our microservices into their own pipeline job and run them in parallel. For each job we then only install dependencies required for the specific microservice. In the end each microservice job create a build artifact (often a docker image) which can then be deployed and used for end-2-end testing. If you have some kind of change detecion in place you can even skip some jobs. So if just one microservice changed, only run the pipeline job for that microservice and for the rest, reuse the artifacts of the previous pipeline runs.
You are able to let different teams work on the same monorepo. If you have a team that is responsible for microservice-a and another team responsible for microservice-b, they should not have any dependency in the code to each other (they might still use a shared package, e.g. for tracing, which is why they reside in the same workspace). With hoisting you implicitly have this dependency, where a package installed microservice-a can be used in microservice-b. This can lead to very unexpected errors and complications accross teams. With hoisting limits and hardlinks you can use GitHub's Codeowner Feature very effectively too.

I hope this answers your question.

oven-sh / bun

Monorepo support #533

What is the problem this feature will solve?

What is the feature you are proposing to solve the problem?

What alternatives have you considered?