[Bug?]: Nested workspaces not installing local dependencies

brentbahry commented 5 months ago

Self-service

[ ] I'd be willing to implement a fix

Describe the bug

I could just have a bug in my configuration, but nested yarn workspaces seem to result in incomplete dependency installs.

I'm in the process of setting up a meta repo (proteinjs) with git submodules (reflection, util). reflection and util are both workspaces themselves, and leverage yarn for their own build/publish pipelines.

When I run yarn install in proteinjs, dependencies seem to be mapped appropriately across the nested workspaces (verified with yarn workspaces list). However, when I look at the node_modules/ of packages, they seem to be missing local dependencies.

An example of an issue in my workspace hierarchy looks like this:

proteinjs/

package.json
.yarnrc.yml
packages/
- reflection/
  - package.json
  - .yarnrc.yml
  - packages/
  - reflection-build/
    - node_modules/ <-- missing local dependencies
- util/
  - package.json
  - .yarnrc.yml
  - packages/
  - ..
..

To reproduce

git clone --recurse-submodules https://github.com/proteinjs/proteinjs cd proteinjs yarn install yarn workspaces list -v --json

{"location":".","name":"proteinjs","workspaceDependencies":[],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection","name":"reflection","workspaceDependencies":[],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection/packages/node-typescript-parser","name":"typescript-parser","workspaceDependencies":[],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection/packages/reflection","name":"@proteinjs/reflection","workspaceDependencies":["packages/util/packages/common"],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection/packages/reflection-build","name":"@proteinjs/reflection-build","workspaceDependencies":["packages/reflection/packages/reflection","packages/util/packages/common","packages/util/packages/node","packages/reflection/packages/node-typescript-parser","packages/reflection/packages/reflection-build/test/examples/source-repository/a"],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection/packages/reflection-build/test/examples/parser","name":"@proteinjs/reflection-build-test","workspaceDependencies":[],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection/packages/reflection-build/test/examples/source-repository/a","name":"@proteinjs/reflection-build-test-a","workspaceDependencies":["packages/reflection/packages/reflection-build/test/examples/source-repository/b","packages/reflection/packages/reflection","packages/util/packages/common","packages/util/packages/node","packages/reflection/packages/node-typescript-parser"],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/reflection/packages/reflection-build/test/examples/source-repository/b","name":"@proteinjs/reflection-build-test-b","workspaceDependencies":["packages/reflection/packages/reflection","packages/util/packages/common","packages/util/packages/node","packages/reflection/packages/node-typescript-parser"],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/util","name":"util","workspaceDependencies":[],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/util/packages/common","name":"@proteinjs/util","workspaceDependencies":[],"mismatchedWorkspaceDependencies":[]}
{"location":"packages/util/packages/node","name":"@proteinjs/util-node","workspaceDependencies":["packages/util/packages/common"],"mismatchedWorkspaceDependencies":[]}

yarn workspaces foreach --all --topological --exclude typescript-parser run build

[@proteinjs/reflection-build-test-b]: Process started
[@proteinjs/reflection-build-test-b]: Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)
[@proteinjs/reflection-build-test-b]: 
[@proteinjs/reflection-build-test-b]: $ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] [--require #0] <scriptName> ...
[@proteinjs/reflection-build-test-b]: Process exited (exit code 1), completed in 0s 479ms
The command failed for workspaces that are depended upon by other workspaces; can't satisfy the dependency graph
Failed with errors in 0s 484ms

ls packages/reflection/packages/reflection-build/node_modules

@types      ts-jest     typescript

Environment

System:
    OS: macOS 14.4.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 20.3.1 - /private/var/folders/x7/2klqb_951nj1yxfyr60hgs080000gn/T/xfs-ee5086da/node
    Yarn: 4.1.1 - /private/var/folders/x7/2klqb_951nj1yxfyr60hgs080000gn/T/xfs-ee5086da/yarn
    npm: 9.7.1 - ~/.nvm/versions/node/v20.3.1/bin/npm

Additional context

No response

clemyan commented 2 months ago

Yarn does not currently support lockfiles in workspaces. See #1223

The yarn.lock inside proteinjs/packages/reflection splits the worktree and causes inconsistent views. When going down from proteinjs to find workspaces, it can find proteinjs/packages/reflection/packages/reflection-build/test/examples/source-repository/b. But when going up from that workspace (e.g. running a script in it), it will find the said yarn.lock and treat proteinjs/packages/reflection as the project root.

Once you remove the nested lockfiles, you will see yarn workspaces foreach --all --topological-dev --exclude typescript-parser run build runs successfully.

brentbahry commented 2 months ago

Thanks for the explanation, I appreciate the time you took to look into this.

Just as a small data point, it is a requirement for us to have lock files in sub workspaces as the meta workspace in question is a developer convenience and not part of our deployment pipelines. So we ended up moving away from yarn workspaces to solve the problem.

arcanis commented 2 months ago

Out of curiosity, what did you use instead?

brentbahry commented 2 months ago

Oh I just wrote some small reusable utilities that satisfied our use cases, working directly with npm instead of yarn.

https://github.com/proteinjs/build

yarnpkg / berry