search

yarnpkg / yarn

The 1.x line is frozen - features and bugfixes now happen on https://github.com/yarnpkg/berry
https://classic.yarnpkg.com
Other
41.45k stars 2.73k forks source link

'yarn workspaces' break yarn2nix workflow #5965

Open qknight opened 6 years ago

qknight commented 6 years ago

feature request

i wish that yarn install --focus would pull in local dependencies which are not in the NPM registry opposed to what @bdwain pointed out in https://github.com/yarnpkg/yarn/issues/5864#issuecomment-391829662 this is not planned nor even wanted behaviour.

why?

my problem is that deploying yarn based node applications had really been a joy until workspaces emerged. so if yarn install --focus would also work with local dependencies, those which are not in the NPM registry, it might be as easy to support yarn deployment for 'workspaces based' projects as it had been for 'general yarn' based projects.

maybe we could call this feature --unifyNodeModules or --standalone

BUT maybe editoria devs are doing something out of the usual in developing their software using 'yarn workspaces' and also using 'yarn workspaces' to deploy them. are they?

in detail explanation

for a more detailed explanaition, what i require, please read this blog post: https://lastlog.de/blog/posts/nix-language-atlas-javascript.html

TL;DR; you guys really messed up declarative deployment with 'yarn workspaces' and i'm totally lost on how to cover such projects in nixpkgs now.

node2nix issue

i've also filled a issue at yarn2nix, see: https://github.com/moretea/yarn2nix/issues/57

bdwain commented 6 years ago

@qknight can you explain how you would want focused workspaces to work with local packages? The whole idea is to avoid building other packages when they have already been built and deployed somewhere else. I'm not sure I see what can be done when the package is local. If the package is already built locally, there's nothing extra that focused workspaces would add.

qknight commented 6 years ago

i'll try!

i want a yarn subcommand which builds a node_modules containing all the dependencies in one directory. this is in contrast to what we have now: a local subproject node_modules and a global node_modules and the loader basically resolves to the local or the global one. thus, ideally the local and global packages are merged into one node_packages.

for development yarn workspaces are really cool (not using them much but i see the limitaions without them) but for deployment they are too complex to handle for me.

editoria example

take the https://gitlab.coko.foundation/editoria/editoria project as an example, they do:

  1. a global yarn initially to build everything
  2. cd editoria/packages/editoria-app and yarn server (which basically starts a nodejs webserver and injects the app.js into it and thus could be done without using yarn at all)

i would so much love to only build a node_modules in editoria/packages/editoria-app/node_modules and then only rely on that subfolder for hosting this frontend. that means:

  1. mv editoria/packages/editoria-app /www
  2. rm -Rf editoria
qknight commented 6 years ago

checking out the https://yarnpkg.com/blog/2018/02/15/nohoist/ feature.

update: added this to the global package.json

  "workspaces": {
    "packages": ["packages/*"],
    "nohoist": ["**"]
  }

but then yarn failed:

yarn install v1.7.0
[1/4] Resolving packages...
[2/4] Fetching packages...
warning Pattern ["react-router-redux@next"] is trying to unpack in the same destination "/home/nixi/.cache/yarn/v1/npm-react-router-redux-5.0.0-alpha.9-825431516e0e6f1fd93b8807f6bd595e23ec3d10" as pattern ["react-router-redux@^5.0.0-alpha.9"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["babel-preset-es2015@latest"] is trying to unpack in the same destination "/home/nixi/.cache/yarn/v1/npm-babel-preset-es2015-6.24.1-d44050d6bc2c9feea702aaf38d727a0210538939" as pattern ["babel-preset-es2015@^6.24.1","babel-preset-es2015@^6.24.1","babel-preset-es2015@^6.24.1"]. This could result in non-deterministic behavior, skipping.
info fsevents@1.2.4: The platform "linux" is incompatible with this module.
info "fsevents@1.2.4" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > eslint-config-pubsweet@0.0.6" has incorrect peer dependency "prettier@1.8.2".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-bookbuilder > pubsweet-client@2.5.5" has unmet peer dependency "pubsweet-server@>=1.1.0".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-bookbuilder > enzyme@2.9.1" has incorrect peer dependency "react@0.13.x || 0.14.x || ^15.0.0-0 || 15.x".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-bookbuilder > react-addons-test-utils@15.6.2" has incorrect peer dependency "react-dom@^15.4.2".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-bookbuilder > react-test-renderer@15.6.2" has incorrect peer dependency "react@^15.6.2".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-bookbuilder > sinon-as-promised@4.0.3" has incorrect peer dependency "sinon@1".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-editoria-dashboard > pubsweet-client@2.5.3" has unmet peer dependency "pubsweet-server@>=1.1.0".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > Editoria > testcafe-react-selectors@1.1.0" has incorrect peer dependency "testcafe@^0.19.1".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > Editoria > winston-daily-rotate-file@3.2.1" has unmet peer dependency "winston@^2 <= 3.0.0-rc9 || ^3".
warning "workspace-aggregator-24ac56cc-c331-42d3-8299-e23e6aab4709 > pubsweet-component-bookbuilder > pubsweet-client > react-apollo@2.1.4" has unmet peer dependency "apollo-client@^2.2.3".
error An unexpected error occurred: "ENOENT: no such file or directory, lstat '/home/nixi/editoria/packages/dashboard/node_modules/cheerio/node_modules/lodash'".
info If you think this is a bug, please open a bug report with the information provided in "/home/nixi/editoria/packages/editoria-app/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

now what?

bdwain commented 6 years ago

@qknight yarn workspaces are not required. You can just have a monorepo without yarn workspaces. That's basically what nohoisting ** does anyway (minus the symlinks and the common lockfile). Have you tried just not setting workspaces at all at the root package.json (or just having no root package.json)?

Another option would be to have a build step that creates a small deployable artifact (yarn itself does this) rather than copying the whole node modules folder (which will also contain dev dependencies if you don't run install with the --production flag). Then you could build the deployable artifact to dist/ or something and not worry about the shape of the monorepo and just copy that.

Other than that, what you're asking for sounds related to #4207 (and kinda #4219 but not as much).

qknight commented 6 years ago

@bdwain i've tried what you suggested below.

and how to create that small deployable artifact? did not find any results with 'yarn' and 'dist/'...

not setting workspaces at all

yarn --ignore-engines
yarn install v1.7.0
info No lockfile found.
[1/4] Resolving packages...
warning @pubsweet/ui > enzyme > rst-selector-parser > nearley > nomnom@1.6.2: Package no longer supported. Contact support@npmjs.com for more info.
warning pubsweet-client > apollo-client-preset@1.0.8: for the best quick start with Apollo Client, please use apollo-boost instead
error An unexpected error occurred: "https://registry.yarnpkg.com/pubsweet-editoria-authsome: Not found".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/editoria/packages/editoria-app/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

removing the top level packge.json

yarn
yarn install v1.7.0
info No lockfile found.
[1/4] Resolving packages...
warning @pubsweet/ui > enzyme > rst-selector-parser > nearley > nomnom@1.6.2: Package no longer supported. Contact support@npmjs.com for more info.
warning pubsweet-client > apollo-client-preset@1.0.8: for the best quick start with Apollo Client, please use apollo-boost instead
error An unexpected error occurred: "https://registry.yarnpkg.com/pubsweet-editoria-authsome: Not found".
info If you think this is a bug, please open a bug report with the information provided in "/tmp/editoria/packages/editoria-app/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
bdwain commented 6 years ago

You won't be able to run a single yarn install anymore if you remove workspaces. You'll have to build and install each package separately.

If you have some packages that are not on NPM, you can change the dependencies of their sibling packages to point file://<PATH_TO_PACKAGE> or link://<PATH_TO_PACKAGE> and that should get rid of those errors.

qknight commented 6 years ago

i tried to fix the sibling path:

If you have some packages that are not on NPM, you can change the dependencies of their sibling packages to point file:// or link:// and that should get rid of those errors.

instead of file://../package1 it should be file:../package1

i've spent quite a while to find that out! but this is actually good to know, thank you!

but yarn install --focus --ignore-engines still won't work, see below

yarn install --ignore engines

from the project root:

yarn install  --ignore-engines
yarn install v1.7.0
[1/4] Resolving packages...
success Already up-to-date.
Done in 2.58s.
nixi@ubuntu-4gb-nbg1-1-editoria-docker:~/editoria$ rm -Rf node_modules/
nixi@ubuntu-4gb-nbg1-1-editoria-docker:~/editoria$ yarn install  --ignore-engines
yarn install v1.7.0
[1/4] Resolving packages...
[2/4] Fetching packages...
warning Pattern ["react-router-redux@next"] is trying to unpack in the same destination "/home/nixi/.cache/yarn/v1/npm-react-router-redux-5.0.0-alpha.9-825431516e0e6f1fd93b8807f6bd595e23ec3d10" as pattern ["react-router-redux@^5.0.0-alpha.9"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["babel-preset-es2015@latest"] is trying to unpack in the same destination "/home/nixi/.cache/yarn/v1/npm-babel-preset-es2015-6.24.1-d44050d6bc2c9feea702aaf38d727a0210538939" as pattern ["babel-preset-es2015@^6.24.1","babel-preset-es2015@^6.24.1","babel-preset-es2015@^6.24.1"]. This could result in non-deterministic behavior, skipping.
info fsevents@1.2.4: The platform "linux" is incompatible with this module.
info "fsevents@1.2.4" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > eslint-config-pubsweet@0.0.6" has incorrect peer dependency "prettier@1.8.2".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-bookbuilder > pubsweet-client@2.5.5" has unmet peer dependency "pubsweet-server@>=1.1.0".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-bookbuilder > enzyme@2.9.1" has incorrect peer dependency "react@0.13.x || 0.14.x || ^15.0.0-0 || 15.x".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-bookbuilder > react-addons-test-utils@15.6.2" has incorrect peer dependency "react-dom@^15.4.2".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-bookbuilder > react-test-renderer@15.6.2" has incorrect peer dependency "react@^15.6.2".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-bookbuilder > sinon-as-promised@4.0.3" has incorrect peer dependency "sinon@1".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-editoria-dashboard > pubsweet-client@2.5.3" has unmet peer dependency "pubsweet-server@>=1.1.0".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > editoria > testcafe-react-selectors@1.1.0" has incorrect peer dependency "testcafe@^0.19.1".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > editoria > winston-daily-rotate-file@3.2.1" has unmet peer dependency "winston@^2 <= 3.0.0-rc9 || ^3".
warning "workspace-aggregator-00514047-12b8-4f7c-9dd3-2b7eb10f4b9f > pubsweet-component-bookbuilder > pubsweet-client > react-apollo@2.1.4" has unmet peer dependency "apollo-client@^2.2.3".
[4/4] Building fresh packages...
Done in 56.33s.

yarn install --focus --ignore-engines

from the editoria/packages/editoria-app folder (a workspace)

yarn install --focus --ignore-engines
yarn install v1.7.0
[1/4] Resolving packages...
error Couldn't find package "editoria@1.2.0" required by "workspace-aggregator-c61a8779-34b2-49b5-a50a-c77a5e3bd071@1.0.0" on the "npm" registry.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: Couldn't find package "jest-environment-db@1.0.0" required by "workspace-aggregator-c61a8779-34b2-49b5-a50a-c77a5e3bd071@1.0.0" on the "npm" registry.
    at MessageError.ExtendableBuiltin (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:243:66)
    at new MessageError (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:272:123)
    at PackageRequest.<anonymous> (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:36403:17)
    at Generator.throw (<anonymous>)
    at step (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:98:30)
    at /home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:111:13
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
Error: Couldn't find package "pubsweet-editoria-authsome@0.0.1" required by "workspace-aggregator-c61a8779-34b2-49b5-a50a-c77a5e3bd071@1.0.0" on the "npm" registry.
    at MessageError.ExtendableBuiltin (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:243:66)
    at new MessageError (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:272:123)
    at PackageRequest.<anonymous> (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:36403:17)
    at Generator.throw (<anonymous>)
    at step (/home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:98:30)
    at /home/nixi/.nvm/versions/node/v8.9.4/lib/node_modules/yarn/lib/cli.js:111:13
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
bdwain commented 6 years ago

don't use --focus for this. You won't be able to since not all of the packages you need are in npm. You shouldn't need it anyway as long as the local ones are built.

qknight commented 6 years ago

oh wow, removing the top level package.json and builing worked! awesome!

qknight commented 6 years ago

@bdwain i've been playing with yarn2nix and the approch to remove the top-level package.json and then mv the yarn.lock to the workspace for local building. i'm not 100% sure what goes wrong but it seems that by doing so yarn wants to update the yarn.lock and wants to access the internet which is something we can't tolerate and thus it fails.

qknight commented 6 years ago

@bdwain i'd like to modify yarn to generate a normalized node_modules and this patch could be nixos specific, meaning we maintain that in nixpkgs.

could you guide me a little? i need

bdwain commented 6 years ago

@qknight sorry i have been pretty busy lately and haven't had a chance to respond. This sounds like a pretty big new feature though. I'm not sure I would be able to help much as I have mostly on worked on the focused workspaces feature and this doesn't seem to have much overlap with that.

Since it's not going to be a small change, I would recommend creating an RFC and discuss with the yarn maintainers what their thoughts are on the feature. It's best not to get too far into implementation details before going through that process.