qknight commented 6 years ago

i've discovered yarn workspaces in the project https://gitlab.coko.foundation/editoria/editoria/tree/master and i assume this is currently not handled by yarn2nix.

the problem is this:

editoria/package.json (with editoria/yarn.lock) can be easily transformed into a node_package created by yarn2nix
editoria/packages/editoria-app/package.json is a subdirectory/subproject which also uses the yarn.lock from the top-level but then generates a different node_modules output as it is based on a different package.json

i'm not sure how to cover this use-case with yarn2nix but hopefully only requires a little patch for yarn2nix. ideas?

qknight commented 6 years ago

interestingly:

Note that Workspaces don’t have their own yarn.lock files, and the root yarn.lock contains all the dependencies for all the Workspaces. When you want to change a dependency inside a Workspace, the root yarn.lock will be changed as well as the Workspace’s package.json.

so i tried to use the global yarn.lock and the yarn.nix, which both contain the webpack dependency to build a node_modules using yarn2nix. however, the produced node_modules from nix-build lacked the .bin/webpack as well as other webpack related directories which is already a difference compared to the outcome of yarn run statefully. bug?

still, the point is when using yarn (and not yarn2nix):

editoria/package.json + editoria/yarn.lock                       -> editoria/node_modules
editoria/packages/editoria-app/package.json + editoria/yarn.lock -> editoria/packages/editoria-app/node_modules

both of these two node_modules contain different stuff. this means we can't just convert the yarn.lock into a node_modules without taking packages.json into account. i don't know the implementation of yarn2nix yet, so maybe this is already taken into account but for now i assume that packages.json is only used to detect that the yarn.lock is outdated and needs a new yarn run.

update: above assumption is probably wrong. yarn2nix probably uses the package.json to also select the dependencies required.

qknight commented 6 years ago

top level package.json

the global editoria/package.json contains a workspace definition:

 "workspaces": [
    "packages/*"
  ]

nested package.json

editoria/packages/editoria-app/package.json does not have the workspaces definition but still uses the nested projects from the top-level workspace:

editoria/packages/editoria-app $ nix-build
warning: Nix search path entry '$HOME/.nix-defexpr/channels' does not exist, ignoring
these derivations will be built:
  /nix/store/9hari9gxg6kdqfvj7q86z144z3difngb-Editoria-modules-1.2.0.drv
  /nix/store/apb7v10zazdjjbhr0i25nm62y6gw0xx1-pubsweet.drv
building '/nix/store/9hari9gxg6kdqfvj7q86z144z3difngb-Editoria-modules-1.2.0.drv'...
configuring
building
yarn config v1.6.0
success Set "yarn-offline-mirror" to "/nix/store/0xksa6xjnzsnp5s3wj7xkz002if8wpb3-offline".
Done in 0.07s.
yarn install v1.6.0
[1/4] Resolving packages...
error Couldn't find any versions for "pubsweet-editoria-authsome" that matches "^0.0.1" in our cache (possible versions are ""). This is usually caused by a missing entry in the lockfile, running Yarn without the --offline flag may help fix this issue.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
builder for '/nix/store/9hari9gxg6kdqfvj7q86z144z3difngb-Editoria-modules-1.2.0.drv' failed with exit code 1
cannot build derivation '/nix/store/apb7v10zazdjjbhr0i25nm62y6gw0xx1-pubsweet.drv': 1 dependencies couldn't be built
error: build of '/nix/store/apb7v10zazdjjbhr0i25nm62y6gw0xx1-pubsweet.drv' failed

looking into https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/, this is how dependency resolving works:

Packages like diff, pretty-format and the symlink to jest-matcher-utils were hoisted to the root node_modules directory, making the installation faster and smaller. The package chalk however could not be moved to the root because the root already depends on a different, incompatible version of chalk.

...

If you run code inside the jest-diff Workspace, it will be able to resolve all its dependencies:

require(‘chalk’) resolves to ./node_modules/chalk require(‘diff’) resolves to ../../node_modules/diff require(‘pretty-format’) resolves to ../../node_modules/pretty-format require(‘jest-matcher-utils’) resolves to ../../node_modules/jest-matcher-utils that is a symlink to ../packages/jest-matcher-utils

which implies that we have to take the other node_modules folder into account also. but how?

workspace dependencies

turns out this was not enough as the workspace contains lots of nested libraries which have node_modules created for each also. so this node_module hierarchy basically is applied all over the place.

for now, this seems to be a far more complex issue then i expected it to be at first.

qknight commented 6 years ago

maybe this is the solution:

https://yarnpkg.com/lang/en/docs/cli/install/#toc-yarn-install-focus

Shallowly installs a packages’s sibling workspace dependencies underneath its node_modules folder. > This allows you to run that workspace without building the other workspaces it depends on. Must be run inside an individual workspace in a workspaces project. Can not be run in a non- workspaces project or at the root of a workspaces project.

update: it is not, see https://github.com/yarnpkg/yarn/issues/5864#issuecomment-391829662

qknight commented 6 years ago

i've been thinking to copy the node_modules path together as a post-yarn step using some code from https://github.com/ripeworks/workspace-pack

experiment manual copying

what i tried:

yarn --ignore-engines and then i have a global node_moduleseach of the subdirectories in packages/ also has an individual node_modules
- bookbuilder
- common
- dashboard
- editoria-app
- editoria-authsome
- jest-environment-db
- theme
- vivliostyle

next i removed all the other subprojects:

 rm -Rf bookbuilder  common  dashboard  editoria-authsome  jest-environment-db  theme  vivliostyle

i tried to start the server: cd editoria/packages/editoria-app/ and yarn server and i get:

yarn server
yarn run v1.7.0
$ pubsweet start
/tmp/editoria/node_modules/config/lib/config.js:948
    throw new Error("Cannot parse config file: '" + fullFilename + "': " + e3);
    ^

Error: Cannot parse config file: '/tmp/editoria/packages/editoria-app/config/default.js': Error: Cannot find module 'pubsweet-editoria-authsome'
    at Config.util.parseFile (/tmp/editoria/node_modules/config/lib/config.js:948:11)
    at /tmp/editoria/node_modules/config/lib/config.js:717:28
    at Array.forEach (<anonymous>)
    at /tmp/editoria/node_modules/config/lib/config.js:713:14
    at Array.forEach (<anonymous>)
    at Config.util.loadFileConfigs (/tmp/editoria/node_modules/config/lib/config.js:712:13)
    at new Config (/tmp/editoria/node_modules/config/lib/config.js:135:27)
    at Object.<anonymous> (/tmp/editoria/node_modules/config/lib/config.js:1782:31)
    at Module._compile (module.js:643:30)
    at Object.Module._extensions..js (module.js:654:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

summary

assembling a global node_packages for editoria/packages/editoria-app means we have to not only take the local node_modules and the top-level node_modules into account but also all the other subprojects which makes it rather complicated.

qknight commented 6 years ago

using a 'hack' i was able to generate a standalone node_modules as described https://github.com/yarnpkg/yarn/issues/5965#issuecomment-396880284 statefully (using yarn on the shell manually) but i wasn't able to adapt this hack into yarn2nix so far.

update: was able to 'update' a yarn.lock by removing all the bundled libraries:

ecdsa-sig-formatter@1.0.10:
  version "1.0.10"
  resolved "https://registry.yarnpkg.com/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.10.tgz#1c595000f04a8897dfb85000892a0f4c33af86c3"
  dependencies:
    safe-buffer "^5.0.1"

"editoria-common@file:packages/common":
  version "0.0.3"
  dependencies:
    react "^16.2.0"
    react-bootstrap "0.32.1"
    react-router-dom "4.2.2"

now i get:

yarn install v1.6.0
[1/4] Resolving packages...
error Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
builder for '/nix/store/6pm1fgaxg7yi748aahgkww2bqrhz8mrq-editoria-modules-1.2.0.drv' failed with exit code 1
cannot build derivation '/nix/store/sdyp8z5rng97khl4cj4gy73j8pd907fs-pubsweet.drv': 1 dependencies couldn't be built
error: build of '/nix/store/sdyp8z5rng97khl4cj4gy73j8pd907fs-pubsweet.drv' failed

nightkr commented 6 years ago

I've hacked in support for this as of https://github.com/teozkr/yarn2nix/tree/de4ddd1d117af93e8874d4dd4d27d97f70d83328. The basic idea is that we build each item in the workspace separately, and then yarn add the local path onto the dependee (overriding the intra-workspace dependency).

It's very hacky, breaks --frozen-lockfile (so you're not 100% guaranteed that the lockfile is up-to-date), and requires you to manually tell Nix about the dependency tree. On the other hand, it seems to correctly build Photo Garden's gateway component, and I'm working on porting the rest (see the .nix files in https://github.com/teozkr/yarn2nix/tree/de4ddd1d117af93e8874d4dd4d27d97f70d83328).

qknight commented 6 years ago

@teozkr been looking into what you did with #58 and #59 and it seems interesting for our work. yet i don't understand how to use your code. the test is a little help but i don't understand how to adapt it to my problem either.

a few questions:

how to select the primary 'workspace package' for which i want to build the node_modules folder (in my case editoria-app)? note: the main application is also a package, see packages/editoria-app/ where the package.json states:

"workspaces": [ "packages/*" ]
what does libexec/editoria-monorepo/deps/ mean? what are deps and how to use them? i assumed that i only needed node_modules and not deps
why use the directory name libexec and not, say share?

nightkr commented 6 years ago

@qknight: To build a package from the workspace you'll want some variant of:

(mkYarnWorkspace {
  src = ./.;
}).editoria-app

deps contains the package.jsons of all dependencies, so that we can cache the result of yarn install even though the package contents change.

I have no idea what the libexec stuff is supposed to do, that seems to have been added after my changes.

qknight commented 6 years ago

@teozkr awesome, got it built. thanks so much! need to verify that it is working still. but the correct command was:

(mkYarnWorkspace {
  src = ./.;
}).Editoria

nix-community / yarn2nix

yarn workspaces #57

top level package.json

nested package.json

workspace dependencies

experiment manual copying

summary