Track nodejs progress

[DavHau]

Let's use this issue for tracking the progress on nodejs stuff.

Tanslators implemented:

• [x] yarn.lock translators
• [x] package-lock.json translator
• [x] package.json translator

• [x] support yarn v2

  Other tasks:

• [x] handle missing integrity
• [ ] extensively test on a number of nodejs projects

[DavHau]

I just stumbled across https://github.com/stephank/yarn-plugin-nixify Maybe we can get some inspiration from that. @happysalada

[wmertens]

Question/feature request: I recently started using direnv and it's great for development, but I wonder if direnv and dream2nix can't be joined into a single devShell + prod package provider?

Ideally, when I'm in a repo and direnv runs, dream2nix provides a devShell output that not only includes the correct runtime devDependencies in the shell, but also for the project. For Node.js that would mean replacing node_modules (with a symlink to the Nix store?).

That would mean that when you checkout another commit, it will automatically update all dependencies.

[DavHau]

@wmertens Could you check https://github.com/nix-community/dream2nix/pull/186 and let me know if that works for you?

[wmertens]

I added my test comments in that PR. Another issue: node_modules/.bin doesn't get created, and thus packages that provide a bin attribute in package.json don't provide binaries (e.g. eslint).

IMHO, ideally, the .bin directory would get populated and also added to the PATH.

[wmertens]

Found a big issue: peerDependencies don't work, for example in apollo-server-errors which can't find graphql because of the symlinking.

There are a couple of ways to solve this:

• treat peerDependencies as dependencies?
• copy modules instead of symlinking
• override require or provide a loader (ESM) that gets the list of modules and converts then to the correct nix store path, and wrap node with it

The latter has the advantage that you don't have to create new module copies just because their dependency versions are different between projects

UPDATE: The translator only looks at package-lock.json's dependencies, which strips the peerDependencies. It looks like it would be better to look at the packages attribute, which seems to contain the entire tree as it should be created and does retain the peerDependencies.

[wmertens]

Issue: the package-json translator uses the default npm package and not the one from the specified nodejs version

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/yarn-plugnplay-and-direnv-packaging/19759/19

[wmertens]

I think the correct translation of package-lock.json is by looking at the packages object, starting at the key "" for $pkg. Each $pkg has the keys dependencies and peerDependencies, which should be concatenated.

For each dependency $dep, use the string prefixes $pkg/node_modules/$dep down to node_modules to find the resolved dependency.

Then build the package as normally. Its node_modules will include symlinks to the peerDependencies, and its full set of dependencies determines its input hash, as usual.

Also keep track of dev: true in each $pkg, so that for the built package the devDependencies are left out, but not for the devShell.

[tgunnoe]

peerDependencies aren't resolved because they were removed a bit ago, because: https://github.com/nix-community/dream2nix/issues/117#issuecomment-1107644910

I'd use an injection temporarily

[wmertens]

Wouldn't the algorithm I lay out be working whether peer deps are auto-installed or not?

[wmertens]

I just verified with npm v8, it works as I explained. I'll try to make it work. Basically npm does all the heavy lifting of deciding which versions to use and all we need to do is provide each package with all the deps+peer deps it should be able to see according to npm.

This will make it work almost the same as npm, except that each module is in the store and symlinked, so any deps that it did not specify will not be visible to it. (which would be a bug in that module imho)

[DavHau]

Wouldn't the algorithm I lay out be working whether peer deps are auto-installed or not?

Your algorithm assumes lock file version >= 2, vs our current package-lock translator only works with lock file version <= 2. But anyways, I just checked and we install exactly the dependencies of the lock file, nothing else. So, if you have a lock file, you should not run into a missing peerDependencies issue.

But the problem you are likely running into, is that the build toolchain used by your package doesn't cope well with symlinked dependencies. You can try setting installMethod="copy" via an override on your failing package, which transforms all dependencies to actual copies.

There was still some problems with linked executables when using installMethod="copy", which I fixed in https://github.com/nix-community/dream2nix/pull/194

@wmertens It would be great if you could let me know if the suggestions together with #194 work for you.

[DavHau]

If you have any projects still failing because of peerDependencies, please open a github issue providing the project, so I can have a look myself.