Locking dependencies is slow

[cab404]

Running node2nix with package-lock.json takes too much time. It looks like fetching is not parallelized whatsoever.

Are there any opportunities to speed things up?

[svanderburg]

A partial answer: the way node2nix currently handles lock files is somewhat "flawed" in the sense that it consumes the entire lock file and generates a Nix expression from it capturing all dependencies and transitive dependencies.

I'm working on a new implementation that should dramatically simplify the integration process. The idea is to directly use a package lock file from a Nix expression and only generate the pieces that are missing, e.g. Git output hashes.

Most likely this will also speed up things a lot, but you have to patient for me to finish that rewrite/re-architecting process.

Another question: do you have a lot of git dependencies in your package-lock.json? for these dependencies, node2nix will download them to compute the git output hashes.

[cab404]

Thank you for your work on node2nix!

I am just curious -- is it impossible to reuse sha512 hashes from lockfile? It would seem like that Nix supports those.

All dependencies are from NPM:

[svanderburg]

yes, that's what my revised implementation will do -- it will reuse all hashes from the lock file, where possible. This makes the integration much "thinner" and a lot easier to maintain.

Basically, node2nix introduced its own variant of a lock file (node-packages.nix), before lock files existed in NPM (node2nix has a long history). Now that lock files are universally supported, and the Nix expression language has the means to work them (e.g. it can read JSON files and deal with SRI hashes), we should use that as much as possible.

[cab404]

e.g. it can read JSON files

Wow that thing's old! Any help needed?

[svanderburg]

Yes it was a decision made in 2014, and node2nix is still influenced by that decision.

The good news is that I already a local PoC implementation that I can use to deploy NPM projects, so most of the hard work is already done.

So the idea of my re-architecting effort is to make its design more modular. There will be separated pieces each taking a specific responsibility: fetching dependency metadata, performing NPM's dependency resolution algorithm, installing NPM dependencies in a build environment etc.

There will also be a library/package that specifically handles project deployments that I intend to publish in the next few weeks. Once this is available, it can be tested.

[cab404]

Thank you for thorough explanation!