Nodejs subsytem: node_modules improvement

[hsjobeki]

I dont know what is exactly different between node2nix and dream2nix when it comes to generating the node_modules folder. But it seems they are different.

I just compared the

• npm native node_modules
• node2nix node_modules
• dream2nix node_modules

seems the folder structure differs from the native one quite a bit. (npm vs dream2nix)

Also we are having so many overrides regarding resolve for each specific package like

• webpack
• rollup
• @rollup/plugin-node-resolve
• enhanced-resolve
• @mattermost/webapp
• tabby

• future packages where no issues yet reported

I dont know the exact differences are and what the architectural considerations where behind those changes, or if they are unwillingly done. But when i compare; in nod2nix, webpack and rollup specifically just work. Without having weird issues and the need for patches.

Wouldnt it be better to have a switchor an enum how to generate the node_modules like:

native (like npm would, better compatibility) pure (like dream2nix does now, better isolation or whatever)

For now i dont really understand why dream2nix has developed its own way of generating node_modules and doesnt stick as closely to the npm way as possible. Because this has already introduced a lot of complexity and compatibility overrides.

[DavHau]

The reasons I went with this designs are:

• performance (node2nix is really slow because everything is built in one large derivation)
• incremental builds. Builds are more incremental if all dependencies are separated in their own derivations.
• overriding capabilities. In node2nix it's hard to fix the build of a nested dependency. One first has to figure out the location etc. Having individual derivations makes overriding simpler, I believe.

But you have a good point here. Maybe these optimizations just come at a too high cost in degraded compatibility.

I'm completely open to try out other strategies. Alternative build logic can easily be included by adding another builder to dream2nix. I'm currently focusing on other core elements of the framework. I'd be really happy if someone would provide a PR.

@wmertens has some open PRs with changes to the build logic. @aameen-tulip is also working on alternative build logic.

[aakropotkin]

If anyone who understands the dream internals could take some time to help me plug my builders and metadata scrapers in I'd be thrilled.

They're fast. Miles faster than node2nix and if you have a partial cache they're even faster than yarn or NPM.

[DavHau]

@aakropotkin Maybe the builders would be interesting, considering the issues with the current one. See my message in matrix. We could pair up to get this started.

[wmertens]

@aakropotkin how do you handle circular dependencies?

[aakropotkin]

I toposort to identify them first with a "lint" style phase. If there's a cycle I print a message griping about software design principles 😆, then I build them by merging them into a single derivation with multiple outputs.

In my package set there's two entries that make the multiple outputs appear like the respective packages, but using either adds both to your dependency list.

The good news is this is only necessary for cycles where a package has a "build" or gyp/pre install script. Cycles for other types of packages are benign and only matter at runtime which is already covered by normal dependency tracking

[wmertens]

@aakropotkin You might be interested in my builder code (need to expand) and also the circular dep detection.

I guess I'm making things harder by symlinking everything, because then the runtime dependencies need to be colocated in the store as well. I assume you're copying?

The state of my code is that I had it working, but needed to split into translator (for v2/v3) and builder, and I got a little stuck due to wanting to support projects and I lost steam.

[aakropotkin]

By default it tries to symlink but on failure it emits a message basically saying "try adding copy = true to see if that resolves your issue".

One important thing about either symlinks or copies is that running the build in the root directory can upset scoped packages that attempt to resolve packages with cycles. So moving to @foo/bar and then adding modules there can resolve most issues regardless of whether symlinks or copies were used. I want to highlight this because it seems like a minor detail but it had a huge effect for my Typescript and Jest usage. This isn't hard to do but very easy to overlook and I'll admit it took me over a week of debugging to finally find that this detail was causing most of my resolution issues.

[hsjobeki]

What if we just add a middle layer, that provides the same flat node_modules like npm. If a dependency changes, then only the thin layer needs to be rebuilt.

[aakropotkin]

That's a pretty good approach for small projects. Jest is the one that will break you unfortunately.

I do build everything in isolation though which accomplished the same performance benefits.

[DavHau]

What if we just add a middle layer, that provides the same flat node_modules like npm. If a dependency changes, then only the thin layer needs to be rebuilt.

Actually, this is somehow like we are currently doing it. All top level packages use installMethod=copy. That means that, the previously symlinked dependency tree is copied and flattened into a more npm native tree before the build starts. The behavior is probably not exactly like the one of NPM but should be pretty close.

This is currently done inside the derivation of the top-level package. We could split this logic off into its own derivation, so it won't have to be re-done on every change.

seems the folder structure differs from the native one quite a bit.

What are the differences, actually?

[hsjobeki]

Sorry for the late answer. Had to spent some time on investigation for this answer.

In a demo case tried to install wasm-loader

Steps:

• installing npm install wasm-loader
• running nix develop
• compare the node_modules of d2n to node_modules of npm

-> see dependency graph here

wasm-loader has only 2 direct dependencies:

wasm-loader/package.json:

"dependencies": {
    "loader-utils": "^1.1.0",
    "wasm-dce": "^1.0.0"
  },

If you resolve the whole dependency tree like I did it resolves to exactly 61 dependencies. node -e "console.log(Object.entries(require('./package-lock.json').dependencies).length)"

find . -maxdepth 1 -type d | wc -l let us discover: -> npm creates 39 folders directly inside the node_modules ->with 13 entries in .bin/ folder -> dream2nix creates 39 folders directly inside the node_modules ->with 14 entries in .bin/ folder (one extra for a link -> to ..

Also both node_modules containing exact same folders at the root level. (directly in node_modules/) so everything seems perfect.

But let's dig deeper, to explore if also the content of dependencies are the same:

let's look at the installed dependency node_modules/wasm-loader

in contrast to npm, d2n created a nested node_modules folder here.

node_modules/wasm-loader/node_modules

Okay. wasm-loader declared it depends on 2 dependencies: wasm-dce and loader-utils

But if we look into the nested node_modules folder of wasm-loader, we discover it contains 3 folders:

@ampproject (empty)
@babel/ (nothing except some .bin/* files)
@jridgewell (empty)

none of those are direct dependencies? if d2n meant to link binaries why did it create the empty folders for @ampproject and @jridgewell?