out of memory for large dependency graphs

[colmbrady]

Hey,

Ive been running out of memory when trying to generate System config for large modules. Increasing the Node max mem is not really helping. Problem seems to be in JSPM-NPM.

Could I contribute a enhancement that only scans "dependencies" and "peerDependencies" and skips "devDependencies" ?? Will really cut down the tree walking!

I guess some command line arg could be used to determine when to ignore devDepends.

Interested?

[alexisvincent]

Yeah I would, not as much to solve the memory issue (want to find a more general solution), but still want to support only doing parts of trees.

The way I'd like to have this implemented is basically as a free transform / filter. Since we can then build a collection of these kind of tree manipulations.

Thanks!

[alexisvincent]

Can you also submit a broken test case PR to the repo under tests

[alexisvincent]

When does this issue occur?

[colmbrady]

https://github.com/colmbrady/systemjs-config-builder/tree/bug/large-npm-module/test

^ this module does not install on my local environment. Changing between different Node versions seems not to work.

Seems to occur in jspm-npm during augmentation.

[colmbrady]

I have been looking at filtering out devDependencies. A problem I am having is that it because the node_modules is the only source of truth - its hard to reconcile the package.json dependencies with whats actually installed. Eg. React ^15.3.0 could actually be React 15.4.0 in node_modules. So if I try to get rid of devDependencies, then I will never be able to find React 15.4.0 in the registry - as my key is 15.3.0.

From what I can see - its impossible to know what actual version of React was installed, without tracking that information during tree walking (by comparing package.json deps), using Yarn.lock or NPM shrinkwrap.

To get my spike working - I used the yarn.lock file to determine what the package.json version of React is Vs the installed version.

Do you know of any alternatives?

[alexisvincent]

I want to solve this problem generally to support "Fully support npm resolve algorithym (upper node_module and single dep node_modules)" (in the roadmap).

For the moment, you can support all use cases that this project currently does without modifying the tree and registry as follows.

Remember that the tree looks as follows:

[
   {
     name: 'react',
     version: '15.4.1',
     deps: <array like one above (recursive)>
   },
   ...
]

So it actually tracks the installed version.

You could use the semver package to reconcile deps in the package.json to those in the top level of the tree, based on it's version.

Does that make sense?

I have an idea to solve this more generally by using the systemjs trace api which I can also specify. But this should be good for now.

[colmbrady]

Yes, I see what you mean I think. Ive had a bit of time today to spike that. Its a first cut, but I think it's closer to what you were intending.

https://github.com/colmbrady/systemjs-config-builder/commit/af01ab8d2f9aac1818385b0cc794c34497645b68

I took the approach of creating a new registry and tree by including only dependencies and peerDependencies. I need to do some testing - buts it definitely resulted in the memory issue disappearing for my sample "large-npm" module.. Ill let you know how the testing goes.

Thanks alexis