search

dominictarr / npmd

MIT License
450 stars 37 forks source link

How does npmd install --greedy work ? #79

Closed nidheeshdas closed 10 years ago

nidheeshdas commented 10 years ago

I have a large package.json with lots of packages each having their own large set of dependencies.

I did a npmd install --greedy expecting a flattened node_modules directory, but instead got a nested ones like npm install .

Does --greedy work only when installing single single package? Or does the --greedy mean something else?

Here's my npmd-config

{
  "0": "/usr/local/bin/npmd-config",
  "dbPath": "/Users/nidheeshdas/.npmd",
  "debug": false,
  "sync": false,
  "registry": "https://registry.npmjs.org",
  "cache": "/Users/nidheeshdas/.npm",
  "user-agent": "node/v0.10.28 darwin x64",
  "prefix": "/usr/local/Cellar/node/0.10.28",
  "port": 5656,
  "_": [],
  "global": null,
  "g": null,
  "greedy": null,
  "dedupe": null,
  "online": null,
  "offline": null,
  "save-dev": null,
  "D": null,
  "saveDev": null,
  "save": null,
  "S": null,
  "$0": "/usr/local/bin/npmd-config",
  "bin": "/Users/nidheeshdas/Sources/xxxxxxxxx/node_modules/.bin",
  "userAgent": "node/v0.10.28 darwin x64",
  "": [],
  "d": null,
  "s": null
}

PS: Is it possible to have a system where all npm packages are kept in a single large flattened directory at somewhere global, and then do a ln -s for each dependent package in each project that uses those packages? plus a db to track down used versions and remove unused ones?

dominictarr commented 10 years ago

can you do post the output of npmd resolve --greedy ?

I think when I implemented it I was mainly focusing on installing one at a time, but it could probably be fixed to install as flatly as possible.

As for links - Yes I also experimented with that - except that when linking you have to link all the dependencies, because when you link to a dep it will not resolve that dep's deps back into the linked directory. https://github.com/dominictarr/npmd-link so when you do it this way it has the branchyiest tree, instead of the flattest.

nidheeshdas commented 10 years ago

Here's my npmd resolve --greedy

https://gist.github.com/nidheeshdas/cd4e15b165799f1f00a7

dominictarr commented 10 years ago

okay so basically the resolve works like this: each module is resolved independently and then those trees are combined. This works well for trees, but not so well in greedy mode. https://github.com/dominictarr/npmd-resolve/blob/master/resolve-tree.js#L166-L181

Basically, in greedy mode this should combine the deps correctly...

Resolve takes a filter option https://github.com/dominictarr/npmd-resolve/blob/master/resolve-tree.js#L140-L150 that is used to implement resolveTreeGreedy. That should probably be used to merge those new things into the old tree... that could be used to flatten the tree into one directory.

nidheeshdas commented 10 years ago

ok, now I tried running npmd-resolve and npmd-install separately using --greedy npmd-resolve --greedy < package.json > resolved.json which produced a flatter tree and then I ran npmd-install --greedy < resolved.json I got the following error

/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/npmd-url/index.js:35
  throw new Error('loading modules from format:' + JSON.stringify(pkg) + 'not
        ^
Error: loading modules from format:{"hash":"dc967448757b278fe35fad25f47fa0a49bbdde19"}not yet implemented
    at module.exports (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/npmd-url/index.js:35:9)
    at cache.getter (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/index.js:56:17)
    at fetch (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/level-content-cache/index.js:108:7)
    at /usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/level-content-cache/index.js:79:9
    at dispatchError (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/levelup/lib/util.js:131:7)
    at /usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/levelup/lib/levelup.js:197:14
    at /usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/medeadown.js:125:5
    at Medea.get (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/node_modules/medea/medea.js:448:13)
    at MedeaDOWN._get (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/medeadown.js:118:11)
    at AbstractLevelDOWN.get (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/node_modules/abstract-leveldown/abstract-leveldown.js:62:17)

I checked the {"hash":"dc967448757b278fe35fad25f47fa0a49bbdde19"} and found it in

    "urlparser": {
      "name": "urlparser",
      "version": "0.3.9",
      "shasum": "dc967448757b278fe35fad25f47fa0a49bbdde19",
      "dependencies": {}
    },

Do I need to clear cache? Can I use regular npm install on this generated resolved.json ?

dominictarr commented 10 years ago

oh, sorry I broke it, reinstall npmd so that you get npmd-install@5.1.6 and it will work again!

nidheeshdas commented 10 years ago

npmd-resolve --greedy < package.json > resolved.package.json generated a json starting like this

{
  "name": "xxxxxxxxx",
  "version": "0.0.1",
  "dependencies": {
    "arangojs": {
      "name": "arangojs",
      "version": "1.0.0",
      "shasum": "e595921faeebbd1059da8835a48f549f67633223",
      "tarball": "http://registry.npmjs.org/arangojs/-/arangojs-1.0.0.tgz",
      "from": "*",
      "dependencies": {}
    },
    "micropromise": {
      "name": "micropromise",
      "version": "0.3.7",
      "shasum": "439b2158f847ea5e5d3d6f2bd2b45fc0a5a50477",
      "dependencies": {}
    },
.........
}

which npmd-install --greedy < resolved.package.json was not able to install complaining

/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/npmd-url/index.js:35
  throw new Error('loading modules from format:' + JSON.stringify(pkg) + 'not
        ^
Error: loading modules from format:{"key":"xxxxxxxxx@0.0.1"}not yet implemented
    at module.exports (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/npmd-url/index.js:35:9)
    at cache.getter (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/index.js:56:17)
    at fetch (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/level-content-cache/index.js:108:7)
    at /usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/level-content-cache/index.js:79:9
    at dispatchError (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/levelup/lib/util.js:131:7)
    at /usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/levelup/lib/levelup.js:197:14
    at /usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/medeadown.js:125:5
    at Medea.get (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/node_modules/medea/medea.js:448:13)
    at MedeaDOWN._get (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/medeadown.js:118:11)
    at AbstractLevelDOWN.get (/usr/local/lib/node_modules/npmd-install/node_modules/npmd-cache/node_modules/medeadown/node_modules/abstract-leveldown/abstract-leveldown.js:62:17)

but when i removed the package details like name, version and kept only the dependencies it worked. Now my node_modules folder is just 46 MB compared to 287 MB without npmd.

When I ran npmd-resolve --greedy > resolved.nopackage.json it generated a very large dependency tree compared to npmd-resolve --greedy < package.json > resolved.package.json I guess its the same tree but with no 'greediness'

So finally this is what worked for me.

  1. npmd-resolve --greedy < package.json > resolved.package.json
  2. Remove the top name and version and keep only the dependencies json
  3. npmd-install --greedy < resolved.package.json

Thank you,

I'm closing this issue.

nidheeshdas commented 10 years ago

ah! It didn't install the devDependencies.