Shrinkwrap and size of dependencies

[millette]

I just used https://github.com/siddharthkp/cost-of-modules on my module, which depends on hapi-boom-decorators to determine why mine seemd so big. I though it was nano (ha, there's an ironic name ;-) but turns out your modules and children take up 70 MiB!

$ cd hapi-couchdb-login/
$ cost-of-modules
┌──────────────────────┬──────────────┬────────┐
│ name                 │ children     │ size   │
├──────────────────────┼──────────────┼────────┤
│ hapi-boom-decorators │ 59           │ 70.91M │
├──────────────────────┼──────────────┼────────┤
│ cloudant-nano        │ 99           │ 12.53M │
├──────────────────────┼──────────────┼────────┤
│ lodash               │ 0            │ 4.88M  │
├──────────────────────┼──────────────┼────────┤
│ hapi-auth-cookie     │ 8            │ 4.80M  │
├──────────────────────┼──────────────┼────────┤
│ boom                 │ 1            │ 0.31M  │
├──────────────────────┼──────────────┼────────┤
│ pify                 │ 0            │ 0.02M  │
├──────────────────────┼──────────────┼────────┤
│ 6 modules            │ 103 children │ 84.72M │
└──────────────────────┴──────────────┴────────┘

I'm not sure how shrinkwrap works, but I found this version from 2.0.1: https://github.com/brainsiq/hapi-boom-decorators/blob/beb86e10918e2a031e365322987c9d2089acb1d6/npm-shrinkwrap.json which is tiny compared to most other versions containing babel stuff and much more.

Am I imagining things, or could we shave of 65 MiB of dependencies here?

[brainsiq]

Youch, that's pretty big! Boom is the only production dependency this module has so I'm not sure there's anything that can be done as it's pretty critical!

Shrinkwrap can vary wildly even if nothing changes underneath so not sure the number of modules in there is indicative of anything. I'll have a look though as it could be interesting.

Might be worth installing boom directly and comparing the size difference as well.

[millette]

Yeah, I don't understand what's going on, looking at package.json.

If I clone your project and run cost-of-modules I get a tiny (expected)

┌───────────┬────────────┬───────┐
│ name      │ children   │ size  │
├───────────┼────────────┼───────┤
│ boom      │ 1          │ 0.12M │
├───────────┼────────────┼───────┤
│ 1 modules │ 1 children │ 0.12M │
└───────────┴────────────┴───────┘

If I run cost-of-modules --include-dev I get

┌──────────────────┬──────────────┬────────┐
│ name             │ children     │ size   │
├──────────────────┼──────────────┼────────┤
│ standard         │ 126          │ 41.50M │
├──────────────────┼──────────────┼────────┤
│ chokidar-cli     │ 73           │ 5.22M  │
├──────────────────┼──────────────┼────────┤
│ babel-traverse   │ 1            │ 5.03M  │
├──────────────────┼──────────────┼────────┤
│ hapi             │ 26           │ 2.25M  │
├──────────────────┼──────────────┼────────┤
│ mocha            │ 31           │ 1.98M  │
├──────────────────┼──────────────┼────────┤
│ chai             │ 4            │ 0.70M  │
├──────────────────┼──────────────┼────────┤
│ request          │ 1            │ 0.27M  │
├──────────────────┼──────────────┼────────┤
│ walkdir          │ 1            │ 0.14M  │
├──────────────────┼──────────────┼────────┤
│ read-installed   │ 1            │ 0.13M  │
├──────────────────┼──────────────┼────────┤
│ boom             │ 1            │ 0.12M  │
├──────────────────┼──────────────┼────────┤
│ underscore       │ 1            │ 0.11M  │
├──────────────────┼──────────────┼────────┤
│ babel-messages   │ 1            │ 0.03M  │
├──────────────────┼──────────────┼────────┤
│ babel-code-frame │ 1            │ 0.02M  │
├──────────────────┼──────────────┼────────┤
│ 13 modules       │ 248 children │ 49.90M │
└──────────────────┴──────────────┴────────┘

Which is also to be expected.

It's only when I clone https://github.com/millette/hapi-couchdb-login/ and run cost-of-modules do I see huge numbers for my dependency on hapi-boom-decorators.

Since I don't have experience with shrinkwrap and it's the only place I see lots of large dependencies, I expect the problem comes from there. There's no shrinkwrap in https://github.com/hapijs/boom either.

[millette]

Also notice in my first comment, boom appears in the table with a light 0.31M

[millette]

As a test, I forked your module and removed the shrinkwrap file.

When I ran cost-of-modules on my module using that fork, I got this result:

┌──────────────────────┬─────────────┬────────┐
│ name                 │ children    │ size   │
├──────────────────────┼─────────────┼────────┤
│ cloudant-nano        │ 99          │ 14.61M │
├──────────────────────┼─────────────┼────────┤
│ lodash               │ 0           │ 4.88M  │
├──────────────────────┼─────────────┼────────┤
│ hapi-auth-cookie     │ 8           │ 4.80M  │
├──────────────────────┼─────────────┼────────┤
│ boom                 │ 1           │ 0.31M  │
├──────────────────────┼─────────────┼────────┤
│ hapi-boom-decorators │ 0           │ 0.03M  │
├──────────────────────┼─────────────┼────────┤
│ pify                 │ 0           │ 0.02M  │
├──────────────────────┼─────────────┼────────┤
│ 6 modules            │ 79 children │ 20.77M │
└──────────────────────┴─────────────┴────────┘

Next, I'll try to regenerate the shrinkwrap file.

[brainsiq]

I get something that looks completely different if I install hapi and hapi-boom-decorators into a new project (excuse the formatting):

┌──────────────────────┬─────────────┬────────┐ │ name │ children │ size │ ├──────────────────────┼─────────────┼────────┤ │ hapi-boom-decorators │ 2 │ 23.06M │ ├──────────────────────┼─────────────┼────────┤ │ hapi │ 26 │ 1.56M │ ├──────────────────────┼─────────────┼────────┤ │ 2 modules │ 26 children │ 24.62M │ └──────────────────────┴─────────────┴────────┘

[millette]

With a brand new shrinkwrap file https://github.com/millette/hapi-boom-decorators/tree/new-shrinkwrap Using that branch in my module https://github.com/millette/hapi-couchdb-login/tree/new-shrinkwrap

┌──────────────────────┬─────────────┬────────┐
│ name                 │ children    │ size   │
├──────────────────────┼─────────────┼────────┤
│ cloudant-nano        │ 99          │ 14.61M │
├──────────────────────┼─────────────┼────────┤
│ lodash               │ 0           │ 4.88M  │
├──────────────────────┼─────────────┼────────┤
│ hapi-auth-cookie     │ 8           │ 4.80M  │
├──────────────────────┼─────────────┼────────┤
│ boom                 │ 1           │ 0.31M  │
├──────────────────────┼─────────────┼────────┤
│ hapi-boom-decorators │ 0           │ 0.04M  │
├──────────────────────┼─────────────┼────────┤
│ pify                 │ 0           │ 0.02M  │
├──────────────────────┼─────────────┼────────┤
│ 6 modules            │ 79 children │ 20.77M │
└──────────────────────┴─────────────┴────────┘

"Almost" the same as without the shrinkwrap file.

The new shrinkwrap file is very tiny (but I have no idea what to expect): https://github.com/millette/hapi-boom-decorators/blob/new-shrinkwrap/npm-shrinkwrap.json

[millette]

In other words, your original shrinkwrap files seems very polluted to me, but again, I have no idea what I'm talking about ;-)

NOTE: It might also have to do with https://twitter.com/eranhammer/status/804439129387716608 and my usage of yarn. But I still think there's something fishy with your shrinkwrap file.

[brainsiq]

Your new shrinkwrap file contains no dependency information so won't have the desired effect of locking down dependencies to the versions I tested with.

It looks like there's maybe an issue with my version of npm shrinkwrap. I removed node_modules, ran npm install --production and then shrinkwrap and it generated a shrinkwrap file containing only boom and it's one dependency (hoek). npm shrinkwrap is supposed to ignore dev dependencies by default but it looks like it might be including some of them (like eslint-config-standard).

Can you try pasting this new one into your shrinkwrap file and checking it makes a difference, before I publish it:

{
  "name": "hapi-boom-decorators",
  "version": "2.2.1",
  "dependencies": {
    "boom": {
      "version": "4.2.0",
      "from": "boom@>=4.2.0 <5.0.0",
      "resolved": "https://registry.npmjs.org/boom/-/boom-4.2.0.tgz"
    },
    "hoek": {
      "version": "4.1.0",
      "from": "hoek@>=4.0.0 <5.0.0",
      "resolved": "https://registry.npmjs.org/hoek/-/hoek-4.1.0.tgz"
    }
  }
}

[brainsiq]

I've used yarn with another project so can add a yarn.lock file too

[millette]

Much better! │ hapi-boom-decorators │ 2 │ 0.47M │

[brainsiq]

I think Eran's comment is just pointing out that yarn ignores npm shrinkwrap.

[millette]

I've been using hapi and yarn for a few months, haven't had any problems.

Thanks for clearing the shrinkwrap file :-) I'll sleep better héhé

[brainsiq]

I've just published 2.2.2. Thanks for raising this and helping sort it. Nice to know someone else is getting some use out of this module :+1:

[millette]

https://github.com/npm/npm/releases/tag/v4.0.1

Another fairly big change that we decided to slap into this version, since npm@4.0.0 is never going to be latest, is to make it so devDependencies are included in npm-shrinkwrap.json by default -- if you do not want this, use --production with npm shrinkwrap.

Maybe you upgraded from npm 3.x to 4.x and the default behavior changed.

[brainsiq]

I've got npm 3 :confused: