Add support for npm5

[JamieMason]

npm install -g npm5 has a different cache structure so shrinkpack will need updating to support that, while being backwards compatible for older versions.

[JamieMason]

Update On Progress

1. Applied this patch from https://github.com/npm/npm/pull/19423 to my local installation of npm.
2. Stopped concatenating hashes in shrinkpack.
3. Tested it out locally against a project.
4. It works – npm installs using only npm-shrinkwrap.json and ./node_shrinkwrap and does not use the network.

Outstanding Problem

Caveat: This is pretty complicated and I don't know for certain, but this is how it seems to me – to start off the conversation.

https://github.com/JamieMason/shrinkpack#create-a-project-specific-cache-optional has been a long-standing problem which was possible to work around in older versions of npm, but so far it looks like this workaround no longer works with npm5.

When I'm back on my own machine I will create branches to reproduce from and open an issue over at npm/npm, but for now here is a fairly common use-case which could be affected:

1. Fork and clone a shrinkpacked repo (let's say that it depends on whatwg-fetch@2.0.3 as an example, just because that has no other dependencies).
2. npm install it, do some work, submit a PR etc.
3. When you're done, delete your local clone and fork.
4. Later, clone some other non-shrinkpacked project that also depends on whatwg-fetch@2.0.3.
5. Run npm install in this other non-shrinkpacked repo.
6. npm errors as it can't find /Usr/you/Dev/that-shrinkpacked-repo-from-that-pr-you-did-but-since-deleted/node_shrinkwrap/whatwg-fetch-2.0.3.tar.

The gist is that as npm install has previously been run against a shrinkpacked repo, from then on npm seems to expect that packages previously installed from the file system will still be there.

My instinct is that the local file system shouldn't be treated in the same way as registries are and should be considered unstable and subject to change without notice but @iarna and @zkat know way more than I do about all this, and I'm probably overlooking something.

Secondary problem

Handling git dependencies doesn't quite work yet, this is entirely down to shrinkpack and I'll fix it.

[timdp]

I don't want to hijack the conversation but how does this currently relate to npm ci (formerly known as cipm)?

[JamieMason]

@timdp can you expand on what you mean by how they might relate to each other?

References I know of between shrinkpack and cipm are this comment above when I first heard about cipm (as a note to self mainly) and that zkat mentions shrinkpack in this PR comment in the cipm repo.

[timdp]

I meant that they're basically trying to achieve the same thing: performing an npm install without network access--or at least that's how I understood it. Hence, I was wondering what the exact case for shrinkpack is versus the one for cipm.

[JamieMason]

If npm provide an officially supported form of shrinkpack that would be the ideal, and we'd have no need for shrinkpack.

I asked back in July 2017 whether there was any need for me to add support for npm5 and cipm might be what they had in mind back then but didn't have time yet to get started.

[timdp]

Okay, but now that cipm is there, is there effectively a difference between cipm with a project-local npm cache (so that you only get the dependencies you need) and shrinkpack?

[JamieMason]

I don't know @timdp but from reading https://www.npmjs.com/package/cipm my interpretation is that cipm isn't necessarily for installing offline but is a stricter, more deterministic npm – so would be better-suited to being used with caching proxies or offline mirrors.

If that's correct, then it seems to me that something like shrinkpack could a good fit for preparing an offline mirror which cipm could later install.

[JamieMason]

Opened Packages installed from "file:..." become default location for package/version pairs at https://github.com/npm/npm/issues/19621

[zkat]

@timdp npm can already do almost fully-offline installs if you have a package-lock.json and a warm cache. This happens -right now-, without any extra flags. The only exception right now is git packages, which need an extra layer of bespoke caching before I can do that with those (though the last real hurdle has been cleared on that).

As far as npm ci goes: @JamieMason's assessment of this is pretty much correct. It works the same as npm i, except it's a bit more strict about a few things and takes advantage of some optimizations specific to CI-like environments. It doesn't change anything major about the installer. You should still be able to use shrinkpack with it.

Thanks all for sticking through this and helping out. I know it's been a while, but I'm looking forward to telling people that they're able to do this, soon! 🎉

Edit: omg this would be a -fantastic- thing to announce in conjunction with the release of npm ci because they're pretty connected! People are gonna be so excited 😻

[JamieMason]

There have been some recent developments which can be followed in the #npm channel of package.community at https://discord.gg/RkxC4cA.

The summary is that @zkat has put together patches for shrinkpack, npm, pacote, and cacache to resolve the issues we've been seeing. This means support for shrinkpack is likely to land in one of the next two releases of npm, with shrinkpack 1.x being released around the same time.

[mobidev111]

npm v5.8.0 is out - with patches for shrinkpack

[JamieMason]

Away at the moment but aiming to work on this Sunday or Monday, thanks all. Almost finally there now I hope.

[JamieMason]

.tgz packages seem to work correctly in 5.8.0, they can be installed offline. .tar packages however have some issues and are hitting the network and warning of integrity issues.

The following output is when npm ci is run against the shrinkpack dev branch itself, when it has been bundled using shrinkpack .:

npm ci output

``` $ npm ci --loglevel http WARN pacote EINTEGRITY while extracting browserify-zlib@0.1.4 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/browserify-zlib-0.1.4.tar.You will have to recreate the file. WARN pacote EINTEGRITY while extracting @ladjs/time-require@0.1.4 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/ladjs-time-require-0.1.4.tar.You will have to recreate the file. WARN pacote EINTEGRITY while extracting pinkie-promise@1.0.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/pinkie-promise-1.0.0.tar.You will have to recreate the file. WARN pacote EINTEGRITY while extracting term-size@1.2.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/term-size-1.2.0.tar.You will have to recreate the file. WARN pacote EINTEGRITY while extracting is-promise@2.1.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/is-promise-2.1.0.tar.You will have to recreate the file. http fetch GET 304 https://registry.npmjs.org/browserify-zlib 929ms (from cache)0.2 -> /Users/foldleft/Dev/shrinkpack/node_modules/json-parse-better-errors http fetch GET 200 https://registry.npmjs.org/browserify-zlib/-/browserify-zlib-0.1.4.tgz 57ms (from cache)e_modules/estraverse WARN pacote EINTEGRITY while extracting symbol-observable@0.2.4 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/symbol-observable-0.2.4.tar.You will have to recreate the file. http fetch GET 304 https://registry.npmjs.org/symbol-observable 65ms (from cache)ldleft/Dev/shrinkpack/node_modules/meow/node_modules/strip-bom http fetch GET 304 https://registry.npmjs.org/pinkie-promise 547ms (from cache)foldleft/Dev/shrinkpack/node_modules/is-buffer http fetch GET 200 https://registry.npmjs.org/pinkie-promise/-/pinkie-promise-1.0.0.tgz 43ms (from cache)ode_modules/fill-range http fetch GET 200 https://registry.npmjs.org/symbol-observable/-/symbol-observable-0.2.4.tgz 152ms (from cache) http fetch GET 304 https://registry.npmjs.org/is-promise 564ms (from cache) -> /Users/foldleft/Dev/shrinkpack/node_modules/is-equal-shallow WARN pacote EINTEGRITY while extracting prettier@1.11.1 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/prettier-1.11.1.tar.You will have to recreate the file. WARN pacote EINTEGRITY while extracting pinkie@1.0.0 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/pinkie-1.0.0.tar.You will have to recreate the file. http fetch GET 200 https://registry.npmjs.org/is-promise/-/is-promise-2.1.0.tgz 72ms (from cache) http fetch GET 304 https://registry.npmjs.org/term-size 539ms (from cache)s/foldleft/Dev/shrinkpack/node_modules/for-in http fetch GET 200 https://registry.npmjs.org/term-size/-/term-size-1.2.0.tgz 17ms (from cache) http fetch GET 304 https://registry.npmjs.org/pinkie 69ms (from cache)Users/foldleft/Dev/shrinkpack/node_modules/for-in http fetch GET 200 https://registry.npmjs.org/pinkie/-/pinkie-1.0.0.tgz 28ms (from cache) WARN pacote EINTEGRITY while extracting path-parse@1.0.5 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/path-parse-1.0.5.tar.You will have to recreate the file. http fetch GET 304 https://registry.npmjs.org/path-parse 49ms (from cache).11.1 -> /Users/foldleft/Dev/shrinkpack/node_modules/regenerator-runtime http fetch GET 200 https://registry.npmjs.org/path-parse/-/path-parse-1.0.5.tgz 15ms (from cache) WARN pacote EINTEGRITY while extracting fsevents@1.1.3 from /Users/foldleft/Dev/shrinkpack/node_shrinkwrap/fsevents-1.1.3.tar.You will have to recreate the file. > fsevents@1.1.3 install /Users/foldleft/Dev/shrinkpack/node_modules/fsevents > node install [fsevents] Success: "/Users/foldleft/Dev/shrinkpack/node_modules/fsevents/lib/binding/Release/node-v57-darwin-x64/fse.node" already installed Pass --update-binary to reinstall or --build-from-source to recompile added 445 packages in 5.619s ```

If I were to have run this while offline, npm would hang during installation.

[zkat]

Are you sure both of the hashes are listed in integrity? Does it work if you use $ npx my-precious instead of shrinkpack? That might be a good reference for you.

[JamieMason]

Out of time for today but I have this branch https://github.com/JamieMason/shrinkpack/tree/npm5 which is a thin wrapper for my-precious. All seems to work as expected but I've yet to get the e2e tests passing. Once I can figure this out I'll have more confidence to release it.

EDIT: Apologies for the email spam subscribers probably got thanks to my repeated pushes to Travis ☝️

[peteward44]

Any progress on this? I don't mind having a look if you're not able to work on it

[JamieMason]

Thanks @peteward44 I appreciate the offer.

I have to grudgingly admit that this issue has worn me down and by now I really struggle to find the energy and motivation to get it through. Kat and Rebecca from npm have been amazing and we couldn't ask any more from them, but in two days this issue will be a year old. That's a long slog for one person and over that time each fix has uncovered new issues which took time to resolve before continuing on again.

The under-development version of shrinkpack works, at least on paper, the overall ongoing issue has been around gaining support in npm >=5 for fully offline installations using only a lockfile containing file: paths, as was the case in npm 3 and 4. Again, the npm devs are swamped and it's no simple task.

There is some good news though that npm have announced they will be:

Integrating the functionality of shrinkpack into npm directly. This would allow for entirely offline deploys even without a cache. It will also let you deploy with git dependencies without having git installed on your production machines. – https://blog.npmjs.org/post/173239798780/beyond-npm6-the-future-of-the-npm-cli

In light of this, I think it would be better to wait until npm archive is released in npm@7. No third party tooling (shrinkpack) will be needed and it's also ridiculously fast compared to shrinkpack.

Sorry everyone for such a long wait, but as a result we will get npm archive so it will be worth it.

[peteward44]

OK fair enough. A shame as shrinkpack is a great tool, but native support for offline installs does make a lot more sense.

[mobidev111]

npm taking over is definitely good - giving it a future as first class citizen.

This probably only arrives in half a year or so. In the blog post they state "late summer and fall of 2018" and given they're loaded with work one could guess that the timeline for this feature slips further.

I would welcome a quicker (interim) solution - and it seems we are already very close with shrinkpack working at the core.

For folks who want to help and take over your excellent work to get it finally out of the door - what would be the next steps to be taken?

To me, the next steps look like:

• set up & fix tests in the npm5 branch - any new ones?
• Report the progress on tests, so that others can chime on problems
• Identify potentially further problems in npm
• create a pre-release version to gather feedback from users

[zkat]

@mobidev111 the native npm version is pretty much already implemented and mostly needs attention and polish. You can try it right now with $ npx npmc archive. Regular npm install as of npm@6 should be able to install off that archive. If you want to add/remove packages, though, you'll need to use $ npx npmc add <pkg> and $ npx npmc rm <pkg> so edits to package-lock.json happen correctly.

[zkat]

The reason it isn't shipped is because I need to take it over the finish line, make sure corner cases are covered, add tests, etc -- but I won't have time allocated to do this for the next couple of months, probably.

[zkat]

(p.s. $ npx npmc unarchive will return your repo to its original pre-archive format)

[peteward44]

Great news @zkat. I'm not brave enough to use this in my production tool chain just yet, but i look forward to the official release :+1: We've been stuck on npm 4 due to this for a long time now and really can't wait to upgrade

[JamieMason]

I have published npm install -g shrinkpack@next which contains the current npm5 branch as 1.0.0-alpha: https://github.com/JamieMason/shrinkpack/blob/npm5/CHANGELOG.md.

It is a thin wrapper for my-precious rather than the ongoing work I did in the dev branch, I expect there may be issues but please let us know how it goes for you 👍

Thanks all for your comments, @mobidev111 thanks a lot for that useful comment and organisation.

[behoyh]

Hi @JamieMason,

Thanks for all your work, I know firsthand how building an open source project can be disorienting and tiring, and I just wanted to say that your efforts are very much appreciated and needed!

Congrats on getting your code into npm archive!

[BogdanDumitruMarian]

Hi. @JamieMason I'm getting the infamous

[ERROR] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: aggregate error [ERROR] [ERROR] npm ERR! aggregate error

when running npm install

I'm using node v9.11.1 / npm 5.6.0 and shrinkpack 1 - alpha. Funky enough, it only happens on the Jenkins Linux machine. I've tried running npm i against the archived-packages locally, with the internet connection turned off, and no problem whatsoever occured.

[themoonrat]

https://github.com/npm/cli/pull/1 got closed down :( Looks like yarn is the only solution, other than staying with vastly slower npm v4

[zkat]

@themoonrat we're looking into ways to reintegrate this and see where it fits into our product plans for the CLI. In the meantime, the core support for this kind of thing did land in npm. Any tool that does the same kind of rewriting should provide similar support. This means you can use $ npx my-precious or shrinkpack can be updated to do things with libprecious itself, and it should be way faster, and npm will do installations off the local tarballs very very quickly. :) I'm also still around to help integrate things if shrinkpack is interested in moving forward with this while we figure out how and whether to land this in npm proper.

[thomashardwick]

So it's been while. I'm still using shrinkpack and npm@4 since it still seems to be the only way to do fully offline installs without zipping up the node_modules folder but it is steadily becoming problematic to be on such an old version. Has there been any movement on getting npm able to install on an offline air-gapped machine or with shrinkpack using newer versions of npm?

[JamieMason]

Hey @thomashardwick, check out https://github.com/JamieMason/shrinkpack/issues/83#issuecomment-386340937 as I think it's still about right as a summary of where shrinkpack is at. Quite some time has passed though and I've lost touch with what npm provides out of the box in this area, there was talk of an npm archive command but again I don't know the latest.

[rockerest]

@thomashardwick I know this is a very different solution, but have you tried something like https://www.snowpack.dev/ or my own much simpler (better, in my opinion) https://www.npmjs.com/package/@thomasrandolph/icepack?

In both cases, instead of saving a local copy of the full dependency tree, those tools build a bundled copy of the appropriate dependencies. You would do that once, while you have an internet connection. Then, all future references to the dependency just reference the bundled file. One key part of both of these tools is the bundled version is always browser-safe, too, so you can run either tool ahead of time and then you have a set of dependencies that you never have to touch again, including compiling (if you ship ES6 modules to the browser).

Again, I know this is a really big deviation from shrinkpack, but I've found it's been extremely helpful to distance myself as far as possible from the npm ecosystem.

[peteward44]

I've since changed to use Yarn and it's offline mirror, which works pretty much like shrinkpack does.

https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/

It would seem NPM has lost interest in allowing users to offline dependencies as that goes against it's business model. A shame

[JamieMason]

Keep an eye on https://github.com/yarnpkg/yarn/issues/541 @peteward44 as Yarn's offline mirror can gradually harm the performance of your Git repo over time.

[iarna]

I've got no idea what npm's plans are these days, but they do have an active rfc process going, and it may be useful to bring this up there again.

[JamieMason]

shrinkpack@0.19.0 has been released which adds support for npm 7 and up. Thanks again to Kat and Rebecca for all of your help during your time at npm ❤️ .

[behoyh]

This is amazing