search

JamieMason / shrinkpack

Fast, resilient, reproducible builds with npm install.
https://www.npmjs.com/package/shrinkpack
MIT License
793 stars 38 forks source link

Discussion: Approaches to bundle node-sass for offline installation? #38

Closed gajus closed 8 years ago

gajus commented 8 years ago

If my project has a dependency on https://www.npmjs.com/package/node-sass (or whatever else library with bindings to programs that require compiling), does shrinkpack bundle binaries associated with the package?

Does it bundle all sources or only sources specific to the platform used to prepare the shrinkpack (which would cause problems when developing on one OS and deploying to a different OS).

JamieMason commented 8 years ago

Hey @gajus, Is this what you mean? If I did this on OSX

mkdir test
cd test
npm init -y
npm i --save --save-exact node-sass
npm shrinkwrap
shrinkpack

Then cloned the "test" git repo on Windows and did an npm install?

I've not tried node-sass, but I know with phantomjs for example that the appropriate binaries would be installed according to the platform doing the installation.

It's the same tarball as a normal npm install uses, so any per-platform logic within that tarball should run the same as normal.

One thing that would likely be a problem is specifying optionalDependencies as described in [https://github.com/JamieMason/shrinkpack/issues/17#issuecomment-202340196]() but that (I think) is an issue that you'd get with npm shrinkwrap even if you didn't use shrinkpack.

Let me know if I'm on the right lines here, thanks a lot @gajus.

gajus commented 8 years ago

I've not tried node-sass, but I know with phantomjs for example that the appropriate binaries would be installed according to the platform doing the installation.

The thing is that node-sass package itself does not contain the binaries.

npm info node-sass
{ name: 'node-sass',
  description: 'Wrapper around libsass',
  'dist-tags': { latest: '3.5.1', beta: '3.5.0-beta.1', next: '3.5.1' },
  versions:
   [ '0.2.0',
     '0.2.1',
     '0.2.2',
     '0.2.3',
     '0.2.4',
     '0.2.5',
     '0.2.6',
     '0.3.0',
     '0.4.0',
     '0.4.1',
     '0.4.2',
     '0.4.3',
     '0.4.4',
     '0.5.0',
     '0.5.1',
     '0.5.2',
     '0.5.3',
     '0.5.4',
     '0.6.0',
     '0.6.1',
     '0.6.2',
     '0.6.3',
     '0.6.4',
     '0.6.5',
     '0.6.6',
     '0.6.7',
     '0.7.0-alpha',
     '0.7.0',
     '0.8.0',
     '0.8.1',
     '0.8.2',
     '0.8.3',
     '0.8.4',
     '0.8.5',
     '0.8.6',
     '0.9.0',
     '0.9.1',
     '0.9.2',
     '0.9.3',
     '0.9.4-rc1',
     '0.9.4',
     '0.9.5-rc1',
     '0.9.5',
     '0.9.6',
     '1.0.0',
     '1.0.1',
     '1.0.2-alpha',
     '1.0.2',
     '1.0.3',
     '1.1.0',
     '1.1.1',
     '1.1.2',
     '1.1.3',
     '1.1.4',
     '1.2.0',
     '1.2.1',
     '1.2.2',
     '1.2.3',
     '2.0.0-beta',
     '2.0.0',
     '2.0.1',
     '2.1.0',
     '2.1.1',
     '3.0.0-alpha.0',
     '3.0.0-beta.2',
     '3.0.0-beta.3',
     '3.0.0-beta.4',
     '3.0.0-beta.5',
     '3.0.0-beta.7',
     '3.0.0',
     '3.1.0',
     '3.1.1',
     '3.1.2',
     '3.2.0',
     '3.3.0',
     '3.3.1',
     '3.3.2',
     '3.3.3',
     '3.4.0-beta.2',
     '3.4.0-beta1',
     '3.4.0',
     '3.4.1',
     '3.4.2',
     '3.5.0-beta.1',
     '3.5.1' ],
  maintainers:
   [ 'am11 <adeelbm@outlook.com>',
     'andrewnez <andrewnez@gmail.com>',
     'deanmao <deanmao@gmail.com>',
     'keithamus <npm@keithcirkel.co.uk>',
     'laurentgoderre <laurent.goderre@gmail.com>',
     'saperski <npm@saper.info>',
     'xzyfer <xzyfer@gmail.com>' ],
  time:
   { modified: '2016-04-20T14:22:16.189Z',
     created: '2012-06-12T19:49:43.331Z',
     '0.2.0': '2012-06-12T19:49:45.950Z',
     '0.2.1': '2012-06-17T20:50:20.200Z',
     '0.2.2': '2012-07-08T17:28:18.030Z',
     '0.2.3': '2012-08-23T11:53:31.647Z',
     '0.2.4': '2012-08-26T20:16:39.035Z',
     '0.2.5': '2012-11-01T09:05:07.591Z',
     '0.2.6': '2013-01-12T10:53:26.399Z',
     '0.3.0': '2013-02-03T21:26:38.510Z',
     '0.4.0': '2013-03-28T17:10:12.248Z',
     '0.4.1': '2013-04-15T23:28:40.016Z',
     '0.4.2': '2013-04-19T12:49:01.523Z',
     '0.4.3': '2013-04-22T09:13:18.917Z',
     '0.4.4': '2013-04-27T22:27:05.192Z',
     '0.5.0': '2013-05-10T10:27:20.255Z',
     '0.5.1': '2013-05-29T07:41:59.970Z',
     '0.5.2': '2013-06-12T10:19:49.001Z',
     '0.5.3': '2013-07-08T08:08:26.682Z',
     '0.5.4': '2013-07-12T21:32:56.983Z',
     '0.6.0': '2013-07-30T20:23:32.043Z',
     '0.6.1': '2013-07-30T21:00:51.314Z',
     '0.6.2': '2013-08-01T21:25:56.657Z',
     '0.6.3': '2013-08-02T19:09:50.064Z',
     '0.6.4': '2013-08-08T19:39:40.917Z',
     '0.6.5': '2013-09-07T08:30:30.404Z',
     '0.6.6': '2013-09-13T16:31:16.896Z',
     '0.6.7': '2013-10-03T09:01:27.390Z',
     '0.7.0-alpha': '2013-10-19T15:13:39.524Z',
     '0.7.0': '2013-10-23T08:44:00.175Z',
     '0.8.0': '2014-01-10T23:56:39.818Z',
     '0.8.1': '2014-01-13T08:47:58.939Z',
     '0.8.2': '2014-03-12T17:35:26.604Z',
     '0.8.3': '2014-03-12T17:41:42.401Z',
     '0.8.4': '2014-03-26T20:03:04.280Z',
     '0.8.5': '2014-04-22T09:14:36.053Z',
     '0.8.6': '2014-04-22T13:08:42.143Z',
     '0.9.0': '2014-06-05T20:34:24.593Z',
     '0.9.1': '2014-06-06T10:28:02.828Z',
     '0.9.2': '2014-06-08T19:48:26.349Z',
     '0.9.3': '2014-06-16T09:26:38.455Z',
     '0.9.4-rc1': '2014-09-20T15:38:26.116Z',
     '0.9.4': '2014-09-21T20:24:00.776Z',
     '0.9.5-rc1': '2014-09-22T16:39:18.564Z',
     '0.9.5': '2014-10-05T11:01:24.538Z',
     '0.9.6': '2014-10-06T07:09:06.084Z',
     '1.0.0': '2014-10-16T20:03:11.849Z',
     '1.0.1': '2014-10-16T20:17:45.086Z',
     '1.0.2-alpha': '2014-10-24T20:49:20.903Z',
     '1.0.2': '2014-10-24T20:57:56.372Z',
     '1.0.3': '2014-10-24T21:42:54.595Z',
     '1.1.0': '2014-10-29T10:19:52.664Z',
     '1.1.1': '2014-10-29T11:44:43.001Z',
     '1.1.2': '2014-10-29T12:36:06.800Z',
     '1.1.3': '2014-10-29T12:47:17.351Z',
     '1.1.4': '2014-10-29T17:07:45.922Z',
     '1.2.0': '2014-11-06T10:30:58.159Z',
     '1.2.1': '2014-11-06T10:36:59.425Z',
     '1.2.2': '2014-11-06T18:51:52.447Z',
     '1.2.3': '2014-11-19T21:44:00.859Z',
     '2.0.0-beta': '2014-12-24T23:40:39.110Z',
     '2.0.0': '2015-02-12T04:04:23.708Z',
     '2.0.1': '2015-02-12T08:43:40.780Z',
     '3.0.0-preview': '2015-03-11T05:24:39.307Z',
     '3.0.0-pre': '2015-03-11T06:29:45.853Z',
     '3.0.0-alpha.0': '2015-03-16T02:17:47.712Z',
     '3.0.0-beta.1': '2015-03-24T21:46:30.571Z',
     '3.0.0-beta.2': '2015-03-25T21:50:29.267Z',
     '3.0.0-beta.3': '2015-03-27T07:45:41.739Z',
     '3.0.0-beta.4': '2015-03-27T10:08:00.495Z',
     '2.1.0': '2015-03-28T17:51:00.008Z',
     '2.1.1': '2015-03-28T18:16:08.690Z',
     '3.0.0-beta.5': '2015-04-06T10:18:06.735Z',
     '3.0.0-beta.6': '2015-04-22T00:40:34.419Z',
     '3.0.0-beta.7': '2015-04-22T01:19:32.029Z',
     '3.0.0': '2015-05-06T13:51:55.529Z',
     '3.1.0': '2015-05-13T12:20:32.248Z',
     '3.1.1': '2015-05-15T06:35:50.720Z',
     '3.1.2': '2015-05-21T02:57:01.566Z',
     '3.2.0': '2015-06-11T17:57:07.143Z',
     '3.3.0': '2015-08-26T02:47:27.877Z',
     '3.3.1': '2015-08-26T10:18:33.259Z',
     '3.3.2': '2015-08-30T04:21:09.834Z',
     '3.3.3': '2015-09-17T16:00:05.990Z',
     '3.4.0-beta1': '2015-09-18T06:59:15.656Z',
     '3.4.0-beta.2': '2015-10-16T09:30:14.525Z',
     '3.4.0': '2015-10-25T03:24:53.616Z',
     '3.4.1': '2015-10-27T01:55:26.915Z',
     '3.4.2': '2015-11-12T16:39:26.875Z',
     '3.5.0-beta.1': '2016-01-29T01:24:46.422Z',
     '3.5.0': '2016-04-20T14:08:44.304Z',
     '3.5.1': '2016-04-20T14:22:16.189Z' },
  author: 'Andrew Nesbitt <andrewnez@gmail.com> (http://andrew.github.com)',
  repository:
   { type: 'git',
     url: 'git+https://github.com/sass/node-sass.git' },
  users:
   { jimmyking: true,
     awaterma: true,
     nschonni: true,
     sebnitu: true,
     andrewnez: true,
     developit: true,
     paazmaya: true,
     octoo: true,
     charlespeters: true,
     jdhartley: true,
     shoobah: true,
     jeffboyus: true,
     travm: true,
     robbschiller: true,
     agtlucas: true,
     xavierharrell: true,
     febley: true,
     ppo: true,
     itonyyo: true,
     manny: true,
     andamira: true,
     yanndinendal: true,
     adambyrne: true,
     drewigg: true,
     'langri-sha': true,
     joaocunha: true,
     asawq2006: true,
     nano: true,
     subchen: true,
     hoitmort: true,
     orlenka: true,
     mjwilliams: true,
     raiseandfall: true,
     d4nyll: true,
     hongbora: true,
     cabrinha98: true,
     runningtalus: true,
     hideki_a: true,
     flftfqwxf: true,
     meabed: true,
     elidiazgt: true,
     levani: true,
     decoded: true,
     nicastelo: true,
     insdevmail: true,
     jimster305: true,
     ugarz: true,
     t1st3: true,
     cypark: true,
     davidnyhuis: true,
     hal9zillion: true,
     leopic: true,
     kontrax: true,
     kurre: true,
     jalcine: true,
     edloidas: true,
     rubiadias: true,
     alvint: true,
     flynntsc: true,
     afelicioni: true,
     realtimecat: true,
     jyounce: true,
     lmhs: true,
     abdihaikal: true,
     csbun: true,
     pandao: true,
     moonavw: true,
     sotosamper: true,
     clholzin: true,
     n370: true,
     jruif: true,
     crowbar: true,
     skullmasher: true,
     jaminroe: true,
     ericwbailey: true,
     vwal: true,
     noobapp: true,
     antixrist: true,
     takethefire: true,
     wkaifang: true,
     redstrike: true,
     schwartzman: true,
     doptrois: true,
     hex20dec: true,
     cfleschhut: true,
     demod: true,
     stryczek: true,
     cestrensem: true,
     anarcociclista: true,
     'thomas.miele': true,
     wiewioraz: true,
     bvmiko: true,
     grantcarthew: true,
     pql0: true,
     dralc: true,
     joannerpena: true,
     monjer: true,
     xiaowtz: true,
     abdul: true,
     crankincrankin: true,
     starknode: true,
     tekket: true,
     daam: true,
     adc39: true,
     dhampik: true,
     plyr: true,
     landy2014: true,
     stretchgz: true,
     nexdrew: true,
     princetoad: true },
  homepage: 'https://github.com/sass/node-sass',
  keywords: [ 'css', 'libsass', 'preprocessor', 'sass', 'scss', 'style' ],
  bugs: { url: 'https://github.com/sass/node-sass/issues' },
  readmeFilename: 'README.md',
  license: 'MIT',
  version: '3.5.1',
  libsass: '3.3.5',
  engines: { node: '>=0.10.0' },
  main: 'lib/index.js',
  nodeSassConfig: { binarySite: 'https://github.com/sass/node-sass/releases/download' },
  bin: { 'node-sass': 'bin/node-sass' },
  gypfile: true,
  scripts:
   { coverage: 'node scripts/coverage.js',
     install: 'node scripts/install.js',
     postinstall: 'node scripts/build.js',
     pretest: 'jshint bin lib scripts test',
     test: 'mocha test',
     build: 'node scripts/build.js --force',
     prepublish: 'not-in-install && node scripts/prepublish.js || in-install' },
  files: [ 'bin', 'binding.gyp', 'lib', 'scripts', 'src', 'vendor' ],
  dependencies:
   { 'async-foreach': '^0.1.3',
     chalk: '^1.1.1',
     'cross-spawn-async': '^2.1.9',
     gaze: '^1.0.0',
     'get-stdin': '^4.0.1',
     glob: '^7.0.3',
     got: '^5.5.0',
     meow: '^3.7.0',
     'in-publish': '^2.0.0',
     mkdirp: '^0.5.1',
     nan: '^2.2.0',
     'node-gyp': '^3.3.1',
     'sass-graph': '^2.1.1' },
  devDependencies:
   { coveralls: '^2.11.8',
     istanbul: '^0.4.2',
     jshint: '^2.9.1',
     mocha: '^2.4.5',
     'mocha-lcov-reporter': '^1.2.0',
     rimraf: '^2.5.2' },
  gitHead: 'd70721869ee7ff3168cdd0abd2f5404c7ed163c6',
  dist:
   { shasum: 'c6755887e5c06d8f7a1ec096330fa9651a989bb3',
     tarball: 'https://registry.npmjs.org/node-sass/-/node-sass-3.5.1.tgz' },
  directories: {} }

If you proceed to download https://registry.npmjs.org/node-sass/-/node-sass-3.5.1.tgz, you will see that it is just a script that chooses which binary to download from URL defined in package.json nodeSassConfig.binarySite.

The production environment in which I am working restricts any outgoing internet connection. Thus I am looking into solutions such as shrinkpack to deploy dependencies. node-sass is proving problematic.

Do you have any suggestions?

JamieMason commented 8 years ago

I see, node-sass would install correctly using shrinkpack under normal circumstances it seems then, but the internals of node-sass specifically dictate that an internet connection is a minimum requirement at install-time.

There are alternative approaches to this problem where caching proxies/mirrors are used, which might be better suited to you than shrinkpack. That said, they might not help either if, like shrinkpack, they just save npm's tarballs.

@glenjamin might be able to shed some light on this as he's a fan of the proxy/mirror approach. Glen, would mirrors like some of the ones you've mentioned only cache npm's tarballs? Or are they better still in that they cache all network activity that occurs during an installation?

Thanks all.

JamieMason commented 8 years ago

That said, you might still be left with the issue where James installs node-sass on Windows and a mirror records the network activity, but Laura runs the cached install on Linux and receives the wrong binary, or misses the cache and tries to hit the network for the Linux binary.

Depending on how these mirrors work, you might be able to prime the cache by running installs of node-sass first from every environment you want to support.

glenjamin commented 8 years ago

Hrm, good question. I've not tried to ensure that node-sass can install without an internet connection.

My usual approach is instead of running npm install on production machines, is to do the install on a CI machine and produce a tarball of the result.

The tarball is then what gets deployed and unpacked to the production machines.

webuniverseio commented 8 years ago

@gajus sounds like you only care about windows. If that is a case I think you can fork node-sass github repo, checkout on your local and run npm run build (or maybe postinstall) so that binaries are downloaded/compiled. Then commit binary together with other changes (I assume you'll want to remove postinstall script, since it is not needed anymore. Maybe something else...) to your forked repository and in package.json point node-sass to gajus/node-sass#tag-or-commit-id-or-branch-name. npm shrinkwrap will point to your github repo tarball with a binary and shrinkpack will download that tarball to local node_shrinkwrap folder. So everything should work like expected. I'm doing something similar in my project.

gajus commented 8 years ago

@szarouski I will use the advise. I will branch node-sass and release a version that includes all the binaries.

gajus commented 8 years ago

I don't think there is a reasonable way to address this within shrinkpack domain. While it would be possible to write custom logic specific to node-sass (for fetching all the binaries prior to packaging), it would be of little value as it does not appear as an approach shared between many libraries.

JamieMason commented 8 years ago

I think you're right @gajus, it is something specific to node-sass.

Thanks all for your help, really interesting issue.