Using prebuildify instead of prebuild

[mafintosh]

Hi. For some time now we've been experimenting with a new approach to prebuilds called prebuildify. It's trying to take the best from prebuild while solving some of it's core issues (full disclosure, I was the original author on prebuild)

The main difference between the two is that prebuildify bundles all builds inside the node_modules folder when you publish. This has a ton of benefits:

• No weird download prebuild issues - everything is available locally after a normal npm install
• No need for prebuild production deps - only a devDep is needed greatly reducing the dep chain
• Support changing node versions locally or using the same install between electron / node. All prebuilds are in the tarball and the correct one is simply picked on runtime.
• The module will work even if npm install scripts are disabled.

The only disadvantage is the tarball gets a bit bigger since it contains all the available prebuilds. This cost is greatly offset by a massive reduction in the production dependency chain though. In fact in the modules we have ported to prebuildify so far the install time is much faster because of the low dep count than it was to install the deps needed to download a prebuild from github.

Here is an example, https://github.com/sodium-friends/sodium-native

[vweevers]

The only disadvantage is the tarball gets a bit bigger since it contains all the available prebuilds.

Hmm, for modules like rocksdb that would be at least 32 MB: https://github.com/Level/rocksdb/releases/tag/v3.0.1

[mafintosh]

You can do a trick where you build the bulk of the module (all the non nan/node.js) stuff as a shared lib that all the other ones link against. greatly reduces the size. that's what we do in sodium.

For sodium all our builds are 16mb unzipped and ~5mb zipped together and it's still much faster to download them all in one code without deps (on most connections at least), especially when cached. We've heard no issues from users about this so I think most users don't even notice.

[ralphtheninja]

I'm not against this at all. I think it would work fine with leveldown but for rocksdb it might be more painful (which can continue to use prebuild/prebuild-install).

The only things that concern me is a) that npm won't crap on us for storing larger binaries and b) the build process. Right now it works really well to build binaries and publish new versions of leveldown (I'm sure it's not an issue, I just don't have the process clear in my head).

[ralphtheninja]

@juliangruber @rvagg @mcollina Any thoughts on this?

[vweevers]

Support changing node versions locally or using the same install between electron / node. All prebuilds are in the tarball and the correct one is simply picked on runtime.

That part is awesome!

[mafintosh]

@ralphtheninja my guess is that leveldown would be the size of sodium which npm has 0 issues with. i talked to someone at npm about it at some point and they liked the idea also at the time.

Just noting that this gives a massive added security benefit as this means the prebuilds will be in the npm shasum. If they do signed modules later we'll automatically get the benefit of that as well.

For the dev process we currently build all our stuff on travis using prebuildify-ci. That supports 32/64 linux, 64 mac and 32/64 windows using appveyor.

If it is easy I'd def consider using the shared lib approach. Also a future n-api based leveldown will greatly benefit from prebuildify as you'd only have to ship one prebuild per platform in the tarball :)

I blogged about it here as well if interested in learning more, https://www.nearform.com/blog/the-future-of-native-modules-in-node-js/

[vweevers]

@mafintosh what's the workflow like for bundling binaries for multiple platforms? Does someone (the person doing npm publish) have to download the binaries from Travis and Appveyor?

[vweevers]

Nevermind, just saw prebuildify-ci download 👍

[juliangruber]

I'm all for it! But likely can't contribute substantial amounts of work right now

[vweevers]

@ralphtheninja do you want to include this in v5.0.0-rc1?

[ralphtheninja]

@vweevers Yep I think it makes sense to do that.