Level / leveldown

Pure C++ Node.js LevelDB binding. An abstract-leveldown compliant store.
MIT License
775 stars 177 forks source link

Using prebuildify instead of prebuild #482

Closed mafintosh closed 5 years ago

mafintosh commented 6 years ago

Hi. For some time now we've been experimenting with a new approach to prebuilds called prebuildify. It's trying to take the best from prebuild while solving some of it's core issues (full disclosure, I was the original author on prebuild)

The main difference between the two is that prebuildify bundles all builds inside the node_modules folder when you publish. This has a ton of benefits:

The only disadvantage is the tarball gets a bit bigger since it contains all the available prebuilds. This cost is greatly offset by a massive reduction in the production dependency chain though. In fact in the modules we have ported to prebuildify so far the install time is much faster because of the low dep count than it was to install the deps needed to download a prebuild from github.

Here is an example, https://github.com/sodium-friends/sodium-native

vweevers commented 6 years ago

The only disadvantage is the tarball gets a bit bigger since it contains all the available prebuilds.

Hmm, for modules like rocksdb that would be at least 32 MB: https://github.com/Level/rocksdb/releases/tag/v3.0.1

mafintosh commented 6 years ago

You can do a trick where you build the bulk of the module (all the non nan/node.js) stuff as a shared lib that all the other ones link against. greatly reduces the size. that's what we do in sodium.

For sodium all our builds are 16mb unzipped and ~5mb zipped together and it's still much faster to download them all in one code without deps (on most connections at least), especially when cached. We've heard no issues from users about this so I think most users don't even notice.

ralphtheninja commented 6 years ago

I'm not against this at all. I think it would work fine with leveldown but for rocksdb it might be more painful (which can continue to use prebuild/prebuild-install).

The only things that concern me is a) that npm won't crap on us for storing larger binaries and b) the build process. Right now it works really well to build binaries and publish new versions of leveldown (I'm sure it's not an issue, I just don't have the process clear in my head).

ralphtheninja commented 6 years ago

@juliangruber @rvagg @mcollina Any thoughts on this?

vweevers commented 6 years ago

Support changing node versions locally or using the same install between electron / node. All prebuilds are in the tarball and the correct one is simply picked on runtime.

That part is awesome!

mafintosh commented 6 years ago

@ralphtheninja my guess is that leveldown would be the size of sodium which npm has 0 issues with. i talked to someone at npm about it at some point and they liked the idea also at the time.

Just noting that this gives a massive added security benefit as this means the prebuilds will be in the npm shasum. If they do signed modules later we'll automatically get the benefit of that as well.

For the dev process we currently build all our stuff on travis using prebuildify-ci. That supports 32/64 linux, 64 mac and 32/64 windows using appveyor.

If it is easy I'd def consider using the shared lib approach. Also a future n-api based leveldown will greatly benefit from prebuildify as you'd only have to ship one prebuild per platform in the tarball :)

I blogged about it here as well if interested in learning more, https://www.nearform.com/blog/the-future-of-native-modules-in-node-js/

vweevers commented 6 years ago

@mafintosh what's the workflow like for bundling binaries for multiple platforms? Does someone (the person doing npm publish) have to download the binaries from Travis and Appveyor?

vweevers commented 6 years ago

Nevermind, just saw prebuildify-ci download 👍

juliangruber commented 6 years ago

I'm all for it! But likely can't contribute substantial amounts of work right now

vweevers commented 5 years ago

@ralphtheninja do you want to include this in v5.0.0-rc1?

ralphtheninja commented 5 years ago

@vweevers Yep I think it makes sense to do that.