Hello, Prebuilders

[zeke]

Found about this org via https://github.com/hypermodules/hyperamp/issues/194#issuecomment-324153380

I don't really know anything about C++, nan, n-api, node-pre-gyp, or any of the other jazz that goes into native node modules. I am, however, willing to help in an organized effort to improve the native module experience for end-users of node, such that they are scarcely aware of their very existence.

So... how do we get to native module utopia?

[zeke]

@juliangruber @ralphtheninja @mafintosh @ungoldman @bcomnes

[ralphtheninja]

@zeke

Hi! And thanks for spearheading this! :heart:

Some Background

Currently there are several different approaches as how to handle prebuilt binaries, node-pre-gyp and prebuild (I think there are others as well)

I believe @mafintosh started working on prebuild to have a simple(r) way of making prebuilt binaries. If you take a look at the node-pre-gyp README, it's rather massive and there's a lot of information to deal with in order to setup a system for your native module.

For this purpose I think simplicity is a key feature. It should be easy to set up and use.

We have used prebuild in leveldown for a while now and it has been working really well. Together with prebuild-ci, building and publishing your binaries is deferred to whatever ci system you're running, e.g. Travis, AppVeyor.

leveldown does this by invoking prebuild-ci during the test phase. prebuild-ci basically checks the version in package.json and compares it with the previous commit to see if anything should be done. If so, prebuild runs and uploads the binaries to github (uses ghreleases to create a new release), see leveldown releases.

Whenever the user installs a module that depends on a native module with prebuilt binaries, it can use prebuild-install to find the binary needed for the current system (based on os, node version etc). Some lore; prebuild used to do this, but this was refactored out to prebuild-install to avoid having a massive dependency just for downloading binaries. leveldown does that here. So if prebuild-install fails, it falls back to node-gyp building the binary.

[ralphtheninja]

cc @mathiask88 @lgeiger @piranna

[ralphtheninja]

@zeke Some projects using this setup https://www.npmjs.com/browse/depended/prebuild

Also, @mafintosh has been working on prebuildify which if I understand correctly is an attempt to actually bundle all prebuilts inside the npm tar ball. So whenever you download something from npm, you have all prebuilts already and just need to unpack them. This is cool because it's super fast.

Check out sodium-native for an example how this approach works.

I think this is the way we want to go, i.e. prebuildify will become the new prebuild (as renamed to prebuild and new major version).

@mafintosh Please correct me if I'm wrong here

[ralphtheninja]

Please let me know if anyone wants to be invited to the organization. I also think that everyone should be owner since it enables people and removes useless friction :)

[bcomnes]

This good info. Ty

[ungoldman]

🖖 I would be interested in helping. The model I'm most familiar with is electron-prebuilt, which is fantastic for the sake of avoiding building from source on every install.

Does tarballing every target distro and including that on npm mean incurring a significant download size increase for all users?

Is this preferable to a postinstall script which is sometimes gross and side-effecty?

Happy to experiment and come up with a good strategy, or to help recruit other projects into adopting prebuild. leveldb, node-sass, sqlite seem like big communities that would benefit from prebuilds a great deal. I look forward to a day when I don't see an install fail because of a weird node-gyp race condition.

[ralphtheninja]

Does tarballing every target distro and including that on npm mean incurring a significant download size increase for all users?

At first glance it might look significant. If you take a look at the binaries for leveldown@1.7.2, there are roughly 50 files and each is around 200kb, so it's like 10Mb (just rough calculations here).

It's worth noting here that npm caches modules, so in theory it should be really fast for consecutive downloads. Although we trade off some disk space for it.

But there's ways to reduce this total size (@mafintosh can pitch in on details here), if need to be. If you're writing a simple c++ layer yourself it might not be worth it, but in the sodium-native case there's huge gain by building libsodium as a shared lib and linking with it (as opposed to what leveldown does, which always compile all c++ code, both the c++ side for leveldown and the c++ leveldb lib by google).

[mathiask88]

Hey, nice to see some action regarding the org :) @juliangruber created an org month ago https://github.com/prebuild but there was no further progress. See https://github.com/mafintosh/prebuild/issues/158 Edit: Sry, @piranna created the org and @juliangruber moved his prebuild-ci ;)

@ralphtheninja there are even more projects using prebuild https://www.npmjs.com/browse/depended/prebuild does not list the projects where prebuild is a devDependency. Try https://www.npmjs.com/browse/depended/prebuild-install instead.

[piranna]

I'm using prebuild and prebuild-install for NodeOS prebuild images and also have done some useful tools like buildDependencies to download dependencies needed to build the module only if a prebuild image is not available, and buho to check for new versions of upstream C/C++ code on a CI server and generate the new versiones images.

[mathiask88]

Happy to experiment and come up with a good strategy, or to help recruit other projects into adopting prebuild. leveldb, node-sass, sqlite seem like big communities that would benefit from prebuilds a great deal. I look forward to a day when I don't see an install fail because of a weird node-gyp race condition.

@ungoldman I created a list of packages that depend on nan sorted by download count per months. Many of the big projects already have some kind of prebuild handling e.g. node-sass does the same prebuild and prebuild-install does but with an own implementation.

[zeke]

I created a list of packages that depend on nan sorted by download count

Ha! Me too.

https://github.com/nice-registry/native-modules#readme

[mathiask88]

Nice and I see your list is missing scoped packages (~400) as well. I didn't find a way to bring the npm rest api to list these. And you check the devDependencies :+1:

[zeke]

list is missing scoped packages

Ah thanks for noticing. I'm collecting scoped packages but was not rendering them in the README. The skimdb.npmjs.com service doesn't serve up scoped packages, but the new-ish replicate.npmjs.com service does. My package-stream module and the underlying changes-stream make it pretty easy to consume metadata from that new endpoint.

Working on an updated list now..

[lgeiger]

Thanks for getting the discussion started!

We've been using prebuild and prebuild-install in zeromq for some time now and it greatly reduced the number of installation issues we receive. 💯 We're using zeromq heavily in an Electron context with electron-builder and even inside an Atom package. As long as people omit the --build-from-source flag the experience has been pretty good so far (apart from the need to rebuild when working with Electron). prebuildify sounds like a good solution if one doesn't mind the extra package size.

A complaint we sometimes get is that prebuild-install doesn't behave well for people behind a proxy (I'm not behind a proxy so I can't easily reproduce).

Has anybody already tried N-API and will it be able to work across Node and Electron?

[mathiask88]

A complain we sometimes get is that prebuild-install doesn't behave well for people behind a proxy (I'm not behind a proxy so I can't easily reproduce).

Hmm.. that's odd, because prebuild-install respects the npm proxy settings and uses a tunnel-agent if a proxy is set. Maybe these people can open an issue next time?

[lgeiger]

Hmm.. that's odd, because prebuild-install respects the npm proxy settings and uses a tunnel-agent if a proxy is set. Maybe these people can open an issue next time?

Thinking about this again it may be the combination of Atom's package installer with prebuild-install that made it fail. I'll try to find the relevant issue. 👍

[mathiask88]

Oh I missed the N-API part.. I watched the N-API repo and IMHO it is far away from being ready for production use.

[MarshallOfSound]

Would definitely be keen to reach a better place in node / electron land for prebuilt modules. If we can come up with some kind of system that removes peoples dependencies on tools like electron-rebuild then that's a win in my book.

Quick thoughts on some discussions here, I'd personally prefer to avoid shiping the prebuilt binaries for native modules inside an NPM module, because a) it increases download time and b) it will increase things like Electron app size (unless users go through and delete the ones they don't need during packaging).

Optimum for me is just a ridiculously simple to use module that uploads binaries to GitHub releases and downloads them on install. This is basically prebuild just with insanely simple documentation and in my head, I see something that can automatically set it up for native module repositories (some kind of automated PR).

On the topic of that last idea, the main problem with native modules in the current node world is that tools like prebuild are awesome, but no one uses them 😢 . I myself am guilty of this with most of my native modules compiling at install not prebuilding on CI somewhere. We need to make it as easy as possible for people to start using this kind of thing, or it's not going to be picked up by the majority of developers.

[ralphtheninja]

@ungoldman I created a list of packages that depend on nan sorted by download count per months. Many of the big projects already have some kind of prebuild handling e.g. node-sass does the same prebuild and prebuild-install does but with an own implementation.

Would be nice to have that list with metadata on what type of system is being used. Could use some simple heuristics to guess from package.json. If prebuild is inside devDependencies we can tag that module as a prebuild module.

[ralphtheninja]

Loads of interesting input in this thread. Keep it up!

I was thinking about prebuild and prebuildify and it seems to me we really want some combination of the two. The advantage of prebuild is that you get all the prebuilt binaries separated (sharded) into multiple files, which is handy, but in turn it makes it problematic with things like electron, which is a nobrainer using prebuildify since the proper binary is just there and can be loaded directly after install, be it in node or in electron, it just works.

[zeke]

https://github.com/nice-registry/native-modules#readme now includes scoped modules.

[zeke]

Would be nice to have that list with metadata on what type of system is being used.

I turned native-modules into a require-able module (not published to npm):

git clone https://github.com/nice-registry/native-modules
cd native-modules
npm i
node

> const packages = require('.')

> packages.filter(pkg => pkg.dependencies && pkg.dependencies.nan).length
2565

> packages.filter(pkg => pkg.devDependencies && pkg.devDependencies.nan).length
95

> packages.filter(pkg => pkg.devDependencies && pkg.devDependencies.prebuild).length
34

Will packages that (dev)depend on prebuild necessarily also (dev)depend on nan?

[piranna]

Will packages that (dev)depend on prebuild necessarily also (dev)depend on nan?

Not necesarily if they use directly the v8 APIs (deprecated behaviour) or the new N-API.

[zeke]

I've updated native-modules to collect all modules that depend (or devDepend) on any of the following:

• nan
• node-pre-gyp
• prebuild
• prebuildify

And here are some new lists based on the data:

• top dependents of nan by dependent count
• top dependents of nan by daily download count
• all dependents of prebuild
• all dependents of prebuildify

[bcomnes]

Wooo! 👏👏👏

[piranna]

all dependents of prebuild

I don't find the NodeOS related ones on this list... :'-( Does it has the list a limit in downloads count or something?

[zeke]

@piranna it should be every dependent of prebuild. It looks like nodeos hasn't been published to npm in a long time. Is there another module that you'd expect to see in the list?

[zeke]

@piranna I just re-ran the collector. Let me know if your npm-published modules are still missing from the list.

[ralphtheninja]

There's a ##prebuild channel on freenode. Not much action going on, but still :)

[ralphtheninja]

prebuild moved https://github.com/home-prebuilders-association/prebuild

[ralphtheninja]

prebuild-install moved https://github.com/home-prebuilders-association/prebuild-install

[ralphtheninja]

Asked @juliangruber to move prebuild-ci as well https://github.com/prebuild/prebuild-ci/issues/6

[piranna]

@piranna it should be every dependent of prebuild. It looks like nodeos hasn't been published to npm in a long time. Is there another module that you'd expect to see in the list?

NodeOS itself is not published, but its layer projects are. I have review the list again and they are still not included. One of them is nodeos-initramfs, that has prebuild as a devDependency. On the other hand, maybe it would makes (more?) sense to check for having prebuild-install as `dependency, isn't it?

[ralphtheninja]

On the other hand, maybe it would makes (more?) sense to check for having prebuild-install as `dependency, isn't it?

We should probably check for both to cover projects that haven't upgraded to use prebuild-install rather than prebuild, so:

• check for prebuild-install as dependency
• check for prebuild as dependency
• check for prebuild as dev-dependency

[piranna]

check for prebuild-install as dependency check for prebuild as dependency check for prebuild as dev-dependency

I agree.

[zeke]

There was a prebuid typo in the collector. Now it's fixed and includes all the nodeos-* modules. https://github.com/nice-registry/native-modules/tree/master#all_dependents_of_prebuild

I've also updated it to includes all modules that somehowDependOn prebuild-install

[piranna]

There was a prebuid typo in the collector. Now it's fixed and includes all the nodeos-* modules. https://github.com/nice-registry/native-modules/tree/master#all_dependents_of_prebuild

That's great, thank you! :-D

[zeke]

I had a good laugh when I saw the "Home Prebuilders Association", but I think prebuild is a better org name. Since there's already some discussion here and repos are starting to move in, what do folks think about moving everything to this org, nixing the existing prebuild org, and renaming this org to prebuild?

[mafintosh]

As long as it maintains it's lego avatar

[MarshallOfSound]

Going off what I said before about making this as easy as possible for native module owners to set up prebuild with there native modules.

--> prebuild-this

It's a pretty simple CLI tool that:

• Sets up AppVeyor and TravisCI automagically for your repository
• Sets up the secure variable for your GitHub token
• Adds the AppVeyor / TravisCI config files to your repo
• Sets up prebuild and prebuild-install

The result of running it on your code looks something like this

[piranna]

--> prebuild-this

Nice! :-D

[billiegoose]

Oh hell yes! I need to be a part of this, can I be a party of this?

[zeke]

@wmhilton I sent you an invitation to join the org.

I've been busy this week, but my plan is to come up with an "action item" list for getting this org into a happy shape. Off the top of my head, that includes:

• [x] rename this repo to about, a de facto userland org convention
• [ ] consolidate the useful bits of this discussion into a new readme.md in this repo
• [x] move more modules to this org
• [x] remove the old @prebuild org
• [x] rename this org to prebuild

[ungoldman]

sounds good @zeke! thanks for staying on top of things. would you mind sending me an invite too?

[ungoldman]

Copying related conversation from #dat on freenode at @ralphtheninja's request:

ogd: this module takes up 10gb on npm https://www.npmjs.com/package/actor-sdk ogd: it broke my dl script causing timeouts ogd: each version is ~24mb, there are 411 versions ungoldman: hmm, there's a reason to consider not bundling builds for all distros in a npm release (re: https://github.com/home-prebuilders-association/discussion/issues/1#issuecomment-324208055) ungoldman: if we do that for a lot of prebuilt native modules that might incur some severe bloat across the registry at large once lots of versions are published

(unrelated chat history removed)

[piranna]

remove the old @prebuild org rename this org to prebuild

Great! :-D

ogd: this module takes up 10gb on npm https://www.npmjs.com/package/actor-sdk ogd: it broke my dl script causing timeouts ogd: each version is ~24mb, there are 411 versions ungoldman: hmm, there's a reason to consider not bundling builds for all distros in a npm release (re: #1 (comment)) ungoldman: if we do that for a lot of prebuilt native modules that might incur some severe bloat across the registry at large once lots of versions are published

Yes, that's the reason why we should still try to promote people to use Github to store standalone prebuild images instead of bundling them inside npm packages. This is easier because people don't need to configure the Github token, but can cause problems if it's enabled by default. Maybe still left Github as default and show a warning to use a flag to explicitly bundle the prebuild images?

[zeke]

@ungoldman it looks like someone invited you already. :)

See https://github.com/home-prebuilders-association

[billiegoose]

ogd: this module takes up 10gb on npm https://www.npmjs.com/package/actor-sdk ogd: it broke my dl script causing timeouts

Maybe we could even use WebTorrent to reduce the burden of distributing binaries? I made a thing last week that creates torrents from a Github release.

From a purely technical point of view, I think the total disk space requirements of prebuilding all the native modules would be within reason for individuals or companies to host mirrors. Even if they all take up 10gb, there's only ~3000 of them (according to zeke's list), so we're only talking ~30TB.

[piranna]

Maybe we could even use WebTorrent to reduce the burden of distributing binaries? I made a thing last week that creates torrents from a Github release.

This is f*cking cool! Thanks for the link! :-D