precompiled binaries?

[Pomax]

In the spirit of sqlite3 moving to precompiled binaries, only using node-gyp compiling when no binaries are found for the platform npm install is called on, I'd like to request the same approach for bcrypt, so that we don't have to worry about having the correct headers and c++ stack on windows or OSX (not everyone has a full VC++ stack or XCode, and with precompiled binaries, no one would need those either =)

[defunctzombie]

Does npm support this currently? Can you point to an example of how to do it? I will 100% not agree to committing binaries into the repo (if that is the only way it can be done right now)

[Pomax]

I don't know if it's baked in, sqlite3 has the binaries on AWS and simply checks if the binary is hosted, if not, it compiled through node-gyp.

https://github.com/developmentseed/node-sqlite3/blob/master/lib/binary_name.js does the magic, but it might be worth asking @springmeyer whether they're using a universal solution (and I 100% agree, checking binaries into the repo would be a tremendously bad idea)

[defunctzombie]

I applaud their creativity but I would wait on an official solution from npm on this one. It is not going to scale for each module to come up with some partially working way of doing this.

I also suspect some fun incompatibility at the libc level on older systems. Right now the requirements to compile are not that stringent (and I must admit I don't have a lot of love for windows folks as their lives are made hard by their own doing ;)

Happy to entertain solutions once npm decides on how this is handled at the binary module level.

On Tue, Oct 8, 2013 at 9:31 PM, Mike Kamermans notifications@github.comwrote:

I don't know if it's baked in, sqlite3 has the binaries on AWS and simply checks if the binary is hosted, if not, it compiled through node-gyp.

https://github.com/developmentseed/node-sqlite3/blob/master/lib/binary_name.jsdoes the magic, but it might be worth asking @springmeyer https://github.com/springmeyer whether they're using a universal solution (and I 100% agree, checking binaries into the repo would be a tremendously bad idea)

— Reply to this email directly or view it on GitHubhttps://github.com/ncb000gt/node.bcrypt.js/issues/186#issuecomment-25941004 .

[defunctzombie]

I should add the above opinions are mine and do not reflect other maintainers of this project so I could be overruled!

On Tue, Oct 8, 2013 at 9:33 PM, Roman Shtylman shtylman@gmail.com wrote:

I applaud their creativity but I would wait on an official solution from npm on this one. It is not going to scale for each module to come up with some partially working way of doing this.

I also suspect some fun incompatibility at the libc level on older systems. Right now the requirements to compile are not that stringent (and I must admit I don't have a lot of love for windows folks as their lives are made hard by their own doing ;)

Happy to entertain solutions once npm decides on how this is handled at the binary module level.

On Tue, Oct 8, 2013 at 9:31 PM, Mike Kamermans notifications@github.comwrote:

I don't know if it's baked in, sqlite3 has the binaries on AWS and simply checks if the binary is hosted, if not, it compiled through node-gyp.

https://github.com/developmentseed/node-sqlite3/blob/master/lib/binary_name.jsdoes the magic, but it might be worth asking @springmeyer https://github.com/springmeyer whether they're using a universal solution (and I 100% agree, checking binaries into the repo would be a tremendously bad idea)

— Reply to this email directly or view it on GitHubhttps://github.com/ncb000gt/node.bcrypt.js/issues/186#issuecomment-25941004 .

[Pomax]

Fair enough, it should be interesting to see if @isaacs et al have given this any thought yet.

Although, I should equally point out that working on windows is only made hard because of that attitude. If everyone operated based on the fact that their fellow devs might not have the luxury of using your favourite OS, or need to simply support all three mainstream operating systems, windows would stop being "hard to work on" because we stopped forcing them into that ;)

[Pomax]

(I run into this problem fairly often as part of opening projects up to contributors, which does not afford the luxury of having a favourite OS. It simply has to work for everyone)

[defunctzombie]

@Pomax The day windows gets an "apt-get" or "ports" equivalent is the day I see this getting easier. I happily accept reasonable patches to make it work on windows and the only reason I support osx is cause supporting it is typically not harder than supporting debian (which is what my primary box is). Totally get that work might require you to support more things; maybe get work to pay these projects to do that ;) Ain't no free lunches here.

[TooTallNate]

Corporate sponsorship of open-source projects is very in these days!

[Pomax]

@shtylman I like to call that the internet, and a readme file that lists the prerequisites. Usually, that's all you need to get things working on windows =)

(although writing code you want the rest of the world to use in a way that lowers the threshold is usually an laudable thing. Sometimes that's quite hard, though, like this issue. It's possible to install a VC++ stack with the openssl headers as additional install and have things run on windows, but as a user it's understandable if they go "why are you making life so hard for me?")

[ncb000gt]

I don't believe that any of the maintainers are against windows persae, the only real issue is that we don't have windows systems setup. I have, in the past, gotten an environment setup so that I could test some patches, but even that didn't go very well and I ran into issues. We've put the call out for someone in windows land to step in and join the ranks of "maintainer" but so far nobody has shown much interest.

As for the binaries- there are a couple ways to handle it. We could take the approach of the sqlite3 project and store them on aws...I'd rather not. We could package them directly with the project and publish them directly, but that would definitely bloat the project. We could try to do something wrt to storing them in a separate github project and download the appropriate file in the same idea as aws. We could wait for some kind of binary support in the npm registry. I'm not sure if there is a best approach here as I doubt support for this is going to come in the form of the npm registry...

[Pomax]

yeah, the risk of repo-bundling is also progressive bloat as you commit updated versions. When doing a clone you would be downloading the entire history for all binaries. A secondary github project for the binaries specifically might work (maybe even as submodule?) but then you want to make sure you essentially wipe the repo every time you update the binaries so you don't grow the git history (git prune works locally, but you can't push the pruned history to github.com, sadly). Any particular properties of AWS that make it a questionable choice? (other than AWS being down more than it reasonably should be at this point.. the free AWS account had a fairly decent bandwidth allocation I though)

[springmeyer]

https://github.com/developmentseed/node-sqlite3/wiki/Binaries are the docs on our approach with sqlite3. It has been working well so far.

[ncb000gt]

@springmeyer Thanks for the link. I'll take a closer look when I get some time.

[springmeyer]

a few extra thoughts:

I also suspect some fun incompatibility at the libc level on older systems.

This was my main fear that prevented me from trying this for a long time, specifically libstdc++ incompatibilities. I feared you would need to statically link libstdc++ to be portable across linux versions. But then I read this message from one of the the libstdc++ maintainers which dispelled this myth. So I forged ahead with the safegaurd that I do a test require to make sure the binaries load (no dynamic symbol loading problems) and if this happens then a source compile is triggered. However, I've had zero reports so far of this happening.

store them on aws...I'd rather not

They are stored on s3, not aws/ec2 just in case that was the impression. I went with s3 just because I'm using it for storage for other projects and its cheap, but any reliable download location would work.

For interest here are the download stats since providing binaries last month. What is interesting is that this is mostly saving linux users from waiting for source compiles, since windows users are the minority with node-sqlite3. Of the thousands of requests only a handful have failed with partial transfers:

[defunctzombie]

Closing this issue as a "wont fix" for now since I do not believe the pain point of compilation warrants us going through all this trouble to setup a complex system which we need to maintain.

[springmeyer]

Update: Since my comments above about adopting a binary deployment system for node-sqlite3:

• I've created a standalone tool developers can use to make this easier: https://github.com/mapbox/node-pre-gyp
• I've started automating binaries on Linux and OS X using travis.ci, and windows using Appveyor.
• There are a variety of modules that are now using this approach: https://github.com/mapbox/node-pre-gyp/wiki/Modules-using-node-pre-gyp

[ncb000gt]

@springmeyer Interesting. I'll take a closer look at it in the next couple days. Thanks for the update.

[GongT]

so is there any precompiled binaries now?

I install bcrypt and it failed to download binary file from https://github.com/kelektiv/node.bcrypt.js/releases/download/v1.0.1/bcrypt_lib-v1.0.1-node-v51-linux-x64.tar.gz
(then it fail to compile, because I do not have python)

I'm doing something like:

SASS_BINARY_SITE=http://npm.taobao.org/mirrors/node-sass npm install node-sass
SQLITE3_BINARY_SITE=http://npm.taobao.org/mirrors/sqlite3 npm install sqlite3

npm install phantomjs --phantomjs_cdnurl=http://npm.taobao.org/mirrors/phantomjs

Is there any options like BCRYPT_BINARY_SITE or --bcrypt_cndurl=?

[Pomax]

I don't know your config actually does, but I don't see anything in it that involves binaries. Just because the environment variable is called ..._BINARY doesn't make the rest of the commands suddenly be binary installations; the instructions you show for those env. vars are just the regular npm install commands, but with a different npm mirror as base URL. If you get installation failures, the obvious first step is to not do things this way and instead rely on npm being able to do everything it needs to do based on what's in your package.json. Package/dependency installation does not require playing a guessing game with environment variables.

[GongT]

I don't understand how node-sass magically did this, but it really work.

If I install node-sass with this, it will take about 2~3 seconds

export SASS_BINARY_SITE=http://npm.taobao.org/mirrors/node-sass
npm i -d --progress=false --global node-sass

I can see this in npm log:

Downloading binary from http://npm.taobao.org/mirrors/node-sass/v4.3.0/linux-x64-48_binding.node
Download complete

but if I use only

npm i -d --progress=false --global node-sass

It will take about 10~20 minutes. I try this 10 times, it will fail 8 times.

---

fucking GFW