What about gyp?

[jbergstroem]

Edit: For all people landing here because you have issues after updating from nodejs to io.js or getting compile errors while building native modules: Your issue is about v8 being updated, not gyp itself. Please file issues at the module's issue trackers about having issues with newer v8 versions instead

Some months ago, I brought up the fact that gyp is pretty much abandonware on the nodejs mailing-list. People chipped in with concerns and questions, but it pretty much stayed dormant after that. I think the outcome of "ignoring" it – especially since the io.js (and/or node) community would now have to cater to gyp development if needed – is not what I'd call planning for a stable future.

I feel that the road to 1.0.0 would be best spent on trying to come to a conclusion as to what role gyp should play in the future of io.js. I'm inclined to summarise what's been said in the previous topic, but will refrain from doing so since it could come off as biased (go read it, now). If gyp will stay around, a decision to keep it instead of "for now" would be optimal. The idea of python requiring ruby to build, the same way io.js requires python is somewhat silly to me though.

[jbergstroem]

I'm unsure what keeping this issue open will contribute with. A consensus in agreeing that gyp should be replaced long term is met. At least one initiative has been started. If there are new ideas or concepts, these should be the grounds of a new discussion instead of having this linger indefinitely.

As suggested last comment for people finding this issue: @indutny, @saghul and others have spent time on a replacement which is incredibly exciting news. You can find @induty's work here and @saghul's here. I think once a replacement is mature enough a new PR/issue should be created that reflects the state at that time. Feel free to reference this.

[saghul]

I would like to know what @iojs/tc thinks about this, before investing any time on it. As it stands GYN does work, but it requires more effort, which I'm willing to put, if there is real interest in adopting it.

[bnoordhuis]

@saghul What's your plan with ninja? Would that be shipped with io.js?

[piranna]

Would that be shipped with io.js?

Maybe better as a npm dependency of gyn, isn't it?

[bnoordhuis]

@piranna I mean in the context of bootstrapping io.js.

[saghul]

@bnoordhuis good question :-) Since iojs ships with npm, which ships with node-gyp, I'd say the installer whould need to bundle the ninja binary (at least on OSX and Windows). The binary could be put somewhere not in the system path, and have a way to override the path to it.

[saghul]

And, for those manually compiling iojs, ninja would be a new thing they need to install, just like Python, or a compiler.

[bnoordhuis]

Right. What I like about @indutny's PYG project is that it removes a dependency rather than introducing a new one.

[saghul]

@bnoordhuis does it? I thought PYG's approach was the same: generate ninja build files and use ninja to compile them.

On the long run I guess removing the dependency on Python is something desirable, but right now a couple more things also depend in Python (which I discovered while trying to compile iojs with Python 3.4).

As I mentioned when I "announced" GYN, it doesn't need to be the last thing to be chosen, it could be used in the interim. (I'm assuming the general idea is to have something that chews gyp files and splits out ninja build files is what's wanted here.)

[bnoordhuis]

Um, you may be right. I thought PYG generated Makefiles but you're right, it only seems to support ninja. Okay, objection withdrawn. :-)

[jbergstroem]

I think embracing ninja (or whatever other minimal build system) is a trade-off we'll have to live with. Both pyg and gyn seems to share this approach. What I prefer with pyg is that it avoids the python prerequisite, which makes io.js users not having to install python (think node-gyp) while packagers or people that'd like to compile io.js does.

@bnoordhuis what are your thoughts on avoiding using ninja for compilation? Contributing something within the gyp replacement? I recall mentioning using makefiles in a previous thread and you replying that it'd be problematic on Windows.

Edit: also, if ninja were to be bundled -- wouldn't it rather be with npm (as in, replacing node-gyp)?

[saghul]

FWIW, in order to completely get rid of Python, these files would need to be rewritten in something else, in addition to the build system itself (I guess JS, to go full circle):

• configure
• tools/configure.d/nodedownload.py
• tools/js2c.py

[jbergstroem]

(reopening since there's still a meaningful discussion)

[bnoordhuis]

@saghul The idea is that a non-python gyp makes it easier for end user to build add-ons because they no longer need to have python installed and on the path. It would make life easier for Windows users in particular.

tools/configure.d/nodedownload.py

That file will probably go away.

tools/js2c.py

Can be done in 50 lines of C. (I have a C version sitting in a branch somewhere; I'd dig it up but it doesn't really matter whether it's C or python at this stage.)

[piranna]

tools/js2c.py Can be done in 50 lines of C. (I have a C version sitting in a branch somewhere; I'd dig it up but it doesn't really matter whether it's C or python at this stage.)

It could be transparently compiled on the build process, the same way Linux kernel compile some auxiliar tools automatically.

[saghul]

@bnoordhuis sure, I don't disagree :-) Just pointing out some surrounding work that would need to be done.

[unbornchikken]

Reading through this thread, regarding native addon build system (not to build io.js itself), there are two viable options mentioned: switching to gn or creating a pure js build system from strach (or by copying gyp functionality). Lemme share my arguments against those with ya!

• gn (or anything that based on Ninja): in a bright Linux based future, it might work, but on Windows, it makes things mutch harder than we have right now by using node-gyp. We can overcome the issue that there is no Ninja Windows distribution exists yet by putting a Windows binary in node-gn, but we loose the ability to create Visual Studio sln files from our projects. This will be a huge PITA since we have to switch from industry leading Visual Studio inegrated debugger to WinDbg console. (I'm willing to switch to my arse instead!)
• pure js build system: yeah, that's my dream too, +1000. But who should develop it? Node/io.js team? I think making an evented I/O for v8 javascript solution and making a cross platform native C/C++ build system, that has features those are comparable to gyp or better, and its first version is stable enough to drive the entire node native addon ecosystem, and is multiplatform of course is two really different things. A pioneer like @TooTallNate? Considering the previous statement, this is gonna be a complex and huge task for an invidual. A company like Joyent? I don't think that anyone here thinks that this is a viable option.

I think there is an other option: adapt an already estabilished build system for this purpose. I propose CMake. It's mature, fast, stable, has a friendly license, used natively by almost all cross platform C++ IDE-s, and of course, there is already exists a node-gyp like module to integrate it with the npm ecosystem (the author is me). It has nice features, for example it uses Ninja and Clang if present. Created module has no dependency on Python 2.x, only CMake.

I created a tutorial, to show you that creating native modules with CMake.js is as easy as creating them with node-gyp. Or easier, because since it's based on (almost standard) CMake, I can develop and debug my native modules even in stuff like Qt Creator.

I'm here because of two reasons:

• I need a high reach. Without anyone notice that there are other options available than node-gyp for creating native modules (right now), there aren't really other options available.
• I want to know the opinion of this community. If you all think that this idea is just plain stupid, then I won't put more effort into it.

[unbornchikken]

ANN: I've added nw.js support and it's Boosted.

[lygstate]

How about QBS? https://wiki.qt.io/Qt_Build_Suite It's based on QML and like javascript.

[unbornchikken]

Yeah, I did investigate that option before decided to use CMake. For simple modules both syntaxes are quite simple (see CMake's there), like your gyp file is right now. Many of native modules are exists because those are intended to wrap an already existing C/C++ library for node. If the actual library uses other build system than your module, then at first, you have to deconstruct its project file, then build your own by hand. It's a huge PITA. So the real question is, what is the build system that is used by the most of libraries out there? That is CMake (at least compared to QMake).

But I think it's not the build system that is the achilles heel of the node platform right now. It's the requirement of installing Visual Studio / Windows SDK (12 GB hog!) on Windows just to code or host JavaScript! (There was the Python 2.x dependency but that has been addressed in CMake.js already.) I think I can address this, and I'm gathering feedback on this topic there: https://github.com/unbornchikken/cmake-js/issues/17

[IonicaBizau]

Is this still an issue? Using Node v4.0.0, latest version of gyp works fine for me. :star2:

[saghul]

@IonicaBizau it still doesn't work with Python 3, you must be using Python 2.

[cjntaylor]

Just to add my own thoughts to this issue, I've created my own build system for node native modules based on CMake: https://github.com/cjntaylor/node-cmake

This stems from a bunch of work I've been doing in the robotics space, where CMake is the defacto standard (especially in ROS). The key is that I needed a way to integrate node module building both into npm, and into a larger CMake build structure, without relying on the presence of a node interpreter. CMake is very powerful in its own right, and is fully capable of finding/downloading/validating the required sources for building an extension.

Beyond this, tighter coupling on CMake side makes the build more flexible for IDE integration and cross-compilation, something I do frequently. This does still provide ncmake to simplify npm integration, and invocation on the node side.

I've included an example in the repository, and I'd love any feedback and comments regarding the module. This is only meant to add to the conversation as an alternative; I share @unbornchikken's concerns about the future stability of gyp/node-gyp.

Has anything been settled on that front? Is there a better place this should be discussed?

[jbergstroem]

@cjntaylor at this point a discussion should probably live in here: https://github.com/nodejs/NG. Me and possibly some other people in the build group is interested in solving this long term as well; feel free to join and chat about it: http://gitter.im/nodejs/build

[unbornchikken]

I believe the only viable solution would be an integrated ffi and web worker thread with shared buffer support. With those, almost anything could be created in JS side by using the modern es6 runtime.

[Trott]

Any objection to closing this and directing people to the NG repo? Maybe open a stub issue there?

[jbergstroem]

Sure, lets. See above for new issue.