What about gyp?

[jbergstroem]

Edit: For all people landing here because you have issues after updating from nodejs to io.js or getting compile errors while building native modules: Your issue is about v8 being updated, not gyp itself. Please file issues at the module's issue trackers about having issues with newer v8 versions instead

Some months ago, I brought up the fact that gyp is pretty much abandonware on the nodejs mailing-list. People chipped in with concerns and questions, but it pretty much stayed dormant after that. I think the outcome of "ignoring" it – especially since the io.js (and/or node) community would now have to cater to gyp development if needed – is not what I'd call planning for a stable future.

I feel that the road to 1.0.0 would be best spent on trying to come to a conclusion as to what role gyp should play in the future of io.js. I'm inclined to summarise what's been said in the previous topic, but will refrain from doing so since it could come off as biased (go read it, now). If gyp will stay around, a decision to keep it instead of "for now" would be optimal. The idea of python requiring ruby to build, the same way io.js requires python is somewhat silly to me though.

[rvagg]

I think the logical path is to move to gn in sync with Chromium, it's something we could start experimenting with already here.

[jbergstroem]

@rvagg: The problem is that the gn devs has shown little to no interest in making gn accessible outside the chromium sources. I've yet to revisit this, but at least last time I was looking at it there was a very tight coupling with headers and includes left and right. Replacing gyp with gn would also be my first suggestion, but if it requires some forking action we might not be much better off than sticking with gyp (besides sans Python) from a maintainer point of view.

[indutny]

I'm a bit afraid of GN, honestly saying. I think we should either stay with GYP, or move to something completely new and public. Probably that thing would compile to ninja, so it'll be a GN alternative in our control :)

[bnoordhuis]

I agree with @indutny: we should either fork gyp (and gut everything from it except make/ninja/VS support) or switch to another build system. It pains me to say it but perhaps the gyp critics were right in 2011 and we should have switched to cmake instead.

@jbergstroem Thanks for filing that issue. Now we know what the position of the GN developers is. We could perhaps lobby hard for a standalone build but there is no guarantee of success and the burden of maintenance would fall upon us. Maintaining a gyp fork where we don't have to care about other users is much easier.

[bnoordhuis]

A tangentially related issue: I think StrongLoop would be willing to donate developer time to replace gyp and node-gyp with a native application that doesn't require a python install. Windows users without python and RHEL 5 users with ancient python versions are the bane of our support department.

[jbergstroem]

@bnoordhuis That's great news. I think that having python as a buildtime requirement for io.js but not within the npm ecosystem would be an interesting part-goal. As for the bigger picture, getting something to work seamlessly over multiple os:es is a major headache, so starting fresh should barely even be considered.

I guess one option I haven't seen mentioned yet (for npm/gyp modules) could be falling back to basic Makefiles with something similar to how pg_config work for the postgres community. It's tried and tested.

I think the guys behind npm should weigh in here as well if possible - their input would be very valuable. gyp is somewhat stuck between io/node.js and npm and both sides could benefit from a well-defined surface (as well as who is responsible for what).

[bnoordhuis]

I guess one option I haven't seen mentioned yet (for npm/gyp modules) could be falling back to basic Makefiles with something similar to how pg_config work for the postgres community. It's tried and tested.

I'm unsure how that would work on Windows. Bundling make.exe with npm is probably not a big deal but writing a Makefile that works with cl.exe on Windows and g++/clang++ on UNIX is likely not a pleasant experience. We could predefine recipes but then we're back to creating our own ad hoc build system.

I think the guys behind npm should weigh in here as well if possible - their input would be very valuable. gyp is somewhat stuck between io/node.js and npm and both sides could benefit from a well-defined surface (as well as who is responsible for what).

That would be @isaacs and @othiym23 and perhaps @mmalecki too now.

Maybe @TooTallNate as well, he maintains node-gyp.

[maxkorp]

Rather than looking at suddenly having to maintain a standalone GN for the global community, what about just forking it, the same way everyone is talking about forking GYP? That way it's still an io.js specific build system, but there's already got a much faster system than gyp as a foundation.

[indutny]

I have an idea, could we write this tool in subset of javascript that is capable of compiling to some C code? Or in C that could be easily transpiled to the JS?

This would simplify life of node-gyp significantly. Otherwise after @bnoordhuis proposal, it'll need to download platform and arch -specific binaries on every install, or build them.

[bnoordhuis]

@maxkorp Two reasons that I can think of:

1. GN stands for 'Generate Ninja'. It doesn't support Visual Studio or Xcode or even make.
2. GN is closely integrated with the massive chromium-base source tree. It's very hard to try and extract (and maintain) a slimmed down version from that.

I tried and didn't exactly fail but it was no picnic either (3.5 MB source-only is the smallest I could trim it down to, in case you're wondering.) I would dread every upgrade and those are bound to happen frequently because of its young age. Forking and maintaining gyp is almost certainly much less work.

@indutny

it'll need to download platform and arch -specific binaries on every install, or build them.

Yes, but I don't see a problem with that. We can include it as part of a release.

Another approach is to remake gyp in pure JS but that would preclude the possibility of building node with that tool because of the chicken and egg problem. A native binary would still need to bootstrap itself first, of course, but that's less insurmountable.

For the record, I'm not married to the idea of a native binary but it would certainly make life easier for some users.

[maxkorp]

@bnoordhuis Regarding CN, a cursory look around doesn't seem to indicate anything contrary to your experience, and I'm inclined to say you're right in wanting to avoid it.

A question to consider then, Is ninja notably slower on windows/mac than msbuild/xcode? If so, that grants a gyp fork a significant edge in my mind. If not, I'd say the the simplicity of a single type of build file is definitely something to consider regardless of the system generating them, be it something new and open as indutny suggests, or a fork of gyp, or any of the other existing options.

Regarding the pureJS option, what about bundling some minimalist interpreter like flathead.js (windows already has jsc.exe) and defaulting to that when io isn't available? You then have something capable enough for an initial build, but something more optimized for subsequent builds.

Regardless, I'll echo the sentiment that dropping the dependency on python (without introducing any others) would be a massive plus.

[jbergstroem]

@bnoordhuis I'm not well versed in Windows, but wouldn't visual studio's nmake.exe be the equivalent of make, not cl.exe?

Built on that, either io.js or npm would grow functionality similar to pg_config (pass linker, compile and version info) and possibly some other magic. Nmake should be able to tackle the Makefiles.

Having python part of iojs bootstrap/compile would be acceptable to me. Platforms like Windows would most likely not build io.js anyway - rather use a binary. Even if gyp went away, there's still ./configureand stuff in tools/ that needs to be taken care of to remove Python completely

@maxkorp Ninja and gyp are two different things, often used together (gyp generating build information that ninja executes). GN is now replacing gyp in the chromium ecosystem.

[bnoordhuis]

@maxkorp

A question to consider then, Is ninja notably slower on windows/mac than msbuild/xcode? If so, that grants a gyp fork a significant edge in my mind.

I don't know if it's faster or slower but GN doesn't know how to generate VS project files and that would likely upset Windows developers. I think the best GN can do is drive cl.exe or msbuild.exe through ninja.

Regarding the pureJS option, what about bundling some minimalist interpreter like flathead.js (windows already has jsc.exe) and defaulting to that when io isn't available? You then have something capable enough for an initial build, but something more optimized for subsequent builds.

It's an option. The interpreter would still have to be bootstrapped first but that's doable for a simple project like that. I'll have to think about it.

@jbergstroem

wouldn't visual studio's nmake.exe be the equivalent of make, not cl.exe?

Sorry, I meant driving cl.exe from make.exe. nmake is not a drop-in replacement for GNU make; forcing the use of nmake would make supporting both Windows and UNIX more complicated for add-on developers.

[rlidwka]

Bundling make.exe with npm is probably not a big deal but writing a Makefile that works with cl.exe on Windows and g++/clang++ on UNIX is likely not a pleasant experience.

Is it possible to write a substitute for make + /bin/sh + builtins in javascript? Just for windows users?

[seishun]

I'm more curious about V8's stance on this. It seems unlikely to me that they would drop gyp in favor of a Chromium-dependent build system. I would rather expect them to either take over gyp or switch to something completely different. I think we shouldn't jump the gun here and instead just wait and see what V8 does.

[jbergstroem]

@seishun v8 started moving in May. Chromium is set to replace gyp throughout their ecosystem by March, 2015 (full roadmap here). This doesn't mean that all .gyp-files needs to go away, since GN already reads them - which means that v8 might possibly stay "backwards"-compatible with gyp for an unforeseeable future.

Edit: Linked to quote about March, 2015.

[jbergstroem]

@bnoordhuis Although its not a drop-in replacement, how much stuff from gnu make does a native module really need? Do you have any worst case examples? When moving between gmake and make on bsd's, most of the "gnu make'isms" I find is usually trivial to avoid. I should probably spend some time reading up on nmake.exe before assuming anything else..

One thing that could improve cross-os compatibility could be accompanying the toolset with a linter for Makefiles which tells you about gnu make'isms.

[maxkorp]

I think the best GN can do is drive cl.exe or msbuild.exe through ninja.

I'm pretty sure you're correct here, digging around a bit. It's not clear how far along that is though, as according to the GN roadmap windows compilation hasn't started, but digging through chromiums build docs it seems like it's decently far along. The uncertainty is more than I had originally thought though, which to me reaffirms the thought that GN probably isn't the best approach, at least for the time being.

Anyways, I was thinking that ninja driving msbuild would be perfectly acceptable (you get a functional build at the end, after all), but I hadn't really considered community acceptance. Clearly VS is still a required part of the toolchain, and while VS is still obviously a huge deal in windows development, I'd be curious to see how attached Windows devs working on core/native modules are to the actual VS environment beyond just it's toolchain. While I wouldn't be too surprised to find people being ok with using over IDEs (and thereby not requiring the project files), I'd be more surprised than if we found that many would still rather develop in VS. And my experience is that if you're using VS, offloading your build to a different system is an absolute hassle (at least as of current, although it seems this might be improving a bit recently).

All of the above is moot though anyways if the end goal is trying to avoid building something from scratch.

If we were to take the leap and concoct something new, a native application doesn't seem so bad. Perhaps taking Indutny's idea of some sort of JS subset that compiles to C, and also offering it for download prebuilt? My tiny interpreter idea seems a bit risky with more thought, as then you have to find/build/port a simple interpreter to work on a new system, before you can build the desired interpreter on the new system. All you've really done is added a lizard to the chicken and the egg.

[seishun]

@jbergstroem

v8 started moving in May.

This hardly looks like moving to me. It rather looks like they're adding gn support just for Chromium.

Chromium is set to replace gyp throughout their ecosystem by March, 2015 (full roadmap here).

That page explicitly states that it only applies to Chromium and downstream of Chromium ("Upstream projects like Skia and V8 can choose to stay on GYP if they like").

[bnoordhuis]

@jbergstroem

Although its not a drop-in replacement, how much stuff from gnu make does a native module really need? Do you have any worst case examples?

I'm by no means an nmake expert but I know it doesn't support .PHONY, .SECONDEXPANSION, functions like $(shell), $(subst), etc.

It's like trying to write a Makefile that works with both GNU make and BSD make; things are sufficiently different that it's a pain and most people won't bother. You are welcome to use Makefiles - npm is relatively build system agnostic - but I wouldn't want to foist that onto every add-on developer.

[jbergstroem]

@seishun You're right; I was somewhat brisk in my assumptions. I went through their issue tracker and couldn't find anything that pointed to a planned switch (or deprecation) - so they're probably not going anywhere at the moment. I think they're using gn for their own builds though.

@bnoordhuis Sure. I just want to explore options.

[piranna]

On NodeOS we are thinking about develop a Javascript version of GNU make just to don't need to use a compiled one, and it could be also useful for Windows. Any help is welcome.

Disclaimer: I'm the lead developer of NodeOS.

[formula1]

I currently am trying to convert all of gyp into node compatible javascript. When it comes down to it, the people leading this don't have an infinite amount of money. Whether or not they are even getting funding currently is questionable. If you want to see io.js succeed, you can be apart of the solution. Though showing there are issues in the first place is being a part of the solution.

My repo is here https://github.com/formula1/node-gyp/tree/gyp-js

You can run with the following command

 in_dir=gyp/pylib out_dir=gyp-js/pylib node gyp-js/automated.js

Its certianly not perfect, nor may it ever be. However, with the files I manually did there wasn't anything crazy that node couldn't do without addons. I added a dependency on async because without it my hand written code would look like poop.

[bjouhier]

+1 for a pure JS build tool.

[bnoordhuis]

My personal preference is to clone (a subset of) gyp as a native binary, to bootstrap iojs and to build add-ons with. Ideally, it would be a single-file build: gcc -o buildtool buildtool.c

A pure JS solution has some attractive properties but only works for add-ons. We would end up with two things to maintain, gyp and gyp-js, and that seems less than ideal.

[bjouhier]

@bnoordhuis Agree. What matters is that build does not add dependencies over what we already have.

[piranna]

We can asume that anyone that want to build Node.js/Io.js has a binary of them, so the pure Javascript version would be enought, isn't it?

[jbergstroem]

FWIW, I'm with @bnoordhuis; but I'd also be ok with a pure-js version since the more important goal is the npm ecosystem (and its dependencies/maintenance), not necessarily building node itself.

[formula1]

Well that makes me feel better about feeling like the conversion is drudgery. Either way, I made another attempt with coffeescript. I also made the debugging alot better (at least for the coffee script version). But this seems like a month+ project alone if not more so I'm probably just going to leave it as is for now.

If it can be compiled to a binary form easier, sounds fine to me!

[indutny]

Again, I'll restate my thoughts on it: let's write it in some easy-to-transpile C or JS, so that we won't need to keep both versions up-to-date manually.

CoffeeScript does not seem to be a right way for this, imho. I'd really do C to JS, because C has more narrow semantics than JS.

[indutny]

This is my personal opinion, though. Looking forward to see anything that works :)

[formula1]

Coffee script was not my first choice but it seems easier since python and coffee share a lot in common that I wouldn't have to change

But this project feels like its going to take a while. So if there is an easier way to do it, I'm all ears. I may not be pushing too hard on this.

On Saturday, December 20, 2014, Fedor Indutny notifications@github.com wrote:

This is my personal opinion, though. Looking forward to see anything that works :)

— Reply to this email directly or view it on GitHub https://github.com/iojs/io.js/issues/133#issuecomment-67761256.

Samuel G. Tobia (609) 751-1341 Samtobia@gmail.com

[piranna]

But this project feels like its going to take a while. So if there is an easier way to do it, I'm all ears. I may not be pushing too hard on this.

My first attempt was to add a REPL to PyPyJS, so we could use Python and vanilla GYP scripts on Node.js... Maybe it's easier...

https://github.com/rfk/pypyjs/issues/26

"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix." – Linus Tordvals, creador del sistema operativo Linux

[Fishrock123]

Again, I'll restate my thoughts on it: let's write it in some easy-to-transpile C or JS, so that we won't need to keep both versions up-to-date manually.

how about rust ;D

[formula1]

@piranna your awesome. I'll see what I can do with it

@Fishrock123 I don't know if your serious or not but I lol'd. Edit* Mostly because It would make a new dependence on a different language. I don't mean to be insulting.

[piranna]

Thank you @formula1 ;-)

[Fishrock123]

@formula1 I wasn't that serious, but I don't exactly feel like learning C.

[formula1]

@Fishrock123 I hear ya! For what its worth, Rust and Go and every other newer language out there looks like they're making some awesome strides to try and make things easier for everyone else. My Lower level adventures in programming have mostly been in java. And Asides from type Strictness and forcing everything to be an Object, It felt right. I'm confident we can all make a living without learning C, but C and C++ are most likely never going away. And so long as we like javascript we'll probably have to face it at some point. Its important to note that v8 and libuv are written for CLanguages. We never have to go to the deep end, but at some point we should

[dylang]

It was frustrating trying io.js 1.0 and thinking I must have done something wrong because my gyp modules installed with Node 0.11 weren't working and reinstalling with io.js 1.0 was giving strange npm errors. I fortunately found this issue but I think we should have something in the changelog and on the web site.

Update I didn't have a ~/.node-gyp/1.0.0 directory so I mistakenly thought gyp was not going to available for iojs users. This was fixed when I upgraded toiojs 1.0.1`. Thanks @othiym23.

[appsforartists]

I'm excited to finally have access to modern versions of V8 on the server, but I sadly won't be able to try it until there's a solution here. Too many critical packages (like node-sass) depend on gyp.

[jbergstroem]

@appsforartists: The topic being discussed in this issue isn't really related to the isse @dylang and most likely you are experiencing. Your issue (sorry for slightly assuming based on your feedback) are related to the fact that a newer version of v8 requires all your native modules to be rebuilt - and some of them fails to build because the v8 api has changed. This issue is about the long term effort of replacing gyp with something else, which necessarily wouldn't solve the fact that native modules needs to be rebuilt when upgrading v8.

[appsforartists]

Thanks @jbergatroem. I'll have to bring it up with the individual module owners.

[jbergstroem]

@appsforartists fwiw, you would encounter similar issues upgrading nodejs from v0.10 to v0.11.

[piranna]

@appsforartists fwiw, you would encounter similar issues upgrading nodejs from v0.10 to v0.11.

We've had this problems on NodeOS when changing to Node.js v0.11 and @netlovers fixed them by porting the modules to use NAN, maybe the same could be done for io.js, and in fact it would be a good idea to left that compatibility issues to an external component like NAN instead of using directly Node.js/io.js APIs...

"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix." – Linus Tordvals, creador del sistema operativo Linux

[appsforartists]

@jbergstroem FWIW, I've been on 0.11.13 for months and not had these problems.

[othiym23]

@dylang I'm not sure there's anything in here that indicates why node-gyp would fail to work with io.js, so maybe you should open an issue here or on npm/npm describing what you were running into?

FWIW, I agree with @bnoordhuis and @indutny that the best solution would be something simple in C or in JS that could be transpiled to C / C that could be transpiled to JS. I've worked with compilers / interpreters that require an existing binary of the compiler to bootstrap the build (Haskell, several Lisps, AliceML (I think?)) and it adds a huge amount of overhead to the build process and complicates maintenance. Something that's simple to build from the command line by hand and uses one of the languages we already require support for for add-ons would be ideal.

[Fishrock123]

Please see https://github.com/indutny/pyg

[saghul]

Hi all,

I had an idea the other day and I felt like I should try it out. Allow me to introduce you to GYN: Generate You some Ninja (code here).

Based on some of the comments here, the investment Node/iojs have in GYP and the fact that @indutny started working on a replacement in C (see pyg) looks like a transition system could help (IMHO), that's what GYN aims to be.

What GYN is:

• A GYP fork
• It has a single generator: Ninja
• Works with Python 2 and 3

Currently I built some "Minimum Viable GYP" of sorts. The code can probably be trimmed further, but just by having Ninja as the only generator, the code is a third of GYP.

There is stuff to be done, but with this minimum version I could build iojs and libuv using Python 3.4 on Linux: https://github.com/saghul/io.js/tree/gyn and https://github.com/saghul/libuv/tree/gyn

If there is interest in pursuing this approach forward I volunteer to finish the work:

• Package it nicely and upload it to PyPI
• Fix the test suite on Python 3
• Run tests on Travis for Python 2.6, 2.7, 3.3 and 3.4 (I hope we don't need more)
• Eventually remove this hack: https://github.com/saghul/gyn/blob/master/gyn/input.py#L176

I also wouldn't mind transferring the project to the iojs org.

All feedback is welcome.

[piranna]

Run tests on Travis for Python 2.6, 2.7, 3.3 and 3.4 (I hope we don't need more)

Running on PyPyJS would be awesome... :-D

[JCMais]

+1 for @saghul and @indutny proposals!