The future of GYP in Node.js

[rvagg]

Inspired by a discussion with @indutny, but this has been brewing for a long time now:

It's clear that Chromium has effectively deprecated GYP and stopped active development on it, with effort moving over to GN. @domenic reports that this is being finished up "this month".

GYP is stuck on Python ^2.7 and we get a regular stream of issues filed against node-gyp because of the pain people feel when using a default installed Python 3. In fact, the strict limited range of Python support (must be at least 2.7) causes pain on our build infra, where very old EL (EL5 / CentOS 5) comes with an older version and much of its system tools are tied to that version so introducing a new Python becomes a bit hacky.

We don't "support" GYP, we regularly get pull requests against node-gyp and nodejs/node that adjust the GYP source and our normal response is to take it upstream, where it sometimes succeeds through the Chromium gauntlet and is sometimes dropped. We've been (generally) unwilling to float patches, which puts us, and therefore our users, entirely at the mercy of Chromium developers. This is obviously an increasing concern as they move to GN and GYP becomes a legacy concern that they need to detach themselves from.

So, we need to come up with a plan for how to deal with GYP, or at least a general goal that we can encourage effort on where appropriate. If we are not proactive about this we're going to be left with a lot of pain, in fact, we've probably left it far too late as it is.

There are two very distinct issues to discuss and we need to make sure we keep them both in mind when discussing:

• Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.
• Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

Some beginning talking points:

• @chrisdickinson has collected some useful links on GN @ https://github.com/chrisdickinson/nojs/issues/3 and that project uses GN
• This thread on Twitter discusses how difficult it is to extract GN from the Chromium codebase and reuse it elsewhere, it's not a separate project we can just pick off the shelf although @chrisdickinson is doing some work on that.
• We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.
• CMake often gets proposed as an alternative, I'm not a huge fan but I also don't have a ton of experience so can't be objective. It's worth discussing since it's a mature solution.
• There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

[saghul]

require('this_comment');

I think replacing GYP (the build file format thingy) would break things too much. We can keep GYP (the syntax) and replace the project with a pure JS thing which just generates Ninja files. GYP and GN files are similar enough that even if Google drops it Node can keep up. And be independent.

We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.

I did that, and called it GYN and I was able to compile iojs with it. Original post. I still volunteer to finish the work, if that's the approach everyone agrees upon.

Note that this is not orthogonal to replacing GYP with a pure JS thing. Since it's the same codebase, GYP could be replaced with GYN, which gives us the advantage of working on Python 3 and switching to Ninja. Then if GYN is replaced with a pure JS implementation, the Python dependency would go away.

/saghul's 2 cents

[seishun]

Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.

We've already survived migration from waf to gyp. Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.

Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

So in the end it depends on what V8 does, doesn't it? I'm not sure how much V8 team cares about third-party users, but if they do, I find it unlikely that they would switch to gn in its current state.

There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

Not a huge fan of introducing another prerequisite for building Node.js.

[bnoordhuis]

We've already survived migration from waf to gyp.

The ecosystem was a lot smaller those days, though.

Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.

But only in one other project (Chromium), just like GYP in 2012.

There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.

[seishun]

But only in one other project (Chromium), just like GYP in 2012.

What about V8?

There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.

Do you mean addons or Node.js itself? I'm not sure what would be worse, using different build systems for addons and Node.js itself, or using a different build system for Node.js than for V8.

[indutny]

I'm working on a JS port of GYP at this very moment. Once it is done - it should be easier for the contributors to support it, and we could possibly modify it to fit our needs better.

[ofrobots]

Some other discussions on this: https://github.com/nodejs/node/issues/6089 and https://github.com/nodejs/NG/issues/24.

[eljefedelrodeodeljefe]

For reference I was working on this week and have a working compiler abstraction here: https://github.com/eljefedelrodeodeljefe/platform-tools

I am able to build C/C++ source files, native addons (haven't pushed win support yet) across all platforms and will head to compiling node itself after the weekend.

Also I disagree that we need a port of GYP or integrating Ninja or GN, since the task of building sources gets overcomplicated and overabstracted in all of those tools, imo, after looking at their sources and having been able to build a prototype really quick. There is a little bit more justification for a build tool on windows, since VS hides a lot of information like include paths in macros, which you have to dig out really. After that a callout to cl.exe is as easy as one to gcc.

[saghul]

@eljefedelrodeodeljefe Thanks for working on this! What I suggested was always based on the assumption that the migration would be slow, and changing one component at a time, trying to limit the risk.

I'm happy to hear you are able to compile addons calling the compiler directly, that's great news! FWIW Python does the same, you just create an instance of Extension, list some sources, call python setup.py build_ext and magic happens.

Keep up the good work!

[indutny]

FWIW, https://github.com/indutny/gyp.js

[indutny]

Update from gyp.js: node.js builds on windows with ninja and gyp.js, it also builds on mac/linux/freebsd as well, and possible on SmartOS with some tweaks to the .gyp file.

Should we consider investing some love into this project and/or using it in node-gyp?

[eljefedelrodeodeljefe]

I hope not. It's good engineering, don't get me wrong, but it is really bad management, sorry. This is 5000+ sloc of a port of a build systems that is by all means mediocre and chromium specific.

I am really unemotional about it, but ...sorry... this .... this is really bad news.

[Fishrock123]

Fwiw, with a JS port of GYP, people will likely no longer require python to build node addons if we use it in node-gyp, could be a big win, there's a good amount of non-compiler python issues that crop up there.

[Fishrock123]

I am really unemotional about it, but ...sorry... this .... this is really bad news.

What's bad news? Supporting an entire existing ecosystem without a another required language dependancy that we can better maintain ourselves? Perhaps not perfect, but that's a lot of wins IMO.

[evanlucas]

I'm pretty +1 on gyp.js. My only concern is what will we do once v8 switches to gn? IIRC that was happening this year sometime?

[eljefedelrodeodeljefe]

That you wanna do something really easy with 5000+ sloc someone needs to maintain and users need to be able to use without the fewest amount of bugs possible. With the rule being: the more sloc the bigger the surface error for bugs. It's specifically bad news, since I don't feel that node has a good track record on rather managerial issues.

[Fishrock123]

My only concern is what will we do once v8 switches to gn? IIRC that was happening this year sometime?

This month. The comments above indicate our ability to build via GYP shouldn't be too difficult to maintain.

That you wanna do something really easy with 5000+ sloc someone needs to maintain and users need to be able to use without the fewest amount of bugs possible.

We need to maintain it anyways for the ecosystem (for now, node-gyp..), so either we maintain Python GYP or JS GYP...

[cjihrig]

I'm +1 to eliminating as much Python as possible (from Node, not Python general).

[eljefedelrodeodeljefe]

no. You are missing out building our own solution, that is then as maintainable as you design it to be. Chromiums reason to build an own full fleshed build systems is rather historic and has mainly to do with the hundreds of repos they have and because they have lots of python people that were the only ones being able to build one back then. I don't know what you expect of a build system, but the way we use gyp we use maybe 5% of it.

It really is as easy as iterating over all .cc files and later link them all. So why not building some 500 sloc own build system that does just that in JS?

[seishun]

This month.

Could you provide a link?

The comments above indicate our ability to build via GYP shouldn't be too difficult to maintain.

You mean building V8 via GYP when V8 drops support for GYP?

We need to maintain it anyways for the ecosystem (for now, node-gyp..), so either we maintain Python GYP or JS GYP...

Ideally GYP support for modules should eventually be phased out in favor of gn. Gyp.js would be useful during the transition period since Node.js users wouldn't need to install Python to install modules that haven't updated to gn yet.

[ofrobots]

My only concern is what will we do once v8 switches to gn?

I don't think gyp as a tool is going away any time soon.

The question of how Node itself is built is a very different one from how native modules are built. I am +1 on reducing dependencies needed to deploy Node.js applications.

[eljefedelrodeodeljefe]

On the addons part:

I can do addon compilations with 300 sloc, doing the same what gyp does with no dependency and in a split of a second, here. Goes for all platforms, all architectures, doesn't need bindings.

The math for gyp-stuff (also ninja and gn) just doesn't add up. Why on earth would you use something big when you can use something lightweight?

  let out = [
    `${process.cwd()}/test/fixtures/sources/addons/node-addon-examples/8_passing_wrapped/nan/addon_8.cc`,
    `${process.cwd()}/test/fixtures/sources/addons/node-addon-examples/8_passing_wrapped/nan/myobject_8.cc`
  ]
  pt.compileAddon(out, {output: `addon_8`}, (err) => {
    if (err) {
      console.log(err);
      assert(!err, 'must not call error here')
    }

    let addon = require('../fixtures/sources/addons/node-addon-examples/8_passing_wrapped/nan/addon.js')

    var obj1 = addon.createObject(10);
    var obj2 = addon.createObject(20);
    var result = addon.add(obj1, obj2);

    assert.equal(result, 30);

  })

[indutny]

@eljefedelrodeodeljefe your point is certainly valid. I'm quite sure that we need to consider alternatives, but what if instead of finding them - we could just cut out all that "is not needed" from gyp.js? I have removed tons of features already, and the ninja generator is written from scratch as just a bare minimum for addons and node.

[indutny]

@eljefedelrodeodeljefe regarding your code sample:

• Does it support defines, escaping defines contents?
• What about include dirs? Dependencies, dependencies includes?
• Actions? Can it copy the output to some directory or run any script?
• Shared libraries?
• How does it support long command line arguments for the cl.exe MSVS compiler?

There are lots of stuff that I have discovered in GYP that we didn't bother ourself much with, because it "just worked".

[bnoordhuis]

either we maintain Python GYP or JS GYP

Short/middle term, I don't think it's either/or. gyp.js undoubtedly has a bunch of features it's missing and bugs that need ironing out. We'll need to maintain both until we're confident it's a drop-in replacement.

[indutny]

@bnoordhuis this could be just one LTS cycle for ironing out differences.

[eljefedelrodeodeljefe]

@indutny that would be possible of course. Can you summarize, what specific gyp or ninja features we use? What the "just worked" components are?

I like ninja over gyp of course, but still I don't find the justification of it (but please change my mind, of course)

re sample:

• Defines: not just yet, but it's a 6 sloc change. Exactly at the central component passing include dirs
• -I, -L etc., yes, all of it, and escaped. Under the hood the above calls two functions .compile(), .link() that, in the .compileAddon case set all the flags necessary for addon compilation in an options object.
• in the addon-case what we compile is a shared library(no?), otherwise yes. You can set, but there is no feature for it just yet.
• long lines: didn't know that was a problem, but this case should be avoided by calling cl.exe /c && cl.exe /link similarly to gcc -c && gcc that would call ld under the hood.

For node that can be tons of flags more, but it's already a lot. The above would result in a spawn callout to the below. (waiting for parallels to spawn)

What we could do is having a build tool like node-gyp and step by step phasing out gyp components first to gyp.js then to none at all, imo

[indutny]

@eljefedelrodeodeljefe GYP over ninja is not a choice, I'm afraid. As ninja builds using pretty raw platform-dependent build files, and GYP is the thing that can generate them.

Relevant to the command-line length:

• https://msdn.microsoft.com/en-us/library/ms404301.aspx

Quote from ninjas manual:

rspfile, rspfile_content
if present (both), Ninja will use a response file for the given command, i.e. write the selected string (rspfile_content) to the given file (rspfile) before calling the command and delete the file after successful execution of the command.

This is particularly useful on Windows OS, where the maximal length of a command line is limited and response files must be used instead.

[indutny]

@eljefedelrodeodeljefe for building node we use all of the features of ninja (it doesn't have many), and some of the GYP features:

• Actions: there are lots of scripts that are run during node.js build, many of them are python scripts, some of them use binaries that were just compiled
• Tons and tons of defines, lots of platform-dependent/flag-dependent conditions
• Of course, dependency graph should be just big enough to support differential rebuilds

Perhaps there are other things, but I can't remember them right now.

[eljefedelrodeodeljefe]

@indutny the node case is a more complex case, but looking at the verbose output, I don't see what the above thing couldn't handle. Except for the scripts that call binaries during, I can't tell, I didn't know. I wanted to make an attempt to build node w/o dependency and would start by just compile and linking the last few build steps.

My problem with make, gyp and ninja is that the amount of boilerplate Makefiles introduce, is that it quickly becomes writing in it as if it were a decent programming language. So you end up having python, js and a pretty complex make scripts that only a few people understand as a whole. Whereas porting the Makefile (or the underlying generated makefiles) to js probably looks ugly initially, but can be read by people.

Let alone there is one big advantage for developers (disadvantage for maintainers): make and msbuild, hide a lot of flags and logic. And, imo, that makes it really complicated to debug and to port stuff. For example, when you call cl.exe directly, it pretty much looks like a call to gcc. But you won't notice, b/c msbuild has tons of path include build in. Including the most basic C headers like errno.h making this visible seems more maintainable, then running gyp that generates a VS project, that generates a msbuild call that generates a call cl.exe

A callout looks like this. Note: this is a reduced flag set, which doesn't for example inlining souces into assembly code and build memory maps, things I think are necessary only for chromium and produce bloat, especially in file sizes of a compilation's secondary files

PLATFORM_TOOLS 1812: "%programfiles(x86)%\Microsoft Visual Studio 14.0\VC\bin\amd64\cl.exe" [ '/nologo',
  '/I"C:\\Users\\jefe-win\\repos\\platform-tools-2/build/deps/6.2.1/headers/include/node"',
  '/I"C:\\Users\\jefe-win\\repos\\platform-tools-2/node_modules/nan"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\10\\Include\\10.0.10240.0\\ucrt"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0V140"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\x86"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\x64"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\arm"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\arm"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\x86"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\x64"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\lib\\winv6.3\\um\\arm"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\um"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\shared"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\winrt"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\atlmfc\\include"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\lib"',
  '/I"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\lib\\amd64"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0V140"',
  '/I"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\um"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\shared"',
  '/I"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\winrt"',
  '/Zi',
  '/W3',
  '/WX-',
  '/Ox',
  '/Ob2',
  '/Oi',
  '/Ot',
  '/Oy',
  '/GL',
  '/D',
  'WIN32',
  '/D',
  '_CRT_SECURE_NO_DEPRECATE',
  '/D',
  '_CRT_NONSTDC_NO_DEPRECATE',
  '/D',
  '_HAS_EXCEPTIONS=0',
  '/D',
  'BUILDING_V8_SHARED=1',
  '/D',
  'BUILDING_UV_SHARED=1',
  '/D',
  'BUILDING_NODE_EXTENSION',
  '/D',
  '_WINDLL',
  '/GF',
  '/Gm-',
  '/MT',
  '/GS',
  '/Gy',
  '/fp:precise',
  '/Zc:wchar_t',
  '/Zc:forScope',
  '/Zc:inline',
  '/GR-',
  '/Gd',
  '/TP',
  '/wd4351',
  '/wd4355',
  '/wd4800',
  '/wd4251',
  '/errorReport:queue',
  '/MP',
  '/c',
  '/FoC:\\Users\\jefe-win\\repos\\platform-tools-2/build/addon_4.o',
  'C:\\Users\\jefe-win\\repos\\platform-tools-2/test/fixtures/sources/addons/node-addon-examples/4_object_factory/nan/addon_4.cc' ]

PLATFORM_TOOLS 1812: "%programfiles(x86)%\Microsoft Visual Studio 14.0\VC\bin\amd64\cl.exe" [ 'C:\\Users\\jefe-win\\repos\\platform-tools-2/build/addon_4.o',
  '/link',
  '/LIBPATH:"C:\\Users\\jefe-win\\repos\\platform-tools-2/build/Release"',
  '/ERRORREPORT:QUEUE',
  '/INCREMENTAL:NO',
  'kernel32.lib',
  'user32.lib',
  'gdi32.lib',
  'winspool.lib ',
  'comdlg32.lib',
  'advapi32.lib',
  'shell32.lib',
  'ole32.lib',
  'oleaut32.lib',
  'uuid.lib',
  'odbc32.lib',
  'C:\\Users\\jefe-win\\repos\\platform-tools-2\\build\\deps\\node\\6.2.1\\node.lib',
  '/MAP',
  '/MAPINFO:EXPORTS',
  '/OPT:REF',
  '/OPT:ICF',
  '/LTCG',
  '/TLBID:1',
  '/DYNAMICBASE',
  '/NXCOMPAT',
  '/MACHINE:X64',
  '/ignore:4199',
  '/DLL',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\bin\\x86"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 11.0\\VC\\bin"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.10240.0\\ucrt\\x64"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib\\amd64"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Lib\\winv6.3\\um\\x64"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\10\\Include\\10.0.10240.0\\ucr"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\atlmfc\\include"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include"',
  '/LIBPATH:"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include"',
  '/LIBPATH:"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0V140"',
  '/LIBPATH:"C:\\Program Files (x86)\\MSBuild\\Microsoft.Cpp\\v4.0"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\um"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\shared"',
  '/LIBPATH:"C:\\Program Files (x86)\\Windows Kits\\8.1\\Include\\winrt"',
  '/OUT:C:\\Users\\jefe-win\\repos\\platform-tools-2/build/addon_4.node' ]

[indutny]

@eljefedelrodeodeljefe agree! Though, JSON is as simple as .js 😉 hehe.

Anyway, I don't use make or msbuild in gyp.js, it invokes clang/gcc/cl.exe directly.

[eljefedelrodeodeljefe]

@indutny ah okay. I'll investigate. My only hope in any case is that we develop a very good design and spec, something beyond ports and deprecation plans, and instead of jumping the gun. Preexisting tools hardly serve as good examples.

One component probably is some declarative format, something like .gyp files ported to json or js. I'd prefer JSON, since you can hardly build in logic. And every fancier build step or some preparation could be implemented in js.

[indutny]

@eljefedelrodeodeljefe first of all, your github id is awesome. I have no idea how you remember it, though :)

I agree that something new may be a better idea in a long term, and indeed JSON is something that I like too!

[Fishrock123]

As for how Node.js itself it built, eliminating GYP or GN seems reasonable. I'm not really too concerned there.

[eljefedelrodeodeljefe]

:) thx @indutny. Maybe some googlers could shed some light on the build issue: I was reading somewhere that node is used in some build steps in the chromium project (?). How big is the love for GYP and family? Could node based tooling eventually be interesting for replacing the python build tools if quality and maturity is good enough and some time passes? maybe @ofrobots

[saghul]

Preexisting tools hardly serve as good examples.

Ignoring what has been done and why may take you doen't the same rabbit hole, making the same mistakes once again.

On a semi-related note, once necent effort seemd to be to find common ground with Electron, and going away from GYP would probably be counter productive on that realm, FWIW.

@eljefedelrodeodeljefe While I understand you want to "cut the middle man" here, can you elaborate a bit what you don't like about Ninja? I think it's a nice thing to be able to have a thin layer of abstraction between us and the different compilers, and Ninja seems to be up for the task. Whatever syntax we use as the DSL (or maybe not even! just calling some functions in a module which internally would generate a Ninja file at some point) can be thought of as a different story.

[eljefedelrodeodeljefe]

@saghul my main concern is accessibility. Having all JS devs being able edit and debug the build process is better in terms of the number of available devs than just the few of us who write C/C++, Gyp, Make, Python, JS, bash, .bat, while of this might not be necessary. I am provoking: gyp is like jQuery :)(http://youmightnotneedjquery.com/)

Proposal

I was able to build node with the above approach today and proposing this as API. Can be done a tools also of course, but wanted it built-in to make windows addon compilation easier. Please find it here https://github.com/nodejs/node/pull/7440

[Trott]

Discussion has been quiet for a few months. I'm going to close this, but feel free to re-open if you think it's useful as a placeholder or needs to be revived.

[ofrobots]

Reopening as this is still relevant, and was discussed earlier in the V8 + CTC meeting.

Here's a document with possible ways forward proposed by @hashseed: https://docs.google.com/document/d/1gvHuesiuvLzD6X6ONddxXRxhODlOJlxgfoTNZTlKLGA/edit#heading=h.13l47ngephw

[rgbkrk]

Before @hintjens passed he started a fork of node-gyp that was aimed at transitioning to ninja: https://github.com/CodeJockey/node-ninja

[indutny]

FWIW, gyp.js has a ninja.js fallback and uses ninja as a default build system too.

[hashseed]

@indutny do you have an update on the capabilities of gyp.js/ninja? More specifically:

• Is it capable of building the whole of Node.js on all supported platforms?
• Have you looked into parsing GN's build files? Or at least part of it?

In case it has not been clear in yesterday's meeting: V8 will end support for GYP by the end of the year. This means that GYP files will be removed from V8's repository.

Once V8 drops support for GYP, Node.js will likely inherit the GYP files. Going forward from that, it might suffice to be able to build Node.js with gyp.js/ninja from the inherited GYP files, and incrementally teach gyp.js to pull at least the (more up-to-date) file lists from GN's build files.

[seishun]

V8 will end support for GYP by the end of the year.

Could you link to that?

Also, is gn usable stand-alone yet, or does it still require checking out the entirety of Chromium codebase?

[bnoordhuis]

@seishun It was discussed in the call with the V8 team yesterday. Since us switching to GN is unlikely we're exploring alternatives; Yang wrote up a list.

For integration with V8's ToT CI we probably need to do something with GN but it's not clear yet what.

[eljefedelrodeodeljefe]

I want to reiterate my concern about gyp, gyp.js, gn and now also cmake. gyp and hence gyp.js are a design we should not continue and I think deprecation and the problems of the last years showed that. gn merely seems a patch for that. Cmake is a very big software suite, that again needs to be installed and can hardly be packaged.

What is entitled as "tabularasa" in the list, actually doesn't go far enough for me, as I would advocate for rolling our own tiny C/C++ build system, which is very easy to do with recent advancement from the MS compiler team, and would diminish the need for any binaries and python. Also building would happen in the JS realm, which more people are comfortable with.

[bnoordhuis]

@eljefedelrodeodeljefe The discussion was mostly about building node itself, add-ons aren't really affected by V8 dropping support for GYP.

For add-ons we'll be stuck with GYP, or something that consumes .gyp files, for a long time; any migration to a different tool would be a multi-year project. gyp.js is a step forward because it drops the python2 prerequisite that is an unending source of pain for users.

[eljefedelrodeodeljefe]

@bnoordhuis I see the point. I was also rather talking about node itself.

My only fear (in general) is that the makeshift solution would stick around, since no-one would take the effort afterwards. I'd rather prefer making a stab / conception / building a team to kickoff the this sort of multi-year endeavour and giving them the responsibility to not break the ecosystem. This way they would need to use and maintain gyp.js in the short run, but have an self-interest to actually.

Since the ecosystem is gonna look at node proper, it's build system should and could be a role model.

[refi64]

Problem with a build system in JavaScript is that now you have a cyclic dependency. There's a reason CPython still uses autotools.

[refi64]

Also, for what it's worth, something I've noticed:

Writing a build system is complicated. It's not hard, it's just complicated. It seems easy, and then you realize everything Windows-related is a hot mess, and then someone wants Node to work on their decade-old System V UNIX box, and then it just keeps on going downhill from there. I used to try and write my own build systems from scratch, but I stopped.

...actually, it's mostly just Windows. There's a reason build systems that don't support it are almost always smaller than build systems that do...

[sam-github]

I'm not sure why we would abandon gyp, which is working.

If we do http://msteveb.github.io/autosetup/ would be an interesting match for Node.js, its designed to be entirely distributed as a dependency of the app's source distribution, which aligns well with node's committing of its deps.

It also does the platform checks, etc., and then generates files for a dependency resolver, such as ninja, so its not trying to be the entire build chain.

But... it's TCL (note the entire, tiny, TCL runtime is distributed, only a C compiler is required as a build dep), but... the only good js scripting engine is node, so its hard to use js for building node.

There is also an autotools based setup, which uv went. autotools is well known, and lots of info is around on it... unfortunately, it needs lots of help because its so arcane, and particularly finicky on Windows.

[eljefedelrodeodeljefe]

@kirbyfan64 the statement about CPython is not quite correct. Python in fact uses python to build itself. Golang too. I don't know where this very simple problem of "cyclic dependency" started off, but it's not really a problem.

Also @indutny and I both where writing more or less complete tools for this purpose separately. It's also not a problem. The windows problem is also already solved since MS ships compiler binaries since last year.

@sam-github I wouldn't call gyp working, since it's the source of a lot of issues and will be deprecated. I can understand being conservative about this. But see above, I think your arguments hardly hold up. Also uv can't be using autotools on windows, since there is none. A JS toolchain would mitigate this.