The future of GYP in Node.js

[rvagg]

Inspired by a discussion with @indutny, but this has been brewing for a long time now:

It's clear that Chromium has effectively deprecated GYP and stopped active development on it, with effort moving over to GN. @domenic reports that this is being finished up "this month".

GYP is stuck on Python ^2.7 and we get a regular stream of issues filed against node-gyp because of the pain people feel when using a default installed Python 3. In fact, the strict limited range of Python support (must be at least 2.7) causes pain on our build infra, where very old EL (EL5 / CentOS 5) comes with an older version and much of its system tools are tied to that version so introducing a new Python becomes a bit hacky.

We don't "support" GYP, we regularly get pull requests against node-gyp and nodejs/node that adjust the GYP source and our normal response is to take it upstream, where it sometimes succeeds through the Chromium gauntlet and is sometimes dropped. We've been (generally) unwilling to float patches, which puts us, and therefore our users, entirely at the mercy of Chromium developers. This is obviously an increasing concern as they move to GN and GYP becomes a legacy concern that they need to detach themselves from.

So, we need to come up with a plan for how to deal with GYP, or at least a general goal that we can encourage effort on where appropriate. If we are not proactive about this we're going to be left with a lot of pain, in fact, we've probably left it far too late as it is.

There are two very distinct issues to discuss and we need to make sure we keep them both in mind when discussing:

• Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.
• Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

Some beginning talking points:

• @chrisdickinson has collected some useful links on GN @ https://github.com/chrisdickinson/nojs/issues/3 and that project uses GN
• This thread on Twitter discusses how difficult it is to extract GN from the Chromium codebase and reuse it elsewhere, it's not a separate project we can just pick off the shelf although @chrisdickinson is doing some work on that.
• We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.
• CMake often gets proposed as an alternative, I'm not a huge fan but I also don't have a ton of experience so can't be objective. It's worth discussing since it's a mature solution.
• There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

[refi64]

@eljefedelrodeodeljefe Actually, CPython uses autoconf + a custom makefile to build itself. Python is only used after it's build in order to build the extension modules. So the build process looks like:

• make builds Python.
• Entirely complete Python executable now builds the extension modules.
• done

In fact, there was a discussion about replacing Python's build system with one written in Python, and the end conclusion was to not do it, because of the "very simple problem" of a cyclic dependency. Node has the problem even worse; unlike Python, it's not like it's a standard package on a lot of Linux distros and such.

Golang too.

Go is actually self-hosting, so they don't have much of a choice. In addition, AFAIK if you build Go without cgo support, it's an entirely static binary, which means cross-compiling the toolchain is much easier than cross-compiling something like Node.

I wouldn't call gyp working, since it's the source of a lot of issues and will be deprecated.

There was a conversation on the Gyp mailing list a while back about this:

https://groups.google.com/d/msg/gyp-developer/d2P83vEvqYg/1ieqmmAqqK4J

I believe GYP will be at least maintained for a while yet, but there is little to no active development taking place; it does what we want it to do and is relatively stable and bug-free.

Sounds more like they've reached a "happy point" than they're abandoning it.

Also uv can't be using autotools on windows, since there is none.

? It works on Windows. It sucks beyond belief, but it works.

[eljefedelrodeodeljefe]

So I don't see anything against a hybrid gyp.js and JS only path. Exchanging nits about other projects doesn't drive ones own innovation.

[hashseed]

For building native modules that come packaged with their own gyp files, there is only using gyp (python) or gyp.js, for the medium term. But imo that's a separate discussion.

V8 will remove gyp files at the end of the year, and some newer changes to the build configuration are already only implemented for GN. For building Node.js core there is no solution that will just keep working. I listed the options here (previously already mentioned).

The option I am favoring most, coming from a V8 perspective, is to build V8 with GN, separately from the rest of Node.js, and simply link the resulting V8 library into Node.js. The advantage is that the maintenance cost is low, and V8 is being built the same way it is tested and developed for Chrome. The disadvantage is that to build Node.js, you would have to check out every dependency that you need to develop V8. It's also unclear how to support niche platforms.

The option probably easiest for Node.js is to simply inherit the gyp files from V8 and maintain it. But that simply moves the burden of maintaining gyp files to whoever wants to update V8. This makes continuous integration unnecessarily complicated.

I wonder whether it's possible to move building Node.js to ninja, and simply check in ninja files generated from GN for the most popular platforms.

[refi64]

I wonder whether it's possible to move building Node.js to ninja, and simply check in ninja files generated from GN for the most popular platforms.

You mean, that way people building wouldn't have to have GN installed? FWIW, Google already provides GN binaries for Windows, OSX, and x64 Linux...

[hashseed]

You mean, that way people building wouldn't have to have GN installed? FWIW, Google already provides GN binaries for Windows, OSX, and x64 Linux...

That is true. However, if I understood correctly, you need a long tail of supporting configuration files alongside GN itself, currently hosted in Chromium, containing platform and toolchain dependent configuration.

I might be wrong though. I'm currently getting in contact with GN folks as well...

Update: to use GN, you definitely need the GN binary itself, the build configuration for the build target, and build configurations for the OS and toolchain. The latter can be quite complicated depending on the required sophistication.

[Qard]

I like the JS option. I don't feel like it'd be a significant burden to have to download a previous build to be able to make a new build. Many modern languages like Go and Rust do that already.

[TimothyGu]

But what if we are trying to extend support to another platform, a platform w/o existing builds?

If we can build a reduced-feature version of Node.js w/o using JS, and then use that temporary version of node to build the entirety of Node.js, that'd be okay with me. However, GYP is used in building V8, and you obviously cannot have a node without V8.

[Qard]

As long as the JS builder can cross-compile, it shouldn't be a problem adding new targets.

[bnoordhuis]

Cross-compiling isn't where we shine, though. There are a lot of assumptions in our build scripts that make it fragile and sometimes downright impossible.

Nice GSoC project: cross-compile node from UNIX to Windows with clang-cl.

[eljefedelrodeodeljefe]

Isn't cross compiling anyhow only relevant for ARM, mips, aix architectures, where you don't want to compile for hours and where we are rather strong?

[Qard]

@bnoordhuis That's fair. I'm just thinking what the ideal situation would be, but certainly the reality may not be close enough. I haven't tried cross-compiling node before, so I wouldn't know.

Do you feel the gyp format is too limited to adequately express our needs for cross-compiling, or is it just that the current state of our gyp files doesn't show enough consideration of it?

[bnoordhuis]

The latter. It has never been a priority so far.

[indutny]

There is still a possibility of using gyp.js with duktape for extending platform support. Yes, it will be slow, but for doing the first build it should be fine.

[Qard]

That's an interesting idea. It might also be worth working with the native API stuff to try and make a sort of reference Node.js build on duktape. 🤔

[piranna]

It might also be worth working with the native API stuff to try and make a sort of reference Node.js build on duktape

I would love that idea! :-D

[ofrobots]

A few things I have learnt recently:

• GN can produce JSON output. This can be useful in building transpilers.
• Here's a project that uses said JSON output to convert GN to CMake: https://skia.googlesource.com/skia/+/master/gn/gn_to_cmake.py

[dpranke]

Hi all,

I am one of the maintainers of both GYP and GN (and I am part of the Chromium project).

If there are specific questions I could answer for you, do let me know :).

[hashseed]

We came up with a plan forward, described here.

TL;DR: V8 will implement a GN wrapper so that both the legacy GYP config and the new GN wrapper can be used to build the v8 build target. V8 will then stop maintaining the legacy GYP config, which Node.js will inherit.

[seishun]

In a second step, the legacy GYP config will be moved into Node.js’ repository. Keeping it in sync with GN build config becomes Node.js’ responsibility, which has to be done every time Node master updates to a newer V8 version

Has this already been decided? Because it seems... suboptimal in the long term.

[eljefedelrodeodeljefe]

I am also rather unpleased. Even though that was rather clear from the beginning, it now sounds good for V8 but 0-improvement for Node.

[hashseed]

Maintenance of V8's GYP configs will stop by the end of year. Node.js has been the only reason it is still being maintained so far, after Chrome has switched to GN more than half a year ago. Maintaining two build systems long-term doesn't sound particularly appealing.

The proposed plan decouples V8's GYP deprecation from Node.js. Obviously, if Node.js wants to move away from GYP, work has to be invested, regardless of whether Node.js wants to adopt GN or something else. I understand that this does not please everyone, but there are no viable and better alternatives on the horizon to my knowledge.

[eljefedelrodeodeljefe]

There easily could be one if @nodejs/ctc would create a policy for one. I wouldn't have expected something creative from your (Google's) corner...which is alright and no offense.

[refi64]

FWIW, I may be alone here, but I like the plan. Seems like it would have the minimal amount of potential breakage and workflow issues.

[Trott]

There easily could be one if @nodejs/ctc would create a policy for one.

Could you elaborate on this? I suspect I'm misunderstanding the meaning because that doesn't make any sense to me at all. (EDIT: What I mean is: I don't understand how a policy "easily" creates options that aren't there now. But there's a lot I could be missing...)

[fhinkel]

In a second step, the legacy GYP config will be moved into Node.js’ repository. Keeping it in sync with GN build config becomes Node.js’ responsibility, which has to be done every time Node master updates to a newer V8 version

Has this already been decided? Because it seems... suboptimal in the long term.

The gyp config is already in Node.js' repository as part of deps/v8. The difference is, that changes to it must be copied and pasted (or slightly modified) from V8's GN config files instead of from V8's gyp config files.

[hashseed]

Previously mistakenly dumped the meeting notes for the meeting between CTC and V8 team here. Moved them to where they belong here.

Sorry for the confusion.

[gibfahn]

@hashseed

Dumping meeting notes here.

Edited to @ everyone who was mentioned

[indutny]

Did I mention dukgyp already? Needs some love from Windows, otherwise is able to bootstrap node.

[seishun]

@trevnorris: A lot of work would be involved here.

Could someone describe what kind of work it would involve? Anything other than rewriting node.gyp, node.gypi and common.gypi and modifying Makefile and vcbuild.bat?

[eljefedelrodeodeljefe]

@Trott what I meant is that we are not leveraging the possibility of writing something better than exiting solutions. Gyp, make, CMake and all the likes are often bound to specific environments, like the dev environment at Google, and/ or overcomplicate things tremendously, because they turn DSLs. A viable option would just to make our own, suited for the dev environment of the Node.js ecosystem and being more simple, namely not requiring anything, but JS and a compiler. Other big software projects utilise authoring sub projects very well by having a plan / policy / vision someone can grab, setting out to spend a couple of months just programming what has been previously discussed as requirement.

[xml]

Please forgive an outsider to this project. But... it sounds like you have a great idea @eljefedelrodeodeljefe, and nobody seems to dispute the value of the ambition. I think people simply have lots of good experience with the gap between ambition and reality, and all the intervening edge cases. Until there's a tangible, viable alternative (at least a nascent one) that people could examine and rally around, the risk of implementation and adoption can't even be assessed, let alone solved. A strong community are going to quite rightly emphasize and invest in an MVP migration path until that's the case. (An MVP migration path might be a necessary parallel/backup option in any case.)

If you could show a baseline project and say 'let's use this, or something like it', it's a much stronger case. You mentioned you had a working build. Can you package that, name it, document it with a To-Do list, and share it to give people a sense of what would be involved? Once people can count lines, examine interface complexity, react to what's supported and what's not supported (especially around Windows)... it becomes a live option.

Until then, folks are almost certainly going to revert to the known possible.

[refack]

[edited to be less opinionated]

what I meant is that we are not leveraging the possibility of writing something better than exiting solutions

@eljefedelrodeodeljefe let me quote the scripture:

---

---

But seriously, I think a node-native platform-tools/autotools solution is great (wink wink), and perfectly suited for small/simple addons, and I'm willing to pitch in, if you're willing to commit to your platform-tools without apriori support. If we have a PoC that shows it can compile 'n' arbitrary existing addons (CITGM style), we could convince the community.

As for building the whole node tree, I feel we're stuck with .gyp files for the time being (speaking from experience nodejs/node#12218 which is a spin off of nodejs/node#11217 and took ~2man-weeks, just to deobfuscate one half of one rather small .gyp file). IMHO as a first step we should concentrate on the addons/node-gyp use case.

[hashseed]

As previously mentioned, V8 team is going to stop supporting the GYP build by the end of 2017. Here's the public announcement.

[Trott]

This issue has been inactive for a while and this repository is now obsolete. I'm going to close this, but feel free to open another issue in a relevant active repository (TSC perhaps?) and include a link back to this issue if this is a subject that should receive continued attention.