One Build System

[voodootikigod]

Since most of the hardware components in node exist outside of node and through native modules somewhere in the stack (most likely), it is important that as a unified front (until a better solution presents itself) that we all understand the external world that we may have to integrate with. For this I have written up my experience from node-serialport here: http://www.voodootikigod.com/on-maintaining-a-native-node-module/

It should provide you with pointers for integration efforts (and resulting tickets)

[nebrius]

Great article @voodootikigod!

[mikeal]

Should probably bring in @TooTallNate

[dtex]

@voodootikigod Have you seen or do you have thoughts on https://github.com/libuv/libuv/pull/379? Does it provide the functionality you need? If so, we may need to help get it over the finish line with testing and docs.

[joshperry]

I agree with this 100%. I have helped maintain a number of native node modules including a fork of node-serialport itself and they are, by far, one of the most frustrating parts of dealing with node.

As the popularity of node grows, and the desire to interact with services and devices that are currently only exposed via native libraries, this is becoming more and more of a barrier to entry for new projects that want to support node.

The reality is that huge numbers of target systems do not have a properly configured build environment. and one of the biggest driving forces behind nodes growth has been the ability to install simply, and get started easily. When native modules are involved, this is hardly the experience that users are having.

Since npm doesn't support the concept of binary distribution of native modules and does not provide a standard way to load a platform specific binary, I see a lot of duplicated code in the wild whose purpose is to prebuild binaries, publish them, pull them down at install time, and decide which to load.

It has been necessary for flocks of packages like node-serialport to go at it alone and each reinvent the proverbial wheel each time.

The loading side of this could easily be mitigated by promoting a lib like node-bindings into the official node org and have it called out in documentation as the way to load binary modules.

As for the build side of the problem, I see a couple options: Adding node-pre-gyp style features to node-gyp, make npm binary aware and include support for hosting binary tarballs alongside code in the registry.

This has been talked about a lot, and for a long time, and it's even on the npm roadmap.

The hangup seems to be that this is being made more complicated than it needs to be and involves many parts of the org cooperating on a spec that we can all agree on so that work can start in earnest.

There is another option though, that could bypass a lot of this work, and that is creating a highly-efficient interop bridge tech for node kind of a la P-Invoke in .NET. This wouldn't be easy, and it would require working with the core and possibly v8 teams to pull off something like this.

While I think a highly efficient interop layer would work for a vast majority of projects, there is no doubt that native modules are needed for I/O intensive workloads and interfacing with complex libraries. I still believe we should work on binary packaging, but I think an interop layer could be done sooner than a native packaging system, and handle most use-cases where binary modules are being used today.

[nebrius]

I'm skeptical that a dynamic interop layer would work for us. FFI, P/Invoke, reflection in .NET and the JVM, etc all take at least one order of magnitude performance hit, and is usually much closer to two orders of magnitude.

This, unfortunately, simply isn't viable for us, because most of our native modules have to run on platforms with low resources where we're already kinda pushing the processor just to run node.

If you have some insight into other methods or how to implement this (or have contacts/resources at Google), then we'd love to see you take a shot at it to try and come up with a new system that doesn't have a performance hit! It's definitely something of a holy grail for us, and so I'm skeptical that such a solution is possible, but then again I'm not an expert in this area either.

[joshperry]

@nebrius I can, unfortunately, only be helpful at a conceptual level when it comes to an interop layer.

It would require a certain amount of binary-fu on the end of the JIT compiler to find and fixup function call-sites to native functions so that they make direct function calls. We may require some kind of metadata, and have to place some restrictions on how and what data can be easily marshalled, but this isn't outside the realm of possibility.

If you take a look at asm.js support, it's kind of a similar beast and it's providing nearly native execution speeds for code that is compiled to target asm.js capable runtimes.

The largest costs, as I see it, are function call overhead, and marshalling data by copying. These are hard problems, but not ones that haven't been solved by other platforms before.