Add unsafe module into core node.js

[tracker1]

Given the existing limitations of building cross platform compiled modules for node, as well as a frequent need for one off calls to compiled libraries that may not need absolute binding performance, I suggest that an unsafe module be added, that is basically the inclusion of ffi, ref, and ref-struct into core...

var unsafe = require('unsafe');
var ffi = unsafe.ffi;
var ref = unsafe.ref;
var Struct = unsafe.Struct;

This would significantly lower the barrier to creating modules for interacting with system-built libraries. By specifically naming the module as unsafe I think that should be enough warning for most in the usage here.

There are already examples of taking similar approaches in other systems. For example, the characteristics of being able to reference libraries is a core functionality in .Net/mono and is used in many lower level modules, but rarely at a higher level.

[indutny]

I'm totally +1 for this, need to hear other Core team members opinion: @trevnorris , @tjfontaine

[sindresorhus]

:+1: This would be very useful for CLI tools and scripts which can live with the overhead.

[hemanth]

:+1:

[silvenon]

:+1:

[TooTallNate]

You'd probably want some of the other C primitives as well: https://github.com/TooTallNate/ref/wiki/Known-%22types%22#type-generators

On Tue, Sep 2, 2014 at 5:14 AM, Fedor Indutny notifications@github.com wrote:

I'm totally +1 for this, need to hear other Core team members opinion: @trevnorris https://github.com/trevnorris , @tjfontaine https://github.com/tjfontaine

— Reply to this email directly or view it on GitHub https://github.com/joyent/node/issues/8285#issuecomment-54142640.

[trevnorris]

I'm -0 on this. Questions I'd like addressed:

1. How much additional code and complexity will this add?
2. How many tests will need to be added?
3. Proper documentation for all of those modules.
4. What are the performance characteristics?

I don't like the name "unsafe". The name does not properly denote what the module contains.

[bnoordhuis]

Nothing against Nathan's fine work but I would suggest to base it on js-ctypes for reasons of standardization and interoperability. Possibly with extensions like .async().

How much additional code and complexity will this add?

That depends. ffi uses libffi and that's a fairly big and complex library. It probably can be slimmed down a bit because node.js doesn't care about e.g. avr32 or microblaze support.

It's possible to approximate libffi in mostly plain C++ with some template magic and compile-time code generation, at least for the architectures we care about.

What are the performance characteristics?

Compared to what? Native code? Terrible, of course. :-)

Jokes aside, it's my experience that it's not the FFI call itself that is outrageously expensive but the boxing and unboxing of parameters and the return type.

You can ameliorate that somewhat with ctypes by allocating parameters up front but of course that only helps when you make repeated FFI calls with the same parameters.

[trevnorris]

@bnoordhuis heh. figured as much for performance. I'm also curious about testing. It seems that we'd need a make test-ffi or some such, since it seems a build of some type would be required before the tests could run.

Honestly I'm mostly worried about increasing the core surface area so much right now. There's a lot of cruft and performance hindrances that need to be taken care of, and I feel like continuously adding functionality (especially at this scale) would turn into a bug fixing extravaganza.

[OrangeDog]

Also, given that these modules already exist without core support, why pull them in?

[saghul]

I'll just throw my 2 cents here :-) In Python we have the "ctypes" 0 module, which a very convenient thing to have, because one can write a module interfacing with whatever libraries in the system without writing C code. It even comes with some Windows API wrappers for extra convenience. This is specially useful when running code in platforms which usually don't have a compiler installed, such as the ones coming from Redmond, WA.

[TooTallNate]

I'd also like to take a step back and ask if there's a more low-level abstraction to FFI and some of these C types that can be exposed in core, where higher level abstractions like node-ffi and ref could be implemented on top of those APIs?

[trevnorris]

@TooTallNate Great question, and, IMO, the better way to go.

[bnoordhuis]

Good question, but I don't know what such primitive operations would look like. ctypes and ffi are already pretty low-level.

[tracker1]

I'm thinking in line with @saghul in that it's really useful for windows, and even osx, not to mention other platforms, but without them in at least the core distribution, it's harder to use. ffi, for example seems to bring in a lot of source that's already included in the core... I'd be fine with js-ctypes over ffi, It looks like ctypes uses the ffi library that node-ffi uses.. which really appears to be the common low-level solution.

I realize that absolute performance will be rather poor compared to truly native compiled modules, but that it is often worth it for a simpler abstraction. I also understand the pain and increased surface area that this would bring to the core.

As to the use of unsafe for the module name, my thought was to make it clear that this is an area you shouldn't touch unless you really know what you are doing and the potential side effects. I do, however feel that it's a critical piece of functionality that would be huge for rounding out the core. And would greatly diminish the need for binary modules in many cases.

Most of the time when I have issues bringing in an npm module, it's related to a binary module. Often for a very limited use.

There have also been several times where I need a very limited interface to a system method or two, and would rather avoid a ton of boilerplate... Sometimes it's just easier to use C# and just use edge.js ... I'd rather have it baked in though.

[trevnorris]

This isn't getting into v0.12 so I'm just throwing on the labels and we'll re-address this in the future.