search

nodejs / nan

Native Abstractions for Node.js
MIT License
3.28k stars 502 forks source link

[discussion] integrating nan or similar with io.js #349

Open trevnorris opened 9 years ago

trevnorris commented 9 years ago

nan has done wonders for the community, but there are still missing bits. If these can be addressed there is a good desire to bring this into io.js itself. This thread should serve as a discussion on what are all the missing bits, what we can do about them and how to eventually get this into io.js.

The most prominent issue is lack of ABI compatibility. Needing to recompile with every new release, especially now with more frequent V8 updates, is difficult. @bnoordhuis mentioned that using templates is not a good form of ABI compatibility because they don't play well with the linker. (Ben, if you have any additional thoughts on this please feel free to include anything I've missed).

kkoopa commented 9 years ago

I'll start by noting that not using templates leads to very clumsy code, and since V8 makes heavy use of templates, this would require not directly exposing anything from V8. Otherwise the issue of ABI compatibility will reemerge.

rvagg commented 9 years ago

Some initial randomish thoughts:

  1. This is a separate effort to NAN and will be until it can completely replace NAN and we can deprecate it - even then we'll probably have overlap while people adjust
  2. Let's start simple and unambitious
  3. We're going to have to accept a world similar to NAN where the API/ABI will have to change over time to adapt to a changing upstream API but the benefit will be that you're dealing with a single API to code against
  4. People are going to want to reach around to V8 regardless of how sophisticated this is, we need to accept that and possibly make allowances for it, the problem with https://github.com/tjfontaine/node-addon-layer as I see it is that it goes too far in one leap and makes no allowances for anything but standard uses (the typical "framework problem"); this and item (2) above are the main reasons I've advocated for the NAN approach over the addon-layer approach in the past
  5. The continued instability from V8 is getting me down, they seem to have an endless appetite to screw with their API and therefore screw users of the C++ API, maybe it's the frustration with having to deal with permanent backward compatibility in JS to ES1 that forces them to take it out on the C++ API. @domenic says he's going to take this up with the V8 team but at this stage I'm much more willing to support efforts to detach Node from V8 as a direct dependency.
  6. Chakra introduces an interesting dynamic here and we may want to engage the Microsoft team on this effort given what they've had to go through to make their shim
trevnorris commented 9 years ago

Three things I believe could be done without templates and be ABI compliant:

  1. Local<Value> with a few methods. Such as Is{Undefined,Null,True,etc.}(), BooleanValue(), NumberValue(), etc. The hurdle to overcome would be casting to other types, and supporting all those APIs.
  2. typedef void (*FunctionCallback)(const FunctionCallbackInfo<Value>& info) could be a normal function call. info[i] could be an array of values, and since they're all initially Local<Values> the naive implementation would be supported. The problematic would be all calls that return Local<Object>.
  3. Buffers are pretty straight forward. This should even be possible using our Local<Value> abstraction, and in fact would be most compatible that way because currently it requires a Local<Object> and in the future will require a Local<Uint8Array>. Possibly something like the following (warning, this is very C-ish):
struct node_buffer_s {
  char* data;
  size_t length;
};
typedef void node_buffer_s node_buffer_t;
void FnCallback(const CallbackInfo* info) {
  node_buffer_t buf;
  Node::Buffer::GetData(info[0], &buf);
}
kkoopa commented 9 years ago

Would definitely want a code generator to generate the code for all the ensuing virtually identical wrapper functions with different types. It would be nice if one could get at the resulting C++ code after template processing, but before compilation.

Have a look at how Python 3 is doing it. Python is also dynamically typed, and everything is an object, but I have to say that writing addons for it is rather painful.

Also, I think the bar for a useful addon API should be set higher than trivial. If you can only do trivial things, you might as well just set up a socket or pipe of sorts and communicate between JavaScript and the real world through that.

trevnorris commented 9 years ago

@kkoopa

Also, I think the bar for a useful addon API should be set higher than trivial.

I was just echoing @rvagg's sentiment of:

Let's start simple and unambitious

So to clarify I was simply saying we start small, experiment and see how well it works. If it works at all.

One important thing is that if we want to truly achieve ABI compatibility then the exposed API needs to be written in C. There have been more than a few people smarter than myself that have expressed that C++ doesn't do well at maintaining proper memory structures when compiled with different compilers, or even different versions of the same compiler.

kkoopa commented 9 years ago

I agree that we have to start simple, implementation-wise. However, that initial implementation needs to scale up to offering a richer feature set.

It does not need to be written in C as such, it's just the external functions (and data) that need to be "C" exported. Without C++, how would the the wrappers interact with v8?

trevnorris commented 9 years ago

It does not need to be written in C as such, it's just the external functions (and data) that need to be "C" exported.

Agreed. I should have made my following statement more clear:

One important thing is that if we want to truly achieve ABI compatibility then the exposed API needs to be written in C.

So yes. I completely agree. It's just the user facing API that should be C.

kkoopa commented 9 years ago

Something like this should do. https://github.com/martine/v8c/blob/v8c/src/v8c.cc

On May 14, 2015 10:46:57 PM EEST, Trevor Norris notifications@github.com wrote:

It does not need to be written in C as such, it's just the external functions (and data) that need to be "C" exported.

Agreed. I should have made my following statement more clear:

One important thing is that if we want to truly achieve ABI compatibility then the exposed API needs to be written in C.

So yes. I completely agree. It's just the user facing API that should be C.

Reply to this email directly or view it on GitHub: https://github.com/iojs/nan/issues/349#issuecomment-102147935

trevnorris commented 9 years ago

@kkoopa good stuff. we'd just have to extend it to support current APIs like Buffer.

kkoopa commented 9 years ago

And new V8. Notice that this project has been abandoned for 6 years.

On May 14, 2015 11:00:11 PM EEST, Trevor Norris notifications@github.com wrote:

@kkoopa good stuff. we'd just have to extend it to support current APIs like Buffer.

Reply to this email directly or view it on GitHub: https://github.com/iojs/nan/issues/349#issuecomment-102151275

bnoordhuis commented 9 years ago

One important thing is that if we want to truly achieve ABI compatibility then the exposed API needs to be written in C. There have been more than a few people smarter than myself that have expressed that C++ doesn't do well at maintaining proper memory structures when compiled with different compilers, or even different versions of the same compiler.

Structures/classes don't have to be a problem when you don't use features such as virtual methods, multiple inheritance, member function pointers, etc.

I suppose name mangling could be an issue for people that mix C++11 code with pre-C++11 code. You can work around it by building everything with -fabi-version=6 but that requires that you use a recent compiler.

kkoopa commented 9 years ago

So, what we'd essentially want is to wrap v8's C++ exports in C exports to get a stable ABI and then have a (NAN-derived) header library on top of that which would restore the C++ nature of V8's API by mapping to the C exports.

Something like this:

extern "C" {
  V8StringUtf8 v8_string_utf8_new(const char* data, int length);
  V8Number v8_number_int32_new(int32_t val);
  V8Number v8_number_uint32_new(uint32_t val);
}
template<v8::String>
v8::Local<v8::String> NanNew(const char *data, int length = -1) {
  return v8_string_utf8_new(data, length);
}

template<v8::Int32>
v8::Local<v8::Int32> NanNew(int32_t val) {
  return v8_number_int32_new(val);
}

template<v8::UInt32>
v8::Local<v8::UInt32> NanNew(uint32_t val) {
  return v8_number_uint32_new(val);
}
kzc commented 9 years ago

Microsoft appears to have created a V8 compatible C++ API facade for Chakra. While it is a nice piece of engineering, it also has the disadvantage of trying to keep up with a constantly changing V8 API that is not managed by the node project.

I like the javascript engine wrapper C API idea proposed by @trevnorris. Node itself and node native modules could only use this C API. That way the javascript engine (V8, Spidermonkey, Duktape, JavascriptCore or other) could be made to be pluggable at runtime depending on the needs of the user. Native node modules compiled against this hypothetical C API could work against any such runtime-pluggable engine without recompilation due to its more stable ABI.

However, one downside with such a javascript engine C API would be that it would impose a little bit of overhead on every javascript <=> c++ operation as compared to header-only inline C++ classes or C macros. Even so, I still think it is worth the cost for the maintainability and the flexibility it would offer.

trevnorris commented 9 years ago

@kzc After doing testing in this area I'm fairly confident that we'd be able to do this with very little overhead. At least minimal enough where users wouldn't feel it in their JS code.

kkoopa commented 9 years ago

Clearly yes. The overhead of context switching from JavaScript to C++ drowns out everything else.

On Friday 15 May 2015 03:06:11 Trevor Norris wrote:

@kzc After doing testing in this area I'm fairly confident that we'd be able to do this with very little overhead. At least minimal enough where users wouldn't feel it in their JS code.

Reply to this email directly or view it on GitHub: https://github.com/iojs/nan/issues/349#issuecomment-102354581

kzc commented 9 years ago

I wouldn't recommend having the proposed C API functions and types mirror v8's C++ API and its concepts, because you would still have the original problem of tightly coupling node against an ever changing v8. Ideally the proposed C API should work equally well with Spidermonkey which cares about registering GC roots and Duktape which uses a Lua-style sandbox approach where you only manipulate copies of the actual data structures. But I recognize that such portability would impose a runtime cost.

kkoopa commented 9 years ago

On the contrary, I would recommend having it mirror V8's C++ API. Abstracting away the engine in a pluggable way is too much unnecessary work. It is not like io.js core would use these bindings internally, so it still would not become Spidernode. Personally, I have zero interest in supporting other JavaScript engines than V8.

kzc commented 9 years ago

Perhaps I mistook @trevnorris' proposal. I thought both node itself and native node modules would be changed to only use this new C API to isolate them both from the javascript engine's API.

If the goal is to only support v8, then I don't think the proposed C API buys you much. By closely mirroring the C++ v8 API any significant upstream change would still require an equivalent change in the C API.

trevnorris commented 9 years ago

@kzc The gain for maintaining a public C API is that native modules will not have to be recompiled every time they upgrade their version of node.

@kkoopa Unfortunately using this API internally is exactly what large businesses want. Microsoft has already done a port to Chakra, Oracle and IBM are independently working to get node running on Java. Anyway, just giving you a heads up that this is exactly what these large companies are after.

kkoopa commented 9 years ago

Oh, really? Why would they want that? What, exactly, are we discussing here? Is it making an ABI-stable API for modules or is it rewriting all of Node to abstract away V8? The former is a lot less work than the latter.

trevnorris commented 9 years ago

They want the ability to use whatever VM under the hood they deem appropriate. But then also have the user facing API ABI stable so they can ship pre-compiled modules. I'm only relaying this information so you're aware, and I dread to think what would happen to core in an attempt to do this.

The reason I opened this was to discuss the user-facing part. Creating ABI stability for native modules so they don't have to be recompiled with every release.

kkoopa commented 9 years ago

Good, then we're on the same page. In that case, I think the best to do is the following:

Take (almost) all the functionality already exposed by NAN (diregarding sugar, 96 % of it is used in native modules in the wild). This is the least a usable API should offer to support all the existing native modules.

Create C-exported wrappers for all the necessary V8 functions. Naming should reflect the original API, so let's use some arbitrary, but consistent, sort of mangling system through underscores. The v8 namespace may be considered implicit. We only expose the longest function signature when there are convenience overloads or default argument values.

Local__String__ String_NewFromUtf8(Isolate *, const char *str, int length, ...) {...}
Local__String__ String_NewFromTwoByte(Isolate *, const uint16_t *str, int length, ...) {...}
...

Then, as I mentioned in an earlier post, we could stick a header-only C++ API back on top, e.g. NAN or derivative. This library would circumvent the clumsiness and loss from having C-bindings, giving the best of two worlds. A C++ API with consistent C ABI. The naming scheme for C-exported functions can be arbitrarily clumsy, yet still hidden away.

All of this even seems like it would be possible to automate, i.e. Write a program that parses v8.h, getting all public functions and spits out a bunch of horribly named C function declarations and a C++ header that reconstructs the C++ API via the C functions. This would be cool, but probably not an easy task.

trevnorris commented 9 years ago

Sounds like a good enough approach to me. @bnoordhuis thoughts?

kkoopa commented 9 years ago

Would still need to export some functions from Node too, buffer, objectwrap, strings, etc. Then have som representation of exported classes as structs (they are rare,but a couple exist).

On May 18, 2015 8:55:18 PM EEST, Trevor Norris notifications@github.com wrote:

Sounds like a good enough approach to me. @bnoordhuis thoughts?

Reply to this email directly or view it on GitHub: https://github.com/nodejs/nan/issues/349#issuecomment-103148532

bnoordhuis commented 9 years ago

Something to consider: V8 uses RAII in a few places to good effect, c.f. HandleScope. A C shim would be quite error prone to use.

Exposing structs in the API might make ABI stability pretty complicated. (If I ever get around to writing a blog post "Libuv, lessons learned", that's going to feature prominently.) I would suggest using opaque pointers, coupled with accessors.

kkoopa commented 9 years ago

Can you give some concrete examples of what you had in mind, e.g. HandleScope?

On May 18, 2015 9:49:15 PM EEST, Ben Noordhuis notifications@github.com wrote:

Something to consider: V8 uses RAII in a few places to good effect, c.f. HandleScope. A C shim would be quite error prone to use.

Exposing structs in the API might make ABI stability pretty complicated. (If I ever get around to writing a blog post "Libuv, lessons learned", that's going to feature prominently.) I would suggest using opaque pointers, coupled with accessors.

Reply to this email directly or view it on GitHub: https://github.com/nodejs/nan/issues/349#issuecomment-103170593

kzc commented 9 years ago

@trevnorris, I agree that a C API wrapper makes sense if you intend to support javascript engines other than just v8. Just be aware that there are features and concepts in the v8 API that don't easily map to other javascript engines, and vice versa. Mirroring the v8 C++ API in C may create more work for the other engine mappings.

trevnorris commented 9 years ago

@kzc At this point I'm neither for or against mirroring the V8 API. Right now I'm just trying to facilitate conversation until we can achieve some actionable items.

bnoordhuis commented 9 years ago

Can you give some concrete examples of what you had in mind, e.g. HandleScope?

I think that's a question for me? Well, I imagine that C++ code like this:

void f(v8::Isolate* isolate) {
  v8::HandleScope handle_scope(isolate);
  // ...
  if (g()) return;
  // ...
}

Would end up looking something like this in C:

void f(v8_isolate *isolate) {
  v8_handlescope_enter(isolate);
  // ...
  if (g()) {
    v8_handlescope_exit(isolate);
    return;
  }
  // ...
  v8_handlescope_exit(isolate);
}

Manually having to balance enter/exit calls everywhere is a bit of a pain and easy to get wrong.

trevnorris commented 9 years ago

@bnoordhuis so the best solution would be to still use a C++ interface but stay away from (as you mentioned) virtual methods, multiple inheritance, member function pointers, etc?

bnoordhuis commented 9 years ago

I don't know about 'best' but a C++ API done right would certainly be easier and more robust to use.

kkoopa commented 9 years ago

Note that the idea was to reconstruct the C++ API on top of the C API. But, if it is less work and doable with a C++ API, then I guess that would be better.

orangemocha commented 9 years ago

+1 on achieving ABI compatibility.

I think that it would be hard to achieve ABI compatibility in C++. Here are two issues that I am aware of: 1) If the class definition changes even just by adding a private field at the end, the size of objects changes and any use of the new operator is broken across module boundaries. I think you could work around this by allowing instantiation only in the main module (node) through factory methods, and disallowing any passing by value in the addon module (because the copy constructor would be affected too). 2) Name mangling. This can break across compiler versions. I am not aware of any silver bullets to address this issue.

orangemocha commented 9 years ago

Chakra introduces an interesting dynamic here and we may want to engage the Microsoft team on this effort given what they've had to go through to make their shim

cc @geoffkizer, @jianchun, @curtisman, @EdMaurer from the Chakra team.

bnoordhuis commented 9 years ago

1) If the class definition changes even just by adding a private field at the end, the size of objects changes and any use of the new operator is broken across module boundaries. I think you could work around this by allowing instantiation only in the main module (node) through factory methods, and disallowing any passing by value in the addon module (because the copy constructor would be affected too).

That's correct. The same issue applies to C structs, however. I don't think C++ is any worse in that respect, it's just different.

2) Name mangling. This can break across compiler versions. I am not aware of any silver bullets to address this issue.

With gcc and clang, you control that through the -fabi-version= switch. I don't know if VS has a similar option.

The C++ ABI changes very infrequently. Before C++11, the last real update was in 2004, I think.

orangemocha commented 9 years ago

The same issue applies to C structs, however. I don't think C++ is any worse in that respect, it's just different.

True. I suppose you could design a C++ API that provides ABI compatibility. As you mentioned, it would have to disallow a few features that are specific to C++.

Another advantage of using C is, well, that you can invoke it from pure C code. It’s the lowest common denominator, on top of which you can build bindings for other languages.

So I am +1 for the approach (suggested by @kkoopa) of building the syntactic C++ sugar on top of a flat C layer, which would help streamline things like balancing the enter/exit scope calls.

With gcc and clang, you control that through the -fabi-version= switch. I don't know if VS has a similar option.

I’ll try to find out more about name mangling stability in VC++.

Aside from the C vs C++ issue, I think that in order to achieve ABI compatibility the API needs to be designed and maintained with that goal in mind. As @kzc pointed out in this comment that means that we shouldn’t expose v8 types directly, or we would be back to square one. v8 types can be encapsulated, or passed as opaque tokens (e.g. void *) but modules should not be exposed to their definitions. Correct?

Also, I think it makes a lot of sense to consider the interface between Node itself and the engine as part of this conversation. Please correct me if I am wrong, but an internal abstraction between Node and v8 would also solve the problem of native modules interfacing with Node, while the other way around is not necessarily true. Well, because the former is probably a bigger beast to tackle than the latter. It would also mitigate the pain of Node core keeping up with v8, and possibly allow for pluggable engines. My point is that at the time that we introduce a new API for native addons, and break compatibility, we might want to consider solving the other problems as well, because we don’t want to change this API again in 6 months. API should be designed to last. I guess that’s an argument against starting small and unambitious ;)

On that topic, it should be noted that JxCore has already implemented an abstraction layer that encapsulates the engine interface for both “Node” core and native modules. They have done so while maintaining compatibility with existing native modules written against v8. And they have made all the change in core to interface with this abstracted engine API. I have had the opportunity to talk with the folks who worked on the project, and they claimed to be willing and able to extricate the interface changes from the other features and submit a pull request to Node. Caveats: their changes are currently based on Node v0.10; and I don’t think their API is 100% ABI-ready in that it uses C++ and macros, but fixing that might be a fairly mechanical job.

So I think one possible course of action to address this issue at large could be the following:

  1. Engage JxCore’s help to….
  2. Port their changes from 0.10 to “converged node” master. This would also be a good test to see how much their abstraction layer is able to abstract v8 from old versions to the recent versions.
  3. Revisit their API to be more ABI-friendly, substituting macros with actual functions. Consider a mapping to C.
  4. Open a PR to Node

Does that sound like a viable option?

bnoordhuis commented 9 years ago

I'm personally highly skeptical that any kind of engine abstraction is workable in the long run and not a huge time drain and source of friction.

JXCore basically shims the V8 API, similar to the Chakra port. Updating V8 can already be big source of friction; now factor in the cost of updating all the shims.

What if you want to use V8-specific functionality? You can't unless the other engines have similar functionality.

What if the V8 API changes in a way that you can't reasonably emulate with other engines? Or vice versa? Do we stop upgrading?

Let JXCore and Chakra show that their approach is viable over a multi-year time span first before we start thinking about adopting it.

pittek commented 9 years ago

I've been recently walking through JXCore sources. Important points to remark;

  1. V8 and SpiderMonkey Javascript engines are already 180 degree different. (Native type incompatibilities, GCRooting, Concurrent GC, No GC in GC, Runtime is not an isolate, lot more)
  2. Emulation is not an option. JXCore doesn't emulate V8. My impression was only the final macro interface pretends like that.
  3. Same interface also covers multiple v8 isolates yet I'm not sure why they use multiple isolates ( I didn't walk through the docs yet)

@trevnorris clearly C,C++ etc. wrapper is not a solution alone. I agree with @orangemocha on JXCore's engagement in order to make the process less painful.

What if the V8 API changes in a way that you can't reasonably emulate with other engines? Or vice versa? Do we stop upgrading?

@bnoordhuis I don't know if you are familiar with SpiderMonkey. V8 and SpiderMonkey are already in a place that you can not reasonably emulate.

What if you want to use V8-specific functionality? @bnoordhuis can you be more specific? Sorry if I'm wrong but Does Node.JS promotes OS specific features ? Besides, other JavaScript engines also have their unique advantages.

One suggestion though. nan and jxcore can collaborate on a common solution for c,c++ node modules. This could be a good starting point and community would benefit from that.

trevnorris commented 9 years ago

@pittek OS specific != V8 specific. The specifics come when doing things like tooling around the VM, or even things like how we attach C++ class instances to Objects, and the Object to the class instances.

pittek commented 9 years ago

@trevnorris actually OS specific = V8 specific if you are exactly referring to tooling around the VM. How VM is different than OS ? Attaching C++ class instances to Objects? is the least concern that an abstraction would take care of. However how you manage the life cycle of that Object / Scope and Runtime is another thing. For example, LibUV plays some kind of an abstraction role between the underlying OS and Node.JS.

kkoopa commented 9 years ago

V8 is a library, not an operating system. They are quite different. V8 runs on many operating systems, so does Node, which depends on V8. Here, libuv abstracts away the operating system, with varying degrees of success.

plika commented 9 years ago

IMHO, there is no single best pattern that works for each JavaScript engine interface and internals. JXcore encapsulates SpiderMonkey while abstracting V8. Chakra is much more suitable for abstraction but in time the differences between the engines may require encapsulation taking the place.

I believe the real problem is something else.

The problem here is abstracting JavaScript engine capabilities to 3rd party solutions. JXcore has a separate abstraction for the JavaScript engine features. It doesn't have much to do how we abstract or encapsulate the JavaScript engine. It's all about sharing a common interface.

Today we use this interface for iOS and Android applications mostly. For example on Windows, once you have JXcore SM or V8 build as a shared library, feel free to switch it without recompiling the host application.

This interface is usable for the majority of c,c++ module scenarios. We can still allow old style module development but don't encourage it. I don't think a module developer would pick an option that will end up updating again and again.

As far as I know, majority of the c,c++ modules doesn't require a special access to core JavaScript engine.

Any thoughts ?

obastemur commented 9 years ago

Hopefully it is not going to be a wrong statement but I consider io.js as the node.js next. So, what will be the next is very important in terms of bringing a rather consistent interface for addons. (c, c++ addons)

I see node.js is engine independent framework in the future. However, node serving on a multiple engine interface is not that easy as a short/mid term goal. I agree with @bnoordhuis at some level that the multiple engine approach should evolve. But this process is not going to evolve alone.

IMHO, it's really nice to have all these (jxcore, chakra) efforts around node.js. I don't see a reason why we shouldn't combine the efforts. We can start with a native addon interface that will be backing the next gen node applications. My humble addition to @rvagg's initial list;

I believe, If we can establish a proper native addon interface, it would open a door to multiple JS engine backend support.

kkoopa commented 9 years ago

...I consider io.js as the node.js next

Yes, that is sort of how it is.

                        now
       (io.js)    v1.6   :  v1.7    v1.x
          |         |    :    |       |
 v0.10.x  /--------------:-----------------\   Node.js 2.0
____|____/               :                  \______|_____
         \               :                  /
          \--------------:-----------------/
          |         |    :     |       |
       (node.js) v0.12.x :  v0.13.x  v0.14.x

The full API consists of a Compact + Platform + Extended APIs

That is sort of how it is intended already.

   +---------------------------------------------------+
   |          Module and Application Ecosystem         |
   +---------------------------------------------------+
        |                |            |              |
        |                | +----------------------+  |
        |                | |  Binary Abstraction  |  |
        |                | |        Layer         |  |
        |                | +----------------------+  |
        |                |     |             |       |
   +----------------+    |     |             |       /
   |  Node.js Core  |    |     |             |      /
   |  Library API   |    |     |             |     /
   +----------------+    |     |             |    /
   |     js impl    |    |     |             |   /
   +----------------+    |     |             |  /
            |            |     |             | /
   +--------------------------------------+  |/
   | Node.js Application Binary Interface |  |
   +--------------------------------------|  |
   |              C/C++ impl              |  |
   +--------------------------------------+  |
                      |                      |
        +---------------------------------------+
        | Dependencies: v8, libuv, openssl, etc |
        +---------------------------------------+
obastemur commented 9 years ago

@kkoopa thanks for sharing the schema. I think I do remember seeing this definition once. Let me remark the differences over the schema then;

Sorry for the naive question; Is there a working group on this subject or how we/someone propose a working group ?

kkoopa commented 9 years ago

This seems to be the 'Addon API' working group. That means the Binary Abstraction Layer, and apparently the Node.js Application Binary Interface, because no existing WG would fit it better and they are sort of related.

Abstracting away V8 is sort of out of scope here. While I think everyone agrees that "it would be nice to have a wonderful abstraction layer for the JS engine", that is like having a desire for world peace: nice, but unfeasible. That being said, as there appears to be interest in spidernode or whatever, it seems reasonable to form a WG around that and let those who care for it work on it.

I don't know how new working groups are added. Do they have to go through a TC meeting, or is it just a matter of someone with admin rights creating a repository?

FFI will be brought to io.js, which should make trivial addons doable straight from JavaScript. That fits in the Core Library and API.

kzc commented 9 years ago

I see JXCore's javascript engine abstraction API makes extensive use of macros to achieve native module source code portability between v8 and Spidermonkey with recompilation. Although Chakra doesn't appear to use macros in its Node facade, it also seems to be a compile-time binding. @obastemur, could you comment on the feasibilty and estimated amount of effort to produce a pure C wrapper approach in order to achieve a stable ABI for node modules independent of the JS engine?

obastemur commented 9 years ago

could you comment on the feasibilty and estimated amount of effort to produce a pure C wrapper approach in order to achieve a stable ABI for node modules independent of the JS engine?

@kzc JXcore's public API for embedder application is a pure C wrapper (otherwise we wouldn't able to embed it into Objective-C app directly) And it's engine independent. Although our intention wasn't providing something for addons, we can update it to cover this part. In other words, this task has already been done for a compact set of JS engine features. It's not a rocket science what we did and the limited set of API can be extended.

referencing from @plika's comment;

For example on Windows, once you have JXcore SM or V8 build as a shared library, feel free to switch it without recompiling the host application.

@kkoopa

Abstracting away V8 is sort of out of scope here.

Let's say v8 version X is either a starting or end point. Considering the differences let say SpiderMonkey is Y. Preparing a compact API from X to Y (vice versa) doesn't abstract the V8 away. Instead, it helps to design something less vulnerable to X to Y or Y to X changes. At the JS engine basics level (compact API) it is doable and covers majority of the use cases.

I think everyone agrees that "it would be nice to have a wonderful abstraction layer for the JS engine", that is like having a desire for world peace

It's rather a sampling problem. If we can pick the right sample from the given group we may produce something that makes sense. I wouldn't also go for something like 'for once for all'.. For me, this is not a 'let's replace v8 to something' problem. We already did that. I'm ready to help if you want to do the same and also fine if you don't. My problem is the core of this framework which is subject to be shared among the solutions. Node.JS needs a core, a compact framework, compact public API that is not dependent (at least this much) to the underlying JS engine / openssl version etc...

FFI will be brought to io.js, which should make trivial addons doable straight from JavaScript.

I see why it's needed and I see how it will be hard to maintain it on many platforms. I don't know if I should repeat compact vs rest of the API puzzle.

That being said, as there appears to be interest in spidernode or whatever, it seems reasonable to form a WG around that and let those who care for it work on it.

I would rather go for a Node.JS compact framework workgroup. I don't know if I'm the only person believes that we need something in the core of everything else. Once we have such a core, rest is a customization around it.

imyller commented 9 years ago

:+1: for this process, but I'm trying to (over)-simplify this a bit:

My vote goes to solution that offers both:

a) Compact API, JS engine independent, as a way of calling native compiled libraries. Forget the Local<> stuff and V8 variable types etc. Just plain native types. This excludes a lot functionality, but that's the point. Maintain this compact API in the core.

b) Full native API that we now use directly or with NAN for those who need/want to have full access to and from JS engine.

I think many packages writing their native addons in C/C++ would just need simple "binding" between their native codebase and JS; not the overly fancy V8 facade, data types etc.

If you look at most of the published native addon packages, they have clean native C/C++ code written without any consideration for V8/JS and then a single "node_binding.cc" that is the necessary evil to bind native world with the JS .. and its usually the only thing that breaks with core upgrades.

pittek commented 9 years ago

@imyller couldn't agree more.