Running Node "Natively" on Mobile Platforms

[ianfixes]

Apologies in advance for my butchering of terminology and unfamiliarity with the finer points of Node JS.

TL;DR we expose JavaScript code to native runtimes across platforms

...so that we can write one SDK in JavaScript, then run it natively in Android, iOS, and Windows mobile environments. That project is here: https://github.com/paypal/manticore-native

I'm curious whether the nodejs team would be interested in using this code to put full-fledged node environments on these platforms -- Node modules would power browsers, servers, and native mobile apps all at the same time.

How our project works

1. browserifying a bunch of JS code so that it can run in an engine like j2v8
2. Generating native stubs for that JS code so that it can be accessed by native code (e.g. Obj-C, Java, C#)
3. Polyfilling the engine at runtime with things that our JS environment expects (HTTP, etc)
4. Loading the JS bundle into the engine at runtime

This provides a thin wrapper of native code that lets you reach into a JS engine at runtime to interact with real JS variables -- read them, write them, call them, and be called by them.

How "Node.js-native" could happen

There seems to be a lot of room for scaling up the polyfill step -- building it out to the point where the full suite of fs, process, (etc) modules would be supported in the native engine.

Limitations

• I'm not familiar with the breadth of the "standard issue" node modules, and what would be incompatible with the sandbox environments of mobile platforms.
• We use browserify, and that mangles certain modules (since they won't be available in a browser environment). We might need something more specific to our needs
• Inheritance and variadic arguments get a little messy in some languages; full and true native support might be tricky or impossible. But simple inheritance seems to work, and we have workarounds for variadic arguments (to a point).

  Questions

• Is a "Node.js-native" worth working towards? What would be make-or-break criteria?
• Assuming it's worth doing, what development milestones on our end would aid that effort?

Tagging @indutny as requested. Tagging @djMax as he is the primary author

[indutny]

cc @nodejs/collaborators

[vkurchatkin]

It's not clear what you are proposing. Everything other than running node itself (as is) on mobile platforms seems out of scope. As far as I know, it's possible to run it on Android already. As for iOS, it doesn't work and probably never will because of sandbox restrictions

[eljefedelrodeodeljefe]

Ignoring contents of the issue @vkurchatkin that is a managerial issue that has been weakened recently. Technically it would be possible and IMO very wanted for the markets' size. There could be negotiaons with the Apple Review team that likely would cause code adaptations e.g. disabling modules

[vkurchatkin]

that is a managerial issue that has been weakened

What do you mean? It has nothing to do with review, v8 simply doesn't work on iOS

[eljefedelrodeodeljefe]

I was under the impression that they just didn't want to have some VM feature but it was generally compiling.

[vkurchatkin]

but it was generally compiling

Well, it might be compiling, but doesn't work. Relaxing this restriction undermines overall OS security, so I believe it will never happen.

[ianfixes]

Fedor suggested that I open an issue here to discuss, and I should have been more clear: my goal is to determine whether manticore-native's scope should be expanded to include Node.js functionality -- not the other way around. And I don't mean to suggest that the two projects should merge in any way.

Maybe I should back up a step.

In the same way that browserify extends the "Node ecosystem" into the browser, manticore-native can extend it onto native mobile or native desktop apps. (We even use browserify in that process.) So on one hand, "browserify for native apps" might be the logical conclusion of our work. On the other hand, we have control of the engine -- we could continue to fill it with things that move us toward "node.js for native apps".

What is the opinion of this community on the value of either of those approaches?

[ianfixes]

@vkurchatkin we use JavaScriptCore on iOS. I'm not sure what you're asking with respect to Apple review and OS security.

[mscdex]

@ifreecarve I think he may be referring to the ability to use a JIT compiler on iOS? I believe the v8 team is working on an interpreter for an upcoming version of v8, so having something like that may help but I do not know when that feature will land yet.

[rvagg]

/cc @nodejs/tsc & @mikeal as this question expands beyond core somewhat.

Thanks for this @ifreecarve and @indutny, this is indeed very interesting and I for one am quite interested in exploring ways that Node can expand its reach, I think this very much in the Foundation's interests. Obviously it's unclear what "Node" might look like on top of these various platforms that you wrap but perhaps the goal of attempting to try and find a unified set of functionality might help us in the efforts we've been poking around within the @nodejs/api scope.

@ifreecarve can you provide us with a link, or perhaps a quick rundown, of what functionality that is Node-like that you're already exposing with Browserify? How much of the Node stdlib is already available through Manticore or is it basically just a JavaScript runtime with hooks into the underlying environment? I don't see a link to docs or anything that might explain the scope of the environment you're already exposing?

[ianfixes]

JavaScript runtime with hooks into the underlying environment

It's this. JavaScript runtime with native hooks into the underlying environment. Apologies for the shoddy state of our docs; this code was in several repos before we released it, and I am in the process of centralizing and rewriting the docs.

Node functionality we're exposing in manticore

I misspoke. We used to expose an HTTP-like module, but in our open-source release we're doing a fetch-like module instead. In any case, here's how it's implemented (e.g. in Objective-C):

1. Provide low-level support on native side: https://github.com/paypal/manticore-native/blob/master/runtime/objc/Manticore/PPManticoreNativeInterface.m#L88
2. Surface low-level support within native JS engine: https://github.com/paypal/manticore-native/blob/master/runtime/objc/Manticore/PPManticoreNativeInterface.m#L56-58
3. Write module code in JS that builds on low-level functions: https://github.com/paypal/manticore-native/tree/master/runtime/common
4. Write polyfill code in JS that is specific to the objc runtime/engine: https://github.com/paypal/manticore-native/blob/master/runtime/objc/js/polyfill.js#L21
5. Add polyfill to native engine: https://github.com/paypal/manticore-native/blob/master/runtime/objc/Manticore/PPManticoreEngine.m#L50

Node functionality would be added in the same way. Is this helpful?

[targos]

What part of Node functionality do you have in mind?

If we take as an example the fs module in an iOS app. Would the idea be that a call to fs.readFile('/a/b/c.txt') actually opens the file from the application's storage ?

[cjihrig]

How similar is this to something like JXCore?

[rvagg]

Well for a start JXCore is basically dead these days. I think this is pretty different in that it seems to be simply a way of interacting with native APIs via JavaScript rather than presenting an opinionated layer like Node's (and JXCore). The opportunity here for us would be to try and take our JS layer and turn it into something that could be backed by the platforms supported here. The idea of separating out a reusable frontend JS layer from a raw backend API has been discussed for so long but we've yet to come up with a way to start that effort, perhaps this might help? I imagine the biggest challenge would be in replicating the whole event loop infrastructure on each of these platforms.

[djMax]

I think it's probably pretty similar to JXCore, though when I originally looked at JXCore it had all sorts of open source problems (seemingly). But my original goal was different - I wanted to make an SDK that worked on any ES5 compatible platform with "first world" connectivity to native. i.e. callbacks worked both ways and not by serializing types in some hokey way. In some ways, manticore is like React Native but NOT for UI. So for example the HTTP stack is implemented primarily in native code for each platform, but exposed in a "JS friendly way" by some glue code that makes it look like the fetch API.

But perhaps more importantly, all JS objects can become first class native objects on all platforms just by using JSDoc comments. So the user of our SDK has no idea there's Javascript underneath, it just looks like any old native SDK. This also means the codegen portion does idiomatic translations - callback blocks in Objective-C, events in C#, interfaces in Java, etc.

So the development process for our SDK was essentially:

1) Write a bunch of Javascript to talk to network services, parse various data formats (we were doing EMV stuff), write lots of flow control and business logic, etc. We did this in ES6+async/await and some other minor ES7 stuff, and then transpile to ES5. 2) Write a bunch of tests for said code and have CI and all sorts of nice things that ran in node. 3) Run manticore codegen which makes us a native looking SDK in iOS, Android, Windows, WinPhone, Node Webkit, node, etc. 4) Put native UI on top of that cross platform SDK where necessary.

(And for us, the magic started here because now we can "certify" a single code base across 10+ platforms, which is very important in an EMV world)

[djMax]

Also, to be clear, it doesn't just expose all native APIs to JS. It creates a well defined way to call native code that you write from native, and as I mentioned to do so without JSON marshaling or losing function context. So I can pass real objects back and forth, not have to worry about property values getting out of sync, and not have to worry about firing events "specially" for native - regular event emitter just works.

[djMax]

@targos - IMHO, I'm not sure manticore is really about "node on mobile." It's much more about "Javascript on Mobile" in some sane way and that leverages the amazing ecosystem of node modules. You pick a good example - we didn't implement fs at all because we didn't need it. So if some npm module uses fs, it wouldn't work out of the box in manticore. And fs is a big ball of wax, so that might not be worth most peoples' time to implement. But browserify does have shims for a bunch of node APIs, like Buffer, and manticore sits on the shoulders of that. So we were able to use many npm modules, but node native modules would be out, and many node APIs are not currently implemented.

[vkurchatkin]

@djMax manticore sounds pretty cool, and to be honest it's exactly what I wanted for my projects for a long time; but I still can't see how it is related to node

[djMax]

@vkurchatkin I agree. (disclaimer: I'm no longer at PayPal, where this was all written so @ifreecarve or @indutny may have had deeper conversations that prompted this thread).

[ianfixes]

EDIT: whoops, should have refreshed before I replied.

@targos I don't have anything specific in mind. I'm more curious whether what we have would provide any value to the Node ecosystem.

@cjihrig our project is not discontinued, for one: http://www.nubisa.com/nubisa-halting-active-development-on-jxcore-platform/ But more seriously, I'm having trouble finding good docs on how JXCore works (http://jxcore.io seems to be dead) and how it interacts with native code.

From what I can piece together, JXCore looks like it's similar to Lua, where you can create an embedded environment and populate it with a "host API". Then, with some effort, you can get data in and out. Manticore-native is a bit closer to SWIG, where your interface to the underlying library consists of rich data types. For example, I can create an instance of a JS class in native -- it's backed by a JS object -- and use it like I'd expect to use it like any other native object.

[cjihrig]

I know that JXCore is essentially dead. I meant the similarities in running Node on mobile platforms. Sorry for that confusion.

[ianfixes]

@vkurchatkin Manticore is unrelated to Node.

As @djMax pointed out, our current use case for Manticore is to take advantage of node modules and JS to write cross-platform SDKs. Sticking to that suits me fine.

However, Manticore has a polyfill process which could be exploited to create a node-like environment, and I'm trying to establish whether it would be worthwhile to expand what we have. It sounds like achieving support for the full set of node APIs is not feasible. Is that accurate?

[djMax]

@ifreecarve the full set of node APIs would be extremely difficult and I think impossible on some platforms (like ChromeOS...)

[mhdawson]

I'll second Rod's request for pointers to examples of the Node APIs (or Node like) already implemented as a good place to take a look to help understand how it might work.

[ianfixes]

@mhdawson did this comment not satisfy? https://github.com/nodejs/node/issues/7573#issuecomment-231052352

I can go into more detail if you have specific questions

[nebrius]

It's worth noting that Appcelerator did something sorta similar a number of years ago at https://github.com/appcelerator/titanium_mobile (disclaimer, I used to work for Appcelerator). There was a time we looked into doing a rewrite, and I encouraged borrowing parts of Node for it, but was vetoed.

We also considered implementing Node APIs as a compatibility layer, but after some investigation gave up on the idea due to difficulty. Creating an API that is 90% the same as another API isn't too difficult, but getting above 99% represents a huge amount of effort.

We at Appcelerator discovered some other difficulties with some APIs that mirrored browser APIs: there's an uncanny valley effect for APIs where they get to a point that they're close, but not quite, in that 90-99% range. User's stop reading documentation of how your specific implementation works at this point, and instead use the original documentation or their own memories, and then get caught off guard when things don't work, and start filing bugs. This ultimately ended in more frustration and time lost for both us and our users than if we'd gone off and created our own API.

IMO the only effective way to have the same API across platforms is to use a shared implementation of the API across said platforms. Chasing another API with a ground up rewrite isn't fun.

[ianfixes]

These are good points. I'm content to close this issue and not worry about pulling [specifically] Node functionality into Manticore. Thanks all for your comments.