Running multiple node instances in a process

Introduction

This is a summary of the issues and proposed solutions to be able to run multiple node instances in a single process.

The requirement arises on Android - and may also exist on other platforms - as a result of a number of resource and lifecycle platform characteristics.

The need arises primarily as a combination of two factors:

In order to do useful work, with custom modules that can call into the Android platform, node must run in-process in an Android application or service. Only by running in-process is it possible to have access to an android.content.Context and other contextual attributes (such as permissions) to be able to di useful work in the platform library.
The platform will only run at most a single instance of any given application or service process. Therefore, in order to be able to run multiple concurrent instances, or even multiple instances serially, it must be possible to initiate and cleanly exit multiple instances.

However, there are other motivations or potential benefits to doing this. First, it opens the possibility of a very lightweight fork() - which just involves the creation of a new thread and v8 isolate, which are very cheap in comparison with spawning a new process. Other helper threads are sharable between isolates.

Second, it would be possible to arrange for individual isolates to be able to run with reduced platform access - ie "sandboxed" - to execute untrusted code. I've not looked at this in detail yet, but it is clear that v8 isolates can provide a high degree of isolation - including resource constraints - between independently running instances.

The changes to run multiple instances go hand-in-hand with the idea that node is loaded and run as a library. So the multiple instance support as been implemented with an eye primarily on the library API, although a trivial main() is still included that behaves exactly the same as the existing executable.

The following sections summarise the current design and implementation. This is implemented and is running but would benefit from review and discussion of the approach, as well as a critical review of the changes and the possibility for those changes being upstreamed.

Overview

The core of the approach is that each node instance runs in a separate v8::Isolate, with a separate uv_loop. The principal abstraction in the API is a node::Isolate which encapsulates those two together with all other per-instance state.

Each node::Isolate runs in a dedicated thread. Just as with the conventional system, the thread that enters the Isolate enters, and blocks in, the event loop until the instance dies.

The following simple test code illustrates use of the API to run two parallel http servers.

#include <node.h>
#include <pthread.h>

void *run(void *arg);

int main(int argc, char **argv) {
  pthread_t t1, t2;
  int result = node::Initialize(argc, argv);
  if(result == 0) {
    pthread_create(&t1, NULL, &run, (void *)"1337");
    pthread_create(&t2, NULL, &run, (void *)"1338");
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    node::Dispose();
  }
  return result;
}

void *run(void *arg) {
  char *argv[] = {
    (char *)"node",
    (char *)"serv-hello.js",
    (char *)arg,
    0
  };
  node::Isolate *isolate = node::Isolate::New();
  isolate->Start(3, argv);
  isolate->Dispose();
  return NULL;
}

The first step is to initialize the library with node::Initialize(). This initialises the v8 environment, and the v8-specific options must be passed in the argv. Two threads are spawned and each creates its own isolate with node::Isolate::New(). The Isolate::Start() method launches that isolate.

The per-isolate argv can contain instance-specific options (such as --eval) as well as options for isolate-level resource constraints; the only such option today relates to stack limit. The isolate argv also obviously includes the isolate's arguments, and is in the same format as the regular node command line.

Each thread will exit the call to Start() under the same circumstances as with the conventional system; if the instance autonomously exits (as a result of there being no more watchers), or there was a call to process.exit() or some other fatal exception. By contrast with the conventional executable, however, it is not possible to send a signal to the hosting process and target any specific isolate; instead, there is a programmatic means to deliver a simulated signal to any created isolate through the API.

The changes have been implemented and tested in the unix universe (ie eio and ev) and there are gaps in the windows implementation. Input is welcomed to implement and test the necessary fixes.

The following sections summarise the issues and approach encountered in the implementation.

Migration of static data to per-isolate data

Much of the static state is in node.cc itself. All static members - with a very small number of exceptions - are migrated to be members of a new node::Isolate class.

Although this state is migrated to the isolate class, there is still a single static defaultIsolate, so the storage of that state has not changed in the default isolate case.

The changes are primarily in this commit.

https://github.com/paddybyers/node/commit/76b51b5abba1406fd791eb7e6a53f9d77123c58f

Retrieving the current isolate from an arbitrary context

Wherever possible, a pointer to the current isolate is obtained directly - for example, the Isolate associated with each of the prepare, check, and gc watchers etc is stored in the data field and retrieved without any lookup.

However, the entrypoints from Javascript - where we are only passed an Arguments - do not provide a way to pass the isolate instance pointer explicitly. In these instances the lookup is performed using v8's v8::Isolate::GetCurrent() which performs an lookup via v8's thread-local storage (which in turn uses pthread_getspecific). I haven't benchmarked this lookup yet. There is potentially a way to use the Arguments argument in these contexts to obtain the isolate directly - if there is a performance issue with the existing lookup then we can investigate this further. It doesn't look possible, however, by using only public v8 APIs.

There is also a potential optimisation in the defaultIsolate case where there is a simple test against (say) the known thread id associated with the default isolate. I haven't tried this yet - still I think the focus needs to be on optimising the general case instead of simply the single-instance case.

Some helper methods are provided to minimise the impact on the builtin module code - eg to get the uv_loop for the current isolate, or to get the last error from the uv_loop for the current isolate.

Hard-coded assumptions about the default loop

Several of the builtin modules have a hard-coded assumption about there being only the default loop. These have been updated to use the loop associated with the current isolate.

Most of the changes are here:

https://github.com/paddybyers/node/commit/376d6c1c05cf5fba30c2925d143e01a3b48b3db8

Default versus non-default isolate

As mentioned above, there is a default isolate, which continues to use the default uv_loop. The only practical difference this makes is that SIGCHLD handling (for spawn()) will work as now in the default isolate, but will not currently work in any non-default isolate.

Static data for builtin modules

Nearly all of the builtin modules use statics to keep persistent handles to certain v8 objects or primitives (typically strings). These values are not sharable between v8 isolates and therefore must now be managed on a per-isolate basis.

The approach to this is to create a way to access all such values from the node::Isolate instance. This means that an isolate lookup is needed, and the values must be accessed indirectly through a ModuleStatics instance instead of directly from static storage. The node_extensions.h mechanism is used to build a table of references to all such objects for a given isolate, so (apart from the resolution of the isolate itself) there is no further cost to obtaining those static values.

This, unfortunately, requires a change in nearly every builtin module.

The changes are primarily in here:

https://github.com/paddybyers/node/commit/729141f683d733ad0a182e17d5286a4d8bb4281c

eio issues with multiple loops

These issues are documented separately in these issues:

https://github.com/paddybyers/node/issues/15 https://github.com/paddybyers/node/issues/14

I am sure that the proposed solution works and is as efficient as possible in this multi-threaded environment (ie resolving to and from a ccontext/thread/queue without relying on statics or thread-local storage) but further investigation is needed to find out if this can be done efficiently without changes to the eio layer.

Argument and options processing

The argv and options processing code is shared between the processing for the system as a whole (which applies to the v8-specific options, and also node options that are specified as isolate defaults) and per-isolate options and arguments.

Much of the processing is moved into a new NodeOptions class but there is still some tidying that can be done.

https://github.com/paddybyers/node/commit/cd1cefc8172c90062224a2ca2d1d50833beefc5a

Exit handling

There are several places where node will simply exit by calling exit(). Obviously, this isn't going to work for a library.

Broadly, there are two sets of circumstances where this happens:

1) Fatal errors in startup. These are dealt with by setting an exit code and returning from the relevant point. Eventually this results in Start() returning with the exit code.

2) Killing the instance, from inside the event loop. This is relevant to process.exit() and other uncaught and fatal exceptions. Here, it is necessary to cause the thread to return abruptly back to the event loop (typically from deep within a v8 call stack, possibly with multiple nested invocations from javascript to native and back) and then exiting out of the event loop.

It is not possible to do this simply with an C++ exception (because v8 is compiled without exceptions enabled, and Android doesn't fully support them in any case) or longjmp (since C++ scopes are not unwound and resources will be leaked).

Returning abruptly back to the event loop is performed by forcibly terminating the v8::Isolate, which causes v8 to propagate and uncatchable termination exception, and so it returns as an uncaught exception back to the event loop.

Exiting the event loop is performed via uv_break(). (The windows analog of this is not yet implemented.)

The conventional exit() behaviour is retained, and the "library" behaviour is enabled via a preprocessor NODE_LIBRARY switch.

The relevant changes are here:

https://github.com/paddybyers/node/commit/f99a26d5b9c398b1825ee39fe93080dee08a0a58

The node::Isolate class includes a stop() method which is intended to deliver a simulated signal programmatically to the event loop. However, ev currently only supports doing this for the default event loop, so a change is needed to make this work for all isolates. In addition, the windows analog would need to be implemented. Forcible termination (ie the equivalent of sending SIGKILL) is implemented for all event loops, by forcibly terminating the v8 isolate.

Also see here:

https://github.com/paddybyers/node/issues/17

fork()

child_process.fork() is changed so that it launches a node::Isolate instead of spawning a separate process. There is a new set of functions in libuv to do the thread creation/destruction, together with the associated watcher (to emulate a process child watcher). This behaviour is enabled by the compile-time NODE_FORK_ISOLATE variable.

So far, in a runtime built with this enabled, there is no way at fork() time to ask that a genuine process is spawned; it either does one thing or the other based solely on the build-time variable. Other spawned processes behave as before.

process.env

Obviously all threads in a process share the same native environment variables. This, and sharing of other OS environment (eg cwd) is an unavoidable limitation of running in the same process. I would have preferred to leave this as a simple state of affairs, but unfortunately fork() relies on being able to set up the child env to have a variable that is not present in the parent env (the NODE_CHANNEL_FD variable.

What has been implemented is that an isolate's env is made up of two sets of properties:

an dictionary of properties purely kept locally in javascript inside the isolate. This is populated initially with the env that is passed in when creating a forked instance;
the real env properties in the underlying OS.

The local properties supplement, and override, the native variables. Pure javascript code running within node will see very little difference between this "simulated" local environment and the case where it is a real env of a separate process. (The difference will be in situations where a local value is deleted, which can reveal the same variable also present natively; and the fact that the local values are not inherited by any child processes spawned by the isolate that owns them.)

Not yet implemented

spawn() SIGCHILD handling for the non-default isolate.

Tidying to AtExit() in the non-library case.

Lots of others.

Performance

Some numbers will be added here to indicate the performance implications of having multiple isolates. The hope is to be able to demonstrate that there is no performance degradation in normal running (for example, from the isolate resolution that occurs on entry to a natively implemented method), and also to demonstrate the cost saving from launching an isolate as compared with a new process.

paddybyers / node