Please make Node.js embeddable (e.g. libnode)

alshdavid commented 6 months ago

What is the problem this feature will solve?

Node.js is prolific throughout the server-side world with many projects wanting to repackage the Node.js runtime and standard library to offer it to customers such that they can reuse their existing tooling/dependencies.

Notable examples of this:

Rspack
Esbuild
Parcel
Supabase
AWS Lambda
etc

Currently, Node.js is not (easily) able to be used as an embedded runtime leading projects to look for solutions with incomplete support - like Deno (which is a great project in its own right, but lacks certain features that would necessitate consumers rewriting project code).

What is the feature you are proposing to solve the problem?

It would be amazing if Node.js released a libnode library that exported Node.js capabilities and enabled napi access to the host context.

Professionally, my organization has a use case that would benefit greatly from the ability to embed Node.js within a Rust based host process - however, practically, no such capability exists.

I'd assume this would be in the form of a c++ library though a Rust crate would be amazing.

What alternatives have you considered?

Embedding Deno
Removing the "Deno" parts of Deno and maintaining a light version with a focus on supporting the Node.js standard library
Looking over the Node.js source to understand how difficult it would be to create a Rust create with bindings

joyeecheung commented 6 months ago

libnode already exists via the --shared option of configure, and it's been used by other projects like mongodb and Electron to embed Node.js. Check out the documentation and the header file for details.

peng commented 6 months ago

Can you show some demo

alshdavid commented 6 months ago

Thanks for the reference and documentation @joyeecheung! Glad to see there is a pathway to integrate Node.js into an application - and I'm a little embarrassed I missed that part of the documentation.

Any reason this is not officially supported?

Is it that the Node.js team doesn't want to offer API stability guarantees or is it simply not a big enough use case to be a point of focus?

Outside of the Node.js executable, are there more examples of usage?

EDIT: I have found this Rust crate which seems to do something with libnode and NAPI https://github.com/branchseer/rust-nodejs

joyeecheung commented 6 months ago

I think it’s already semi-officially supported. The reason why it’s not fully officially supported is probably mostly that not enough volunteers have stepped up to maintain it or provide support for it. But usually whenever we touch Environment or the NodePlatform we do think about API/ABI breakages and do compatibility patches on older release lines. We also have some cctests and whenever embedders get broken accidentally they will send patches and there usually will be someone reviewing them & try to get them landed (through the flaky CI…). The documentation could probably use more updates and elaboration but not enough people have volunteered to do that either….PR welcomed though :)

Is it that the Node.js team doesn't want to offer API stability guarantees or is it simply not a big enough use case to be a point of focus?

Not sure where the policy document is but I am fairly certain that we do provide API/ABI stability guarantees on LTS.

addaleax commented 6 months ago

and it's been used by other projects like mongodb

Just to clarify: libnode is too impractical for us (= mongodb), so we're just building our embedded application as if it is a regular Node.js build from source. 🤷‍♀️ If somebody did step up to properly maintain libnode as a library that we could use, that would be awesome for us though.

alshdavid commented 6 months ago

@addaleax does that mean you're vendoring in parts of the Node.js binary source and rewriting the entry points to be callable programmatically?

as in, translating

node ./index.js

to something like

let runtime = NodeWrapper::exec("/path/to/file.js");
runtime.eval();

Do you need to inject napi functions into your Node.js environment from the host application and can you do that with your approach?

In my application I am finishing off integrating Deno before I begin exploring libnode - But now I am wondering if libnode is worth exploring.

I'd be happy to take on work to get it to a point where it could be embedded but if there is too much missing I'd imagine it would require substantial changes which would be difficult to get into Node.js (PRs, reviewing, etc).

alshdavid commented 6 months ago

Another option is if there was an ability to somehow have an adapter that goes between an existing Node.js process and an external program with the same performance as Napi.

I tried experimenting with using a memory mapped file to act as a place to store shared memory, where the Node.js side has a napi module which opens the shared memory and both the host and Node.js can work with it - but building such an adapter is very cumbersome and limiting (e.g. sharing dynamic data structures like HashMaps and Vectors) so I gave up on that

joyeecheung commented 5 months ago

To see how to use the existing libnode to embed Node.js into a C++ program, you can check out some of our tests, for example https://github.com/nodejs/node/blob/main/test/embedding/embedtest.cc and https://github.com/nodejs/node/blob/main/test/cctest/test_environment.cc

jasongin commented 3 months ago

https://github.com/nodejs/node/pull/43542 adds a simpler ABI-stable API for embedding, that would make libnode much easier to use from other applications and programming languages.

nodejs / node