Building node as a shared library instead of an executable

[kobalicek]

I think that there should be something like libnode.so or node.dll and node executable which simply calls an entry point in the shared object. Then, C++ addons can simply link to a library instead of linking to an executable.

What do you guys think about this?

I remember that there was an option in gyp for that, but all distributions use the executable-only model and I remember the option didn't work out for me even in the past.

I think that node can be used for much more things than just writing servers and having a model that other applications can use seems important. I really think that node can be embedded in other applications. It would be interesting to see more native addons wrapping other libraries and also native addons depending on other native addons (is this possible today?).

Thanks

[bnoordhuis]

I'm afraid I don't have time to summarize it but see joyent/node#7336 for background; the short version is that it's not that simple (unfortunately.)

[kobalicek]

Thanks, interesting discussion.

I think that in such case node shouldn't allow to register custom constructions when building node itself, this responsibility can be moved out and people embedding node would just:

1. Create the environment
2. Register modules
3. Use the environment

The problem I see is that if I create two node addons they export the same symbol. This can be fixed by introducing a naming schema, so for example if the addon is called myaddon the initializer would be something like node_myaddon_init exported as "C". In that case it would be just fixing the loader to hook up the right symbol based on the shared object name, and moving the responsibility on the embedder to call that function when creating the environment.

Well, I think that I see it simpler than it actually is, but I also think this can be fixed.

[rvagg]

It's kind of sad that it was dropped from joyent/node, this is an interesting concept and could be a good addition to the roadmap, it could be great for expanding Node's reach. If there are technical problems then perhaps they can be dealt with simply by limiting the scope of what a shared library is actually able to do. I fear that some of the objection to landing this feature is mostly about waiting for Node internals to be perfect.

@kobalicek (and others), could you outline some of the use-cases you imagine for this?

[Qard]

That "perfect" certainly deserves some quotation marks. I don't think the node codebase will ever fully escape the current hackiness.

[rvagg]

@Qard indeed and it should be embraced lest we be crippled by it and have trouble getting simple releases out the door ... >cough<

[Qard]

Agreed. The pursuit of a clean and friendly codebase is a noble cause, but not at the expense of progress. :)

[kobalicek]

Basically, I would like to see node as a runtime rather than application. For me it would be really interesting to create a native C++ addon for UI and experiment with doing some UI stuff with node. I mean, I don't see it worse to ship some app with libnode instead of libQt.

I'm working on a vector graphics library (blend2d.com) and I have a working node addon, would be nice if it can be used for more than some server image processing in the future ;) However, I see that maybe node is not a right choice for these kind of experiments. I myself see the whole addon system a bit hacky.

But, my question remains, is it possible to create a node addon that depends on or extends another addon?

[rvagg]

Yes, it's sort of possible to make node addons work with each other including depending on each other although it's not done in practice. I did some experimenting with this in https://github.com/rvagg/node-downer-rangedel and the main problem is simply the way npm orders installs and builds but that's mainly due to it not being on the development radar for npm at all.

[kobalicek]

Guys how can I help with this? I'm still definitely interested of having node as a shared library by default.

[bnoordhuis]

@kobalicek I think the following practical issues need to be resolved:

1. Fix auto-registration of built-in native modules.
2. Make the build type for the 'iojs' target in node.gyp configurable, it's always 'executable' now.
3. Add --enable-static and --enable-shared switches to the configure script.
4. Fix the build system so it can build static and shared libraries at the same time (nice to have, not mandatory.) Perhaps it's possible to define an iojs_base target that's depended on by iojs_static and iojs_shared targets.

We can evolve the API as the need arises.

[mhart]

Does https://github.com/iojs/io.js/pull/1341 now close this?

[bnoordhuis]

@mhart Not yet. You can build io.js as a static library now but not a shared library. It may be as easy as setting node_target_type to 'shared_library' but I haven't tested that.

[mhdawson]

@bnoordhuis to fully close would we include delivering shared libraries as part of the binary drops ? Is that part of what you mean for "Fix the build system" above ?

[bnoordhuis]

@mhdawson That wasn't my intention. If building as a shared library works and the resulting .so is usable(ish), I'm happy.

[mhdawson]

@bnoordhuis understood. Has there been any prior discussion of whether we should ship shared libraries in addition to the binaries ?

[kobalicek]

I think it should be either static or shared, but not both.

[bnoordhuis]

Has there been any prior discussion of whether we should ship shared libraries in addition to the binaries ?

Not to my knowledge.

[mhdawson]

@kobalicek why only one or the other ? There are advantages/disadvantages to both

[kobalicek]

@mhdawson I think it's always good to avoid diversity, can't imagine how native modules will deal with this. I think the biggest example is what happened in io.js that just used different "node" name.

[mhdawson]

It looks like the changes required to support a shared library are along these lines. We'd want to refactor a bit so that the changes only apply when building a shared library but it captures what would need to be changed. Once we get agreement that we want to do this we can put together an refactored version as a PR.

diff --git a/common.gypi b/common.gypi
index 4c1b90b..84eaef2 100644
--- a/common.gypi
+++ b/common.gypi
@@ -202,7 +202,7 @@
         'ldflags': [ '-pthread' ],
       }],
       [ 'OS in "linux freebsd openbsd solaris android"', {
-        'cflags': [ '-Wall', '-Wextra', '-Wno-unused-parameter', ],
+        'cflags': [ '-fPIC','-Wall', '-Wextra', '-Wno-unused-parameter', ],
         'cflags_cc': [ '-fno-rtti', '-fno-exceptions', '-std=gnu++0x' ],
         'ldflags': [ '-rdynamic' ],
         'target_conditions': [
diff --git a/configure b/configure
index f101ef8..ed421ec 100755
--- a/configure
+++ b/configure
@@ -325,6 +325,11 @@ parser.add_option('--enable-static',
     dest='enable_static',
     help='build as static library')
+parser.add_option('--enable-shared',
+    action='store_true',
+    dest='enable_shared',
+    help='build as shared library')
+
 (options, args) = parser.parse_args()
 # set up auto-download list
@@ -677,6 +682,8 @@ def configure_node(o):
   if options.enable_static:
     o['variables']['node_target_type'] = 'static_library'
+  if options.enable_shared:
+    o['variables']['node_target_type'] = 'shared_library'
 def configure_library(lib, output):
   shared_lib = 'shared_' + lib
diff --git a/node.gyp b/node.gyp
index 2b530f1..a198bf6 100644
--- a/node.gyp
+++ b/node.gyp
@@ -397,7 +397,7 @@
           ],
         }],
         [ 'OS=="freebsd" or OS=="linux"', {
-          'ldflags': [ '-Wl,-z,noexecstack',
+          'ldflags': [ '-Wl,-z,noexecstack,--allow-multiple-definition',
                        '-Wl,--whole-archive <(V8_BASE)',
                        '-Wl,--no-whole-archive' ]
         }],

[mhdawson]

@kobalicek are there some specific issues that you think would be different between a static library versus shared so that I can better understand ?

[kobalicek]

I think there can be only build issues of native modules, I'm not sure how node-gyp handles this, but I know I have to use pangyp when building my addons for io, so yeah, I think this can happen also here.

[mhdawson]

@kobalicek are these issues different between using a shared library and a static one ? Do you have a test case that shows one ?

@bnoordhuis what's your take on doing static, shared or both ?

[mhdawson]

I'm hoping we can get back to this. I think the next step is that we prepare a Pull request adding support for shared libraries and we can discuss there.

[davedoesdev]

This is at a slight tangent but I've been building a libnode.so for OSv for a while:

https://github.com/cloudius-systems/osv-apps/tree/master/node

It works fine. Needs tweaking for 4.x but should be okay.

[vroad]

libnode.so builds and works on Android, but some manual adjustments were needed. (https://github.com/nodejs/node/issues/3074).

[drom]

Any progress on this front?

[yan-foto]

If I'm not totally mistaken, GitHub's Electron does something like this. You can find libnode.so in the released archive of Electron. They have their own fork of node. Also take a look here.

[ChALkeR]

@yan-foto Btw, that's more like a patchset than a fork — it's 19 commits (doing various things) re-applied on top of Node.js on Node.js releases. It's synced with 6.1.0 now.

[ChALkeR]

Ah, also see https://github.com/nodejs/node-v0.x-archive/pull/6744 and https://github.com/electron/node/issues/2.

[mhdawson]

See https://github.com/nodejs/node/pull/6994 for current work. I'm hoping we can get this in as the first step in the next few weeks.

[Trott]

Closing all issues in this archived repository.