How to expose an API to multiple heaps/contexts?

[rcdailey]

I'm working on an app that has multiple, distinct "modules" (I'm not referring to modules mentioned in the duktape documentation). Each of these are scriptable with javascript, so I integrate duktape for this purpose. I want each module to serve as an automatic namespace, in that when global variables are defined, they will not interfere with scripts between modules. For this I think I need multiple heaps. However, at the moment I have a "common API" which is simply some C++ functions I bind to javascript so that the scripts can call my engine code. Right now I have 1 context per module in my system, which means for each module I need to re-run the interface binding code. I feel like this will duplicate memory between modules for the common API pieces.

To further complicate this, module C++ code may define more APIs that it can share with other modules. So you get into a web of APIs.

What would be nice is if I could have 1 heap shared between all modules that is used only to define the interface to C++ code, but it wouldn't be used to allow global variable definitions. And each module owns a heap for the global variables, to prevent scripts in modules from interfering with each other. The case I want to handle is, for example, when module A defines a global variable "foo" and module B defines a global variable "foo". Those should be 2 separate "foo" variables, but they both need access to C APIs exposed to javascript.

What is the best process for this? I didn't see any binding examples in the duktape programmer's guide. I wish that the act of mapping C functions to javascript didn't involve the normal runtime heap stuff. I apologize for asking a question in the issues section, I wasn't sure where else to ask questions. Thanks in advance.

[wenq1]

Register these common functions to each heap created.

[fatcerberus]

Register these common functions to each heap created.

Not necessarily - Duktape allows you to create multiple global objects that can share a heap. So you’d only have to add references to the same functions to each global object, without duplicating the function objects themselves.

In this way you could even share objects between the two contexts without serialization, but of course they would be isolated by default.

[rcdailey]

Is there an example somewhere I could look at? Thanks for the idea.

On Wed, Jan 31, 2018, 8:29 AM Bruce Pascoe notifications@github.com wrote:

Register these common functions to each heap created.

Not necessarily - Duktape allows you to create multiple global objects that can share a heap. So you’d only have to add references to the same functions to each global object, without duplicating the function objects themselves.

In this way you could even share objects between the two contexts without serialization, but of course they would be isolated by default.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/svaarala/duktape/issues/1841#issuecomment-361948486, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr6dnfRKEbhYdywTZUK7pg4OPVZ6bmrks5tQHi1gaJpZM4RzP9L .

[stevefan1999-personal]

Sadly, this is very hard to do, as Duktape is fully-reentrant, meaning that there're no global states and all states/contexts are isolated from each another. The exceptions are ROM-builtins and ROM strings, where all states would share them if enabled.

So if you want inter-state communication, your last resort would be an internal call through the use of JSON.

Let there be 3 states, state A, B, and Global. State Global would declare and require all the functions you needed. You create a call wrapper for state A and B for every referenced code to state Global in C (so maybe a hash table is needed), then by serializing the arguments to JSON, you call the global functions and serialize the return value once again and feed it back to state A and B respectively.

Well, this is effectively IPC...There are still some problems though, for example, cyclic data cannot be handled. You could definitely try some optimization technique such as context sandboxing, where you use a global state and you separate environment for each object when needed, but you will be limited to single-threaded application, although you should use DT single-threaded as ECMAScript itself is single-threaded.

[svaarala]

@stevefan1999 If multiple global environments share the same Duktape heap, they are actually not isolated from each other: although there's no easy Ecmascript way of sharing objects (or passing them around), you can easily do that from C code. ES2015 actually formalizes this: objects created in one realm may appear in another realm.

The caveats in this are that "foreign" objects may behave in somewhat unexpected ways. For example, if an Array object is passed from global environment A to global environment B, then:

• Array.isArray(x) in B will be true.
• But x instanceof Array will be false, because x inherits from A's Array.prototype, not B's Array.prototype.

[Wolfleader101]

for anyone curious about how to achieve this - I have found a solution

m_ctx is the global context that created the heap. Script just just a simple struct that contains duk_context pointer.

  duk_idx_t thread_index = duk_push_thread_new_globalenv(m_ctx);  //! creates a new heap and pushes a context onto it (does not copy global context)
        script.env_ctx = duk_require_context(m_ctx, thread_index);

        if (!script.env_ctx) {
            std::cerr << "Failed to create Duktape context" << std::endl;
            return script;
        }

        //! Register the global vars and functions
        duk_push_global_object(m_ctx);
        duk_enum(m_ctx, -1, DUK_ENUM_OWN_PROPERTIES_ONLY);

        while (duk_next(m_ctx, -1, 0)) {
            // Here, the top of the stack contains the property key
            const char* key = duk_safe_to_string(m_ctx, -1);

            // Fetch the associated value
            duk_get_global_string(m_ctx, key);

            // Move it to the new context
            duk_xmove_top(script.env_ctx, m_ctx, 1);

            // And set it as a global in the new context
            duk_put_global_string(script.env_ctx, key);

            // Clean up the key
            duk_pop(m_ctx);
        }

Hope this helps anyone who may be wanting to do something similar in the future!

[seropigeorgedev]

duk_push_global_object(ctx); duk_xcopy_top(test.ctx,ctx,1); duk_put_global_string(test.ctx,"global"); //or put it all into global object but I think it's better to switch to Squirrel or something with classes & better env control

[Wolfleader101]

duk_push_global_object(ctx);

duk_xcopy_top(test.ctx,ctx,1);

duk_put_global_string(test.ctx,"global");

//or put it all into global object but I think it's better to switch to Squirrel or something with classes & better env control

What library are you referring to that has better env control and classes?

[seropigeorgedev]

duk_push_global_object(ctx);

duk_xcopy_top(test.ctx,ctx,1);

duk_put_global_string(test.ctx,"global");

//or put it all into global object but I think it's better to switch to Squirrel or something with classes & better env control

What library are you referring to that has better env control and classes?

My apologies it seems that a miscommunication occurred, what I meant was switch to a different scripting engine for example Squirrel (uses Squirrel lang not js) which offers both classes and a very good env control because the global env is basically a table (object)

[Wolfleader101]

duk_push_global_object(ctx);

duk_xcopy_top(test.ctx,ctx,1);

duk_put_global_string(test.ctx,"global");

//or put it all into global object but I think it's better to switch to Squirrel or something with classes & better env control

What library are you referring to that has better env control and classes?

My apologies it seems that a miscommunication occurred, what I meant was switch to a different scripting engine for example Squirrel (uses Squirrel lang not js) which offers both classes and a very good env control because the global env is basically a table (object)

No you're all good mate, got what you meant!

Yeah there's not really to many easy to implement JS engines that can be embedded into a C++ project :( Only other one that I've tried was V8 but that's not as easy to integrate as duktape and is huge!