concerns about realm sharing self hosted intrinsics

[allenwb]

If I understand correctly, the central idea of spawn is that realm can share immutable intrinsics and that TheFrozenRealm provides a particularly useful set of sharable intrinsics.

However, I'm concened that the proposal does not explore deeply enough the currently specified semantics of realms and intrinsic functions and that the current semantics actually invalidates assumptions made in this proposal.

The Summary section of the proposal starts out with the statement "a realm consists of a global object and an associated set of primordial objects...". That is an overly simplified models of realms. Clause 8.2 defines other state that is associated with each realm which would have to be accounted for when "spawning" a new realm. One of those fields reference a global environment record that is uniquely associated with the realm. Note that it is that global environment record that is used to resolve global references from code defined within the realm rather than the realm's global object directly.

Also note that intrinsic functions are specified to be Built In Function Objects which are permitted to be either ECMAScript function objects or an implementation provided exotic function objects. An intrinsic would be an ECMAScript function object when an implementation (or Realm creator, assuming the Realm API permits it) "self hosts" the intrinsic using ECMAScript code.

All functions that are Built In Function Objects are required to have a [[Realm]] internal slot. 9.2 defines the [[Realm]] internal slot of a function as "The realm in which the function was created and which provides any intrinsic objects that are accessed when evaluating the function. " ECMAScript function objects (including those used to implement intrinsics) also have a [[Environment]] internal slot that is "The Lexical Environment that the function was closed over.". Functions defined at the top level of a script have have their [[Environment]] set to the Global Environment Record of the realm within the creating script is running. Functions defined at the top level of a module have their [[Environment]] set to the Module Environment Record of the modules. All Module Environment Records are inner scopes of some Global Environment Record that is uniquely associated with some realm.

The implication of the above is that any intrinsics that are "accessed when evaluating a [intrinsic] function" would be resolved from the intrinsics of the realm for which the function was originally created and not from any realm with which it has been shared via spawning. Concretely a reference to %eval% would using %eval % from the original realm and never a spawned realm. Similarly any lexically scoped references (particularly to globals) from intrinsics defined using ECMAScript function objects will be resolved using the global environment record found on the function's [[Environment]] chain. This probably makes sense from a closure perspective. But it may be surprising to anybody who expects intrinsics to have access to the globals of the realm from which they were accessed.

Finally, the ES specification recently added a new internal slot [[ScriptOrModule]] to function objects (including all Built In Function objects) that references a Script Record or a Module Record. Script Record and Module Records are uniquely associated with a realm. It is apparently set to the script/module/realm of the job that created the function. [[ScriptOrModule]] is apparently used to set the active realm when the associated function is activated to start a new job (for example, if the function is used as an event handler??).

So, it appears that spawning realms with shared intrinsic function may lead to various situations where execution is associated with a realm other than the realm from which the function as accessed as an intrinsic. It is not clear whether this is expected/desirable/etc.

[caridy]

@allenwb I have encounter the issue described here when trying to spec this proposal. @erights, @FUDCo we will have to discuss the alternatives ASAP. It seems that the concept of "lightweight realms" can't be really that lightweight, at least not under the current circumstances.

[erights]

@allenwb @caridy

Leaving aside modules and [[ScriptOrModule]], I think Allen's analysis is correct. The consequences are surprising. But the only objects that are necessarily unique to a spawned realm are

• a global object
• an eval function
• a Function constructor

Given these, I don't see many ways for the surprise to actually result in bad surprises.

Once we enhance the spawning mechanism, in collaboration with the rest of the realm API, to allow overriding other intrinsics, then partial overriding of intrinsics can result in many more surprises. But the current proposal does not yet provide for overriding of other intrinsics. Here's an idea:

Add an optional second parameter to spawn, which is a record mapping from an intrinsic name to the object that is to be the value of the intrinsic of that name. Given that, there are two possibilities for the treatment of intrinsics not named in that record:

• All intrinsics not named in that record remain bound to the value of that intrinsic in the spawning realm. This is minimal but will create many such nasty surprises.
• If intrinsic A depends on intrinsic B and the record explicitly overrides B, then it implicitly overrides A as well to be just like the inherited A except that it depends on the overriding B, etc. This will be complex to spec well, but means that the dependency structure can be maintained to stay current with the platform, rather than being unreliably tracked by libraries.

[erights]

All speculative and half baked

I do not yet understand how modules and [[ScriptOrModule]] fit into this proposal. The frozen realm has no loader instance or only a transitively immutable loader instance, and so clearly has no non-builtin modules. If it does have builtin modules, clearly their module instances need to be transitively immutable. Regarding spawned realms, it must be possible to create frozen realm as a polyfilled substitute root (as in the polyfill example) where added module instances must be able to be made transitively immutable, and where the loader for that realm is also transitively immutable. We should also think about inheritance among loaders, perhaps drawing inspiration from inheritance among Java's ClassLoaders.

[caridy]

I haven't think much about modules in the context of the realms, and frozen realms, but one of the features that we have spec'd already in the loader API (work in progress) is the ability to define a new loader instance with a custom hook that will resolve modules from another 3rd party loader registry, as a result, you can create a membrane in user-land to import modules from other realms, and on top of that, if that membrane needs to be physical, you could create a reflective module record that can simply act as a proxy for each named export.

module instances must be able to be made transitively immutable

I don't know what that sense means, but the way I see it, importing a module from another realm is not different than passing endowments into the spawn call, I think of it as endowments on demand via a membrane defined by in user-land.

[allenwb]

Here are couple more potential concerns:

1. Having a realm share another's intrinsic probably isn't semantically a big deal in most cases given the current ES definition of realms. But if every realm was wrapped in a membrane like Spidermonkey apparently does, then every call to a built-in within such a realm is a membrane crossing. That sounds too heavy weight for many use cases.
2. Each realm has a templateMap which canonicalizes template site objects based upon the realm association of the ECMAScript function that contains the corresponding template literal. A problem??
3. %GeneratorFunction% probably needs to be treated just like Function constructor.

[erights]

But if every realm was wrapped in a membrane like Spidermonkey apparently does, then every call to a built-in within such a realm is a membrane crossing. That sounds too heavy weight for many use cases.

Already discussed this with Eric Faust and Shu (github @ links please). This is not the way to map Spidermonkey membranes to realms, for these and other reasons. Let's call the units that Spidermonkey wraps in a membrane "compartments". The tree consisting of a root realm (such as a frozen realm made by Realm.immutableRoot()) together with all its spawned lightweight descendant realms should all be in one compartment. None of the use cases motivating the placement of Spidermonkey membranes requires that compartments be any finer grain.

[erights]

Regarding the templateMap, I don't think there would be any problem sticking with the spec as already stated. In a realm spawning tree, each function is still associated with exactly one realm. It might as well use the templateMap from that realm.

Alternatively we could have a lightweight child realm inherit the templateMap from its parent by default, so that it would normally come from the root realm of that spawning tree. A concern here is whether we would need to admit in the semantics that this involves mutation, since the immutable root realm is supposed to omit any mutable state. I think the answer for the templateMap is no, but we should be very sure before sharing the one from a spawning tree's root.

[erights]

%GeneratorFunction% is the interesting one. The treatment of the Function constructor is specific to its role as something reachable by naming from a global object. Within a normal spawning tree, the Function constructor reached by (()=>{}).constructor will be the Function constructor of the root realm, not the overriding Function constructor. One reaches the override on by the global name Function. Thus, there's no place to put an overriding %GeneratorFunction% constructor.

This is already covered in README.md by the paragraph:

In ES2016, the GeneratorFunction evaluator is not a named global, but rather an unnamed intrinsic. Upcoming evaluators are likely to include AsyncFunction and AsyncGeneratorFunction. These are likely to be specified as unnamed instrinsics as well. For all of these, the above name-based overriding of spawn is irrelevant and probably not needed anyway.

[caridy]

Ok, I think I found the root of my confusion, question:

const o = someRealm.spawn();

Is o a "Realm Object" or a "Spawned Realm Object". Initially, I thought it was just a realm object (and somehow, we could make it a lightweight and without the identity discontinuity of the existing creation process), which seems to align with the thoughts from @allenwb as well, but it seems that @erights has something different in mind.

According to @erights messages, It seems that they are different. When spawning you just create the "Spawned Realm Object" that contains those 3 objects (global, eval and Function), plus an internal slot to the spawning realm's Realm Record, and relies entirely, or partially, on that. Although plausible, it seems to me that we will have to adjust the current spec considerable to allow this notion of a foreign "global" when evaluating code in spawned realm objects. I will investigate more.

[caridy]

To add more details, calling the abstract operation NewGlobalEnvironment, then setting the outer lexical environment reference of this new env to corresponding from the spawning realm. Plus having a way to run eval evaluator in the bound environment record associated to the new global environment, which is very similar to the existing PerformEval, might be sufficient to guarantee that any call to Eval from within or from outside will define new globals in this new global env.

[caridy]

@erights I need your input about "Realm Object" vs "Spawned Realm Object".

[erights]

@caridy I was not thinking of "Spawned Realm Object" as a different kind of object, or an instance of some class other than Realm. Rather, I was thinking of the adjective "spawned" as modifying "Realm Object" to describe the difference in how it is created, not in what it is.

However, now that you raise it, a plausible observable difference is that a spawned realm object might provide access to its parent realm object --- the realm object it was spawned from. If spawning attentuates authority, then we must not do this. But if spawning-plus-endowing only enhance authority, such that climbing the realm-parent-chain only attenuates authority, then such a parent pointer would be nice. This seems plausible to me. We could only resolve this looking at the Realm and Frozen Realm proposals together.