Any signal about supporting ESModules?

[Jack-Works]

I need to run ESModule code in my project. But realms doesn't support it currently. I have the willingness to resolve this problem but I don't know if it is acceptable for this project.

Proposal:

1. I'll use TypeScript's compiler to compile ESModule into SystemJS Module ( an implementation example: https://github.com/Jack-Works/loader-with-esmodule-example/blob/master/src/typescript/typescript-shared-compiler.ts )
2. After ESModule is translated into SystemJS Module, I'll provide a fake System object to execute and get the binding of the module.
3. I'll expose an import handler that allows Realms to resolve each import statements. For example:

   const options = {
   importHandler(origin, importSrc) {
       if (importSrc === 'std:virtual-scroll') return import('std:virtual-scroll')
       else return fetch(new URL(importSrc + '.js', origin).toString()).then(x => x.text())
   }
   }

4. I'll also handle the dynamic import() by the transform functions. But static import is only supported on the top level of the file.

This will make the Realms shim slower and bigger. Is that acceptable for realms-shim? If not, I will do that in my own project instead of contributing it to the upstream

[michaelfig]

Hi,

Interestingly enough, I've been working on something that may suit your needs. It's not quite finished yet, but transform-module is an import expression and module body rewriter that uses Babel. It works for all import/export syntax.

The module rewrite it uses was designed by @erights and I.

My next goal is to get make-importer up-and-running to complete the picture.

But yes, transforms were specifically designed for supporting ESModules, so you're on the right track! I don't think the module support will be folded into realms-shim, but feel free to create a wrapper that uses your transforms and importer. That's what I intend to do.

Pick your poison: Typescript or Babel. Offering users diversity in this regard is worthwhile.

[Jack-Works]

I have just make a proposal for it. Check it out and give some support!

https://github.com/Jack-Works/proposal-ecmascript-parser

[Jack-Works]

Hi,

Interestingly enough, I've been working on something that may suit your needs. It's not quite finished yet, but transform-module is an import expression and module body rewriter that uses Babel. It works for all import/export syntax.

The module rewrite it uses was designed by @erights and I.

My next goal is to get make-importer up-and-running to complete the picture.

But yes, transforms were specifically designed for supporting ESModules, so you're on the right track! I don't think the module support will be folded into realms-shim, but feel free to create a wrapper that uses your transforms and importer. That's what I intend to do.

Pick your poison: Typescript or Babel. Offering users diversity in this regard is worthwhile.

Okay... Not a built-in one... But we can try to do this to enable import transform if developers already have typescript or babel installed

let enabledImport = false
try {
   enabledImport = ['typescript', require('typescript')]
}
catch {}

[michaelfig]

I read the URL you pointed to, and wanted to ask: are you targeting only the browser? Or are you aiming for a kind of SystemJS support that would be compatible with Node.js and other runtimes? The goal for the ESM system I'm working on is to have something which would be independent of the particular ECMAScript environment.

[BTW, It seems that there is also a Babel transform for ESM-to-SystemJS, so that may be a worthwhile second translation front end for you if you want to offer diversity, regardless of your target.]

I must admit to unfamiliarity with SystemJS, so I would be interested in hearing about the tradeoffs of your design. Feel free to take this offline to mfig@agoric.com unless you'd like to respond publically.

Thanks, Michael.

[michaelfig]

Okay, rereading your comments leaves me with the understanding that this could be independent of ES runtime. Interesting!

[Jack-Works]

Ah. I'm not familiar with SystemJS before I do this. I choose SystemJS because import.meta is available when translate target is ESModule or SystemJS in TypeScript compiler. If I choose CJS or AMD, it cannot use import.meta anymore

[caridy]

What you explained in the description seems to be something that you should do today, and if you can't, then just let us know where the problem lies, and we can try to address it.

SystemJS format is a script sourceText, which can be evaluated inside a realm. The SystemJS global references can be installed in the realm, where you can intercept those operations and carry on the proper operation under the hood.

[Jack-Works]

I have already done this before, but not for realm. I just wonder if I do this for realm, will realm accept it as a pr

[erights]

Where can I read about the transformation and the API of the System object that it assumes?

[caridy]

@Jack-Works I don't think we want to add SystemJS support to the realms shim as a out of the box feature, but certainly you can create another project on top of the realm to add support for it... seems very straight forward. /cc @guybedford

[Jack-Works]

No, I'm not supporting SystemJS, but internally transform ESModules into SystemJS so we can run all ESModules files in the sandbox.

Here's an example:

import x from './dep-a.js'
import { x1 } from './dep-2.js'

export function main(path) {
    return import(path)
}

Will be transformed to

System.register([], function (exports_1, context_1) {
    "use strict";
    var __moduleName = context_1 && context_1.id;
    function main(path) {
        return context_1.import(path);
    }
    exports_1("main", main);
    return {
        setters: [],
        execute: function () {
        }
    };
});

And we wrap it in a wrapper

;(function (System) {
    // translated code here.
})

Then the ESModule file will return a function, and we can let all its dependencies run in the realms.

const currentFileModule = translate(sourceText)
const fakeSystem = { register() { ... } }
const currentFileExecutedModule = currentFileModule(fakeSystem)
const dependencies = resolveAllDependenciesAndExecuteInRealms(currentFileExecutedModule)

[Jack-Works]

I have already done this in another project:

Wrap the translated module in a function:

https://github.com/Jack-Works/loader-with-esmodule-example/blob/master/src/typescript/typescript-shared-compiler.ts#L45

The fake SystemJS object, it's register function: https://github.com/Jack-Works/loader-with-esmodule-example/blob/master/src/typescript/typescript-shared-compiler.ts#L87

Resolve all dependencies and also translate them into the internal module representation:

https://github.com/Jack-Works/loader-with-esmodule-example/blob/master/src/typescript/typescript-shared-compiler.ts#L108

Transform all import declarations from import x from './some' to import x from '/compiler.js?src=./some' (not useful in the realms's case):

https://github.com/Jack-Works/loader-with-esmodule-example/blob/master/src/typescript/typescript-shared-compiler.ts#L195

Transform all dynamic import(x) calls to import((x => x[0] === '.' || x[0] === '/' ? new URL('/compiler.js?src=' + x, import.meta) : x)(x)) (also not useful in realm's case): https://github.com/Jack-Works/loader-with-esmodule-example/blob/master/src/typescript/typescript-shared-compiler.ts#L226

[Jack-Works]

Transform to another module standard because we can't just eval('import(...)'), it's a sandbox escape. We need to handle how to resolve and execute the module source.

Actually use any the internal format is okay, commonjs, amd, SystemJS. I use SystemJS becuase it is the only format for my compiler that can support import.meta syntax. (import.meta will be transformed into something like context_1.meta, and it comes from System.register)

[caridy]

@Jack-Works yes, that's precisely what I was saying above, you can do that in an abstraction layer on top of the shim to provide a shim implementation that supports SystemJS, don't need to be here. This shim is first and foremost a polyfill of the Realms proposal, plus a bunch of other things that help us to implement it and bend the rules to the limitations of polyfilling it. IMO, the support for SystemJS should be something that works with this shim and with the eventual native implementation... it is just a new layer that works independently of how the polyfill is implemented.

[Jack-Works]

I'm afraid you didn't catch my meaning. I'm not "supports SystemJS", I'm supporting ESModule (and system is just an approach ). Since ESModule is the standard module system of ECMAScript, it is meaningful to support it in the Realms shim.

[guybedford]

If there's anything I can advise on here let me know. Although Caridy is probably the right person to suggest / advise re SystemJS here as he was a co-contributor when we were developing the System format! The runtime with TLA support (although not based on the latest spec changes re timings yet) can be easily copied from here if necessary - https://github.com/systemjs/systemjs/blob/master/src/system-core.js, to create a scoped interpretation. But there are many ways to skin modules too. esm's implementation used another approach for live bindings which may be worth looking into. Reexports and cycles are the things to watch out for.

[erights]

@michaelfig and I are working on a different module rewrite and linkage scheme for SES. We expect to present this at the SES meeting on Thursday Sep 19. We should examine whether it satisfies your goals.

[erights]

From the example at https://github.com/Agoric/realms-shim/issues/47#issuecomment-532073248 I was not able to infer what the transformation is in general. Is the full transform explained anywhere?

[guybedford]

The format is described in https://github.com/systemjs/systemjs/blob/master/docs/system-register.md.

[Jack-Works]

Maybe I need to explain it again.

First, we want to run the following code in the realms without breaking the sandbox.

import x from './dep-a.js'
import { x1 } from './dep-2.js'

export function main(path) {
    return import(path)
}

To archive this goal, we need to make modules dep-a.js dep-2.js and the dynamic import run the code in the realms.

We can not run code with a static import in eval (Uncaught SyntaxError: Cannot use import statement outside a module). And run code with a dynamic import will cause a sandbox escape.

So we need to translate the code into another form(let's call it "internal module format") we can handle it safely.

To avoid confusion, I'll not use any existing module format. Let's transform it (use a compiler) into an imaginary module format "RealmsModule".

(function (RealmsModule) {
    RealmsModule.staticImport("./dep-a.js", "./dep-2.js", (_a, _b) => {
        const x = _a
        const { x1 } = _b
        RealmsModule.exportBindings.main = function main(path) {
            return RealmsModule.dynamicImport(path)
        }
    })
})

Let's call this file "index.(translated).js"

After this translation, we can handle all kinds of import statements in the source file.

Then we run the code in the index.(translated).js in the save eval function, we will get a function returned. Let's call this function "moduleExecutor". It will receive a "RealmsModule" object to handle its dependencies and exports.

Run the function "moduleExecutor" with RealmsModule object we will get RealmsModule.staticImport called with it dependencies and a callback. We can resolve all its dependencies recursively. After all modules resolved in the realms, we can call the callback to execute the module body.

When someone calls the main function in the index.js, it actually calls RealmsModule.dynamicImport so we can hijack it into the realms.

---

This is how I plan to support ESModules. Instead of creating a brand new module format, I reuse the existing module format so I don't need to write a compiler myself. I happened to choose SystemJS because the TypeScript compiler can compile the ESModule syntax import.meta into "context_1.meta", dynamic import into context_1.import()

[Jack-Works]

And yes, I prefer SystemJS as the internal module format because it "support the exact semantics of ES6 modules"

[Jack-Works]

I have implemented a class SystemJSRealm extends SystemJS implements Realm in my project and it works well (somehow) 🤔