Spike: Transpile to System module loader

[josh]

I had this idea recently after talking with @wycats at the Extensible Web Summit (4/4/2014). He mentioned I should get involved here.

Assuming native or polyfilled System loader support, we should be able to transpile ES6 module syntax into ES3 syntax compatible System loader definitions.

ES6 syntax

// app.js
import $ from "jquery";
import _ from "underscore";

function start() {}

export { start };

System loader output

Promise.all([
  System.import('jquery'),
  System.import('underscore')
].then(function(imports) {
  var $ = imports[0];
  var _ = imports[1];

  function start() {}

  System.set('app', new Module({ start: start }));
})

I'm not totally sure on which Loader APIs this should use (I'm not quite sure I'm reading the latest agreed upon spec). Hopefully the gist of this idea makes sense.

TODO

• [ ] Figure out what to do with missing this.moduleName
• [ ] 10% Complete test suite

cc @thomasboyt @wycats @guybedford @dherman

[domenic]

This makes serious amounts of sense to me. It is a good decoupling and ends up encouraging people to write reusable System implementations that implement their desired semantics. (E.g. I can easily think of how to write System.import and System.set for RequireJS, and at least System.import for Node.)

[stefanpenner]

:+1:

[guybedford]

Great to see work in this direction!

The only issue with this approach is that module registration is asynchronous, which can allow a race condition if a user attempts to import app while jquery is still being loaded. In this scenario, we then lose the benefit of having a bundle.

The solution to this we've been working on is a registration function, System.register which would be able to avoid the race condition.

There is an ES6 Loader implementation of this here - https://github.com/systemjs/systemjs/blob/master/lib/system-register.js.

The template for this function is:

System.register(moduleName, unnormalizedDepArray, function(normalized, deps) {
  System.get(normalized);
  System.get(deps);
  return Module({...});
});

The benefit of this form is that it can also support circular references since System.get is an executing call.

There is actually a minor caveat in this that the circular reference support through System.get only works out if there is a spec change which I have proposed to @wycats. But even if this is not taken on board, a require argument would still allow circular references to work out.

Hope that is not too much of a mouthful for one comment! Let me know if you have any questions at all!

[josh]

(E.g. I can easily think of how to write System.import and System.set for RequireJS, and at least System.import for Node.)

@domenic yeah, I feel like RequireJS should ultimately define its modules onto System's registry if its available instead doing it internally.

@guybedford Rad, System.register was more what I was looking for. Is that going to be standardized? Or maybe at least safe to assume people use SystemJS for this transpilers purpose?

More of a es6-module-transpiler specific question, I'm not sure how to handle "anonymous module names". System.set (or System.register) are going to need an explicit module name. But it seems like this.moduleName can be blank. I'm not really sure what to do here.

[guybedford]

We're trying to at least make System.register a standard loader API convention, with implementations in SystemJS and Traceur currently. I don't think it makes sense for it to be in the spec, since it will go away in time anyway (a long time, but time none the less!).

Feedback on this current spec suggestion is probably the biggest hurdle to being able to completely confirm the exact API with respect to these circular references.

@josh it could be worth having an anonymous form of System.register as well:

  System.register(unnormalizedDeps, function(normalized, deps) {
    return Module(...);
  });

In this way it really is a complete AMD define analogy.

[caridy]

I do like the promise variation. I feel it is way better than the infamous System.register(), especially because we are not introducing the alien API under System, an the fact that it is readable and easy to understand, but there are a couple of issues that we should keep in mind though:

• Semantically, ES modules are not bound to a loader instance, and by doing System.register() or System.set() we will be doing just that. This is not a blocker since we are treating that transpiled module as a dynamic module (define() and YUI.add() have the same issue).
• It will not work for anonymous, but that's the less of my concerns at this point since the transpiler can name them.

[caridy]

I just recall another issue, you can't set() a module if that module is in the middle of the fetching/execution process, but I think we can have a workaround for that at the loader level if we know we are loading a module that uses the Promise.all() and the imperative syntax for import to load the dependencies. @guybedford can you validate that this is possible with the current polyfill code?

[guybedford]

@caridy System.set is designed to allow overriding modules fine so that isn't an issue at all in the module pipeline. The only issue would occur if you are defining app, when an import to app has already been made. Because modules set after already starting an import but before that import has completed throw an assertion error (15.2.5.2.5 2a).

For me the critical arguments for System.register here are firstly supporting circular references, secondly the race condition.

[josh]

I'm :smiley: that others are just as excited by this direction. If anyone else with spare cycles wants to take this over, I'd be happy to stand by and assist. I wouldn't consider myself an ES6 modules expert.

Otherwise, I can continue hacking with your guys help.

[caridy]

@guybedford that's precisely the one that I was referencing to (https://people.mozilla.org/~jorendorff/es6-draft.html#sec-finishload), and what you describe is what is going to happen. You will do System.import('app') and at some point in between that and the finishload, a System.set('app', mod) will be invoked.

It is a bummer that circular references will not work though, but maybe I'm wrong here.

[wycats]

@guybedford can you explain the specific cycle problem you tried to explain to me over dinner?

[guybedford]

@wycats there is no problem with cycles for dynamic modules, this works out fine, I had misunderstood the instantiate interpretation. The suggestion I'm making is to solve the issue of deferred execution for dynamic modules, which is not possible in the current spec since they execute during linking only, and in the process simplify the handling of circular references as a byproduct. See https://gist.github.com/guybedford/10328654. Please do let me know how I can clarify this further.

@josh sorry to hijack the issue, and thanks for sparking these discussions.

[caridy]

@guybedford thanks for the link. most of that makes sense to me. I think the key here is to have a way for a dynamic module to specify its named exports, (exports: ['these', 'are', 'the', 'export', 'names'] in your gist). Although, I think this might be orthogonal to the defer execution.

[guybedford]

@caridy yes providing the exports upfront is exactly what allows deferred execution to work through a shell module constructed at Link. This is because a fully-defined module object with all its exports is expected at the Link stage, so AMD modules are currently being executed into their full module object during Link instead of EnsureEvaluated.