oj and ES6 modules

[iccir]

ES6 introduces the concept of Module and Script goal symbols. When a source file is parsed with the Module goal:

1. import and export are allowed at the top-level
2. await is a reserved word
3. Strict mode is automatically enabled

Unlike existing module systems, ES6 Modules are designed to be static. All information about the dependency tree can be know after a parse. This is an improvement: previous module systems were dynamic and resolved at runtime, which conflicted with oj's compile-time philosophy. Due to this static nature of ES6 modules, it's theoretically possible to integrate them tightly with OJ.

However, there are some philosophical differences between ES6 modules and oj's C/Objective-C heritage.

1. export must be used with a Function/Variable/Hoistable declaration (See section 15.2.3). OJ class identifiers aren't any of these, they are in a special namespace that can be messaged to (this is the same as Obj-C, see my comments in #81). The closest is [Foo class], but that's a runtime object and ineligible for the static nature of ES6 modules. OJ consts and enums are also special identifiers which live in the compiler, due to the way that they can be inlined.
2. Modules don't allow star imports into the current namespace/environment, whereas #import/#include/header files only allow star imports.

Assume a file with a Foo class, FooType enum, and several FooName constants. In Objective-C, all of these can be imported via a single #import. In a hypothetical world where oj integrated with the modules natively, imports would be verbose and explicit:

import { Foo, FooType, FooType1, FooType2, FooType3, FooType4, FooName1, FooName2… } from "Foo";

compared to the equivalent in Obj-C:

#import "Foo.h"

We've used a similar approach to this in the past, and discovered that it's quite unwieldy when porting a large Obj-C source base to oj.

There's also a lot of incompatibilities with the as syntax. For example, if Foo is a JS class rather than an OJ class, we could use it like this:

import * as Bar from "Foo";
let foo = new Bar.Foo()

or:

import { Foo as Bar } from "Foo";
let foo = new Bar();

However, this is completely incompatible with oj's concept of global identifiers. Bar.Foo can never be an oj class. Likewise, this is also incompatible with the philosophy of inline consts and enums. Assume Foo.oj has a InlinedEnum enum which is inlined and PlainVariable.

import { * } as Bar from "Foo";

// Legal, from my understanding, as there will be a PlainVariable property on Bar.
console.log(Bar["Plain" + "Variable"]);

// InlineEnum is suppose to be an inlined identifier and its name should never appear in compiled code.  What should either of these do?
console.log(Bar.InlinedEnum);
console.log(Bar["Inlined" + "Enum"]);

I like the ES6 module syntax, as it's much cleaner than existing solutions. However, I think it's incompatible with the concept of oj's global realm for classes/consts/enums/globals/squeezer. Even if oj were to integrate tightly with modules, all files would still need to be compiled at the same time, in order for the compiler to keep track of which identifiers are JS ones and which are oj ones.

That said, oj should co-exist with the ES6 module system, and allow import/export to pass through such that bundlers/transpilers can convert modules into ES5 single-file scripts. You should be able to export JS functions / variables / JS classes from oj files, and import them into other files.

I see two things we can do to enhance compatibility:

1. There needs to be a compiler flag that allows import/export to be passed through to the output files. This means that esprima is put into module mode. Optionally, this could be a module-strategy flag which takes a string, and we could introduce additional strategies in the future.
2. There needs to be a compiler flag which adds the generated source code per file to the results object, as bundlers may need to see the files individually rather than already concatenated. Use of the on-compile hook in oj2 may be too early for some bundlers.

[IngwiePhoenix]

I like the way you detailed that, and it makes the differences very clear.

First, I totally agree that there is no way that OJ will "natively" work with ES6 due to the "global" identifiers. Using module-strategy would let users, such as me, still use ES6's module system while being able to use OJ. Currently, I store the OJ state in my WebPack compiler instance (makes sense; attaching OJ's compiler state to the WebPack compiler). So I always get the names and identifiers resolved.

I am not perfectly sure on the 2nd option you pointed out. True, bundlers like to know the files - but usually, the idea is to transform a single file at a time. Here is how WebPack does it:

• Instantiate compiler
• entries.forEach(...) - Compile each entrypoint.
• Transform the entry point through a loader. For instance: {entry: "./App.oj"}
  • oj-loader gets the source contents of App.oj and transpiles it, returns the generated JS.
  • Additional loaders can be chained. For instance, Babel.
• Find require(), require.ensure() and define() statements in the obtained source, and repeat the transform step for each encounter.
• Build a dependency graph from all the files and plot a model of the dependencies.
• Build one or many chunks, depending on the module strategy used in the source code.

I do not know about Rollup and the others out there, but Browserify and WebPack are similar as they only take one input source at a time to a loader/transform and expect proper ES5 code in return.

In my case, I can access the WebPack compiler, attach OJ's state and cache to it, and keep the state around for each compilation. This is probably how the others work, too.

So, OJ should easily be able to maintain state across files if a loader/transform is written properly.

Unfortunately I am not 100% on the import and export statements myself.

But usually, these define the order of files in a bundle. If file a requires b, which requires c, then we'd get: [a, b, c] (top to bottom resolution). That means, that currently, I have to either use a preprocessor or a require statement, to have it work. So I am not sure on a proper solution but since you mentioned it, I might as well add that.

[iccir]

I am not perfectly sure on the 2nd option you pointed out. True, bundlers like to know the files - but usually, the idea is to transform a single file at a time. Here is how WebPack does it:

import and export must be at the topmost-level of a file.

Consider the following, in which Foo and Bar are JS functions or JS classes:

Foo.oj

…
export default Foo;

Bar.oj

import "Foo";
…
export default Bar;

If oj passes through the import/export statements and also concatenates the output, then you end up with:

Output.js

…
export default Foo;
import "Foo";
…
export default Bar;

You now have multiple exports at the root-level.

If instead you add IIFEs in the output:

(function() {
…
export default Foo;
}());
(function() {
import "Foo";
…
export default Bar;
}());

You've produced invalid ES6, as import/export cannot be located anywhere other than the top-level of a file.

[iccir]

Transform the entry point through a loader. For instance: {entry: "./App.oj"} oj-loader gets the source contents of App.oj and transpiles it, returns the generated JS. Additional loaders can be chained. For instance, Babel.

I've mentioned this before, but compiling individual files may cause subtle issues. For example, assume you have App.oj and ViewController.oj.

App.oj

let c = [[ViewController alloc] init];

ViewController.oj

@implementation ViewControllllllllllllller
…

You compile everything and get a runtime error in App, as ViewController is undefined (you misspelled it). You go back and correct the typo in ViewController.oj.

At this point, the next compile must recompile App.oj, as the oj global realm has changed, even though you haven't altered App.oj at all. The compiler needs to know that ViewController is now an oj class. Similar issues can result with typos in @const/@global/@enum.

oj 2 keeps track of all of this for you, and recompiles files as needed.

Going back to 2: the issue is that oj right now always outputs a concatenated file. The new on-compile hook allows you to see individual contents, but there probably should be a flag that also includes the individual contents in the output.

[iccir]

But usually, these define the order of files in a bundle. If file a requires b, which requires c, then we'd get: [a, b, c](top to bottom resolution). That means, that currently, I have to either use a preprocessor or a require statement, to have it work. So I am not sure on a proper solution but since you mentioned it, I might as well add that.

ES6 import and export work through static analysis, no order file is needed. Check out the "Design Goals" section here: http://www.2ality.com/2014/09/es6-modules-final.html

[IngwiePhoenix]

I see. Thank you for your insights.

Plus, Im sorry for replying so late. I have had to do a lot of schoolwork and barely got to hang out in the open-source world. :)

Currently I am moving my project to use Babel on the backend and frontend, allowing me to have very consistent code across the two. I would like to use this consistency in OJ too.

I can see the issue with current OJ concatenating all the files together. Though, due to the hacky way I run OJC, I actually get around that. I am looking forward to oj2 to see how it'll perform. Currently, I am actually able to recompile dependant files, no problem. This works by telling WebPack that file x belongs to file y, and it actually recompiles the dependant files as well. That means that I usually end up with no errors during recompilation. The state of compilation is stored within the WebPack compiler, allowing it to keep track of the files that run through it. The require calls simply add the files to the bundle. Well, speaking of require. Currently:

require("Dialog");
[Dialog showMessage:"foo"];

That is how I am currently getting it done. ES6 would probably make this look nicer:

import "Dialog";
[Dialog showMessage:"foo"];

Anyway. I wanted to ask if you have decided on a solution for using import and export with oj.

[IngwiePhoenix]

I moved my project completely - and I am pretty happy so far. Only one little bit is not moved yet, but that will happen soon. Though, the above approach is almost the same, except that I once again ran into the issues of my bundler and had to hack my way around it. (See this test. Removing var Dialog = causes Dialog to be undefined.)

I still have to require() things, although it seems that WebPack2 is progressing well, which will have tree-shaking support.

So, it'd be nice to know if you have made a decision here yet :)

[iccir]

Sorry, I got swamped and forgot to reply :) What did you do in your local repository? I'm probably going to just have a script vs. module flag in the compiler in 2.x.

[iccir]

Option parser-source-type will be passed directly to Esprima as sourceType.