Traceur Node API modularity

UltCombo commented 10 years ago

Could we have a more modular Node API?

As long as Traceur is modifying globals, it won't be of use to upstream packages. Thinking a bit more about it, even a simple require('traceur') will modify primitive prototypes, so I guess achieving modularity is close to impossible at the moment.

I haven't tried to use the Traceur runtime+compiled modules in Node yet, but I believe that including the runtime would have the same issue with overwriting globals/built-in prototype methods, right?

For the time being, installing Traceur only in the bottom-most downstream package seems to be the only workaround to avoid these overwriting conflicts, though of course this still means that it is impossible to install dependencies which "depend" on different versions of Traceur side-by-side.

(moved some inaccurate assumptions to https://github.com/UltCombo/traceuroso/issues/1 and added some proper explanation)

johnjbarton commented 10 years ago

To me this is a problem with the node makeDefault solution. If all of the packages were written in ES6, then no such problem would exist.

Also we have many more pressing issues to be honest.

UltCombo commented 10 years ago

@johnjbarton Yep, I'm aware that Traceur usage as a node package is not the main focus of this project, and as I said, this is close to impossible to have a proper fix AFAICT.

I was just trying to make modular packages as well as trying to abstract away the ugliness, see my implementation and accompanying documentation.

I wrote this mainly as a beacon for those trying to achieve the same goal as me (using Traceur for modular packages), as well as organizing my ideas and getting some feedback if possible.

arv commented 10 years ago

I agree that using Traceur in Node is too hard/hacky.

[ ] Remove eval. Right now we load the script as text and eval it
[ ] Generated code depends on a global $traceurRuntime. When compiling to CJS modules we could have this be require('traceur-runtime@version') or something like that.
[ ] traceurRequire does not work with relative paths.

UltCombo commented 10 years ago

Thanks for looking into this, @arv.

Remove eval. Right now we load the script as text and eval it

What would be the alternatives? Compile, save to disk and require()?

Generated code depends on a global $traceurRuntime. When compiling to CJS modules we could have this be require('traceur-runtime@version') or something like that.

Sounds like an excellent idea. =]

arv commented 10 years ago

What would be the alternatives? Compile, save to disk and require()?

Use the Node.js module loader

UltCombo commented 10 years ago

Cleaned my previous comments a bit, moving some inaccurate clutter to https://github.com/UltCombo/traceuroso/issues/1.

guybedford commented 10 years ago

Would I be right in thinking that the use case here is to publish modular ES6 code to npm for consumption?

If so, this would be my ideal worklow:

In development, we run devindex.js which uses traceur. Alternatively just run traceur index.js.
For publishing to npm, we build all our files separately into CommonJS. For this to work, we need to store all our ES6 modules in a subfolder, lib-es6, and build this entire folder into CommonJS, lib-cjs. Our index.js can then reference this folder.

Such a workflow currently breaks down when loading node core modules due to https://github.com/google/traceur-compiler/issues/818, which I still really want to work on if there is interest.

UltCombo commented 10 years ago

@guybedford

Would I be right in thinking that the use case here is to publish modular ES6 code to npm for consumption?

Yes, exactly! =]

If so, this would be my ideal worklow [...]

Yes, I agree that storing compiled files would provide a significant improvement to published packages' performance. However, for the time being I'm focusing on the development and skipping build steps to make things simpler. I've started working on a package which is basically a re-usable small abstraction on top of traceur.require.makeDefault() (see traceuroso), it has no problems loading node core modules.

I believe both workflows (pre-compiling and compiling during runtime) should be supported, I'll comment on your linked issue as well.

briandipalma commented 10 years ago

It easy to get this to work, I wrote a simple generator that sets up projects like this https://github.com/briandipalma/slush-node-esnext and an example of a project generated with it https://github.com/briandipalma/global-compiler

UltCombo commented 10 years ago

@briandipalma that's useful, thanks. Though, the topic here is more focused on the Traceur issues that affect project such as ours (e.g. Traceur runtime being a global) rather than scaffolding new projects. ;)

For instance, installing a package which uses traceuroso and another scaffolded with node-esnext in the same project wouldn't play out very well, as these depend on different Traceur runtime versions.

guybedford commented 10 years ago

@UltCombo there are a number of issues that come with the approach of shipping ES6 + traceur for compilation. You now need Traceur locked to the right version for each individual module, you have global issues as you have said, and startup times increase. NodeJS startup times are bad enough, and server start time is key for a good deployment process and development experience. The approach doesn't scale.

If you compile modules into CommonJS as part of your publish step only, locking down just the traceur runtime to the right version, then when we finally get modular runtime it would work in this full ES6 modularity scenario.

The only thing you would need here is an npm prepublish script to run the build and set the right index, so there is effectively no custom build step necessary at all.

UltCombo commented 10 years ago

@guybedford Personally I don't think Node.js startup times are that bad, but I agree that the time taken to load the full Traceur compiler growing in O(n) times (where n=number of ES6 packages), added with the time to compile each module which varies according to the module's code size/complexity, does not scale. I'm willing to create a generator/scaffolding with traceuroso + prepublish script once I finish working on traceuroso's basic functionality, but that's another topic. As said before, I support the workflow you described previously.

samcday commented 10 years ago

Stumbled across this ticket. This is something I'd love to see happen.

So far the discussion seems to be revolving around having compiled Traceur code consume relevant bits of $traceurRuntime via package imports or the like. How would the polyfills and shims be handled though? If I bring in two libraries that each bring in Traceur runtimes for 0.0.50 and 0.0.55 respectively, which one "wins" when it comes to providing the System implementation? Or Symbol? etc.

johnjbarton commented 10 years ago

Two different ideas are discussed here.

Some of the discussion concerns multiple versions of runtime functions. In my opinion that is not possible by design.

Another issue is allowing only required runtime functions to be loaded. Each transformation would add the functions it needs to a list and the list would be prepended to the output. First step is to support this manually in the make file.

UltCombo commented 10 years ago

@johnjbarton

Some of the discussion concerns multiple versions of runtime functions. In my opinion that is not possible by design.

Would you mind expanding a bit on that? As far I can see:

Globals (new constructors, new primitive prototype methods) would still conflict/overwrite;
The correct $traceurRuntime version could be added as a dependency of all compile files as suggested by @arv. Having the $traceurRuntime version match the compiler is the most important part into getting modular ES.next packages.

If we can't add the proper $traceurRuntime version as a dependency into compiled files, it would be impossible (or very hacky) to have @guybedford's suggested workflow of distributing pre-compiled packages.

johnjbarton commented 10 years ago

$traceurRuntime could in principle be an optional dependency along the lines of any polyfill. Automatic construction of a custom runtime is feasible (automatically detecting polyfills could be hard).

Mixing different version of the runtime or different versions of the polyfills is not valid: these are global functions and the meaning of the program could depend upon identical values.

I guess we should not use the word 'modular' with runtime/polyfills. These are intrinsically not modules.

UltCombo commented 10 years ago

@johnjbarton leaving the polyfills alone -- the polyfilled global constructors (Promise, Map, Set, Symbol, etc.) and prototype methods --, wouldn't it be possible to turn $traceurRuntime into a module?

There's high coupling between the generated code and the $traceurRuntime, making it a module which can be depended upon by the compiled code would be a huge step forward imho.

UltCombo commented 10 years ago

As far as I can see, Traceur-generated code have 2 kinds of dependencies: global polyfills and the runtime library.

Global polyfills have low coupling with the generated code, as they may be:

Already present in the environment
Polyfilled by 3rd parties
Polyfilled by the Traceur runtime library

Now, the $traceurRuntime has very high coupling with the generated code, as the Traceur version used to generate the code must match the $traceurRuntime library version.

So, is it feasible for a transform step to add an import $traceurRuntime from '...'; to the generated code instead of putting the $traceurRuntime in the global scope? This way, running multiple versions of Traceur-generated code in the same environment seems feasible.

I believe polyfill conflicts are a rarer case, and can be dealt with by implementing them with proper feature tests.

Thoughts?

//cc @johnjbarton @arv

johnjbarton commented 10 years ago

I think the more interesting case would be more selective, eg import $traceurClasses from 'traceur@0.0.60/src/runtime/classes.js'; That way smaller builds would be possible, an important issue for browser use. The multi-version case works in the opposite direction, using more space, but I suppose it would also be solved by this direction.

UltCombo commented 10 years ago

@johnjbarton agreed, that seems to be an excellent direction for both the browser and Node.

arv commented 10 years ago

One issue is that import is not allowed in Script. I think it would only be safe to do this if we compile to CommonJS or AMD.

UltCombo commented 10 years ago

@arv the System API can be used for Script, right?

johnjbarton commented 10 years ago

No, because the System API is asynchronous.

UltCombo commented 10 years ago

@johnjbarton oh yeah, I was away for a bit and just noticed that adding a dependency through the System API when the script is not already using the System API would break the Script's synchronous nature.

It would be safe to add the dependency when the modules option is set to commonjs, amd or instantiate, I believe. This would also seem feasible for inline and register modules, though these cases would require packaging the traceur runtime in the compiled output -- not sure if this would bring any significant drawback.

arv commented 10 years ago

I'm reluctant to depend on the module runtime too. It is large and not needed in a lot of cases.

UltCombo commented 10 years ago

@arv not sure if I follow. Do you mean that the generated code shouldn't have a dependency on the whole traceur runtime library? That's how things are done currently AFAICS (except that currently the runtime is a global instead of an export), but I agree it would be better to have more selective imports as commented by @johnjbarton.

google / traceur-compiler

Traceur Node API modularity #1209