CS2 Discussion: Features: Flag for ESNext output from current compiler

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-11 17:22

This is a proposal for generating ES2015+ syntax from the current compiler. It’s rather simple. We add a flag:

--ecmascript    Output the latest ECMAScript syntax whenever possible.

To implement this, in the compiler itself we can either have if blocks within node classes that generate different output based on whether this flag is set; or we can have a second set of node classes that override the first set if this flag is set.

This flag would only apply to features like await, where we have a PR that implements an ES5 polyfill but others might prefer the await keyword output as is; and features like => that are supported in ECMAScript now and people might prefer to output them as is rather than polyfilling them. Eventually this flag could allow us to automatically output let or const as appropriate, without worrying about breaking backward compatibility. Modules support would be unaffected by this flag, as using the import or export keywords would be opting in to ECMAScript for those lines; there’s no point in making such lines throw an error if --ecmascript is unset. Likewise for generators. With regard to classes, I’m assuming based on coffeescript6/discuss#22 that we will likely implement a new keyword that produces ECMAScript classes, leaving the current class keyword alone and allowing projects to contain both types; so those keywords would also be unaffected by this flag.

This flag isn’t meant to foreclose the effort at creating a new compiler; on the contrary, it might pave the way for it. We can implement this flag now, with maybe just one or two features compiled differently, with the list growing over time; but when the new compiler is ready and can handle all the same features that the legacy compiler plus --ecmascript can handle, the --ecmascript flag could trigger opting into the new compiler.

[coffeescriptbot]

From @rattrayalex on 2016-09-11 17:31

In terms of implementation, couldn't it simply wrap decaffeinate?

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-11 17:47

In terms of implementation, couldn't it simply wrap decaffeinate?

No. decaffeinate doesn’t convert all code correctly. It’s meant to preserve code including comments and so on. There are things it can’t parse, that it leaves for the developer to convert by hand. See https://github.com/decaffeinate/decaffeinate/blob/master/docs/conversion-guide.md

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-12 04:56

Update: I just found this comment where @jashkenas proposes a flag for ES6 output. So I guess a flag for this purpose would actually be blessed after all.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-12 05:29

There are some questions we would need to hammer out (@lydell?):

• How do we divide the code? The simplest approach would be an if block within the relevant classes in nodes.coffee, but a better one might be to create a separate nodes_ecmascript.coffee that overrides those classes from the first one. And over time as we implement more ECMAScript-output versions of features, nodes_ecmascript.coffee would grow.
• We would also need tests that only run in --ecmascript mode, and vice versa. How would we enable that?

This also ties into the discussion of a new compiler. Basically, with this flag I’m assuming a new compiler is a ways off. Both await and tagged template literals would have a use/need for this flag now, literally this week or next, and I don’t want to hold up those features or build them only as ES5 shimmed versions because we lack this flag or a working new compiler. If and when the new compiler arrives, --ecmascript could be redirected to triggering the new compiler, or it could get its own opt-in flag or new repo/NPM module.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-12 05:41

@rattrayalex Decaffeinate might not be an option, but Decaf might be . . .

[coffeescriptbot]

From @greghuc on 2016-09-12 07:12

If there's an --ecmascript flag, the compiler could be more blunt about erroring. So if someone uses an ESnext-backed feature (like await or tagged template literals) without the flag, the compiler errors and says "you need the --ecmascript flag to use this feature". And there's no polyfill for these features.

From a documentation perspective, the flag is also useful. We put a section at the end of the Readme listing the ESnext-only features which need the flag. And this sets the narrative for what's happening: if/when the compiler switches to just ESnext output, it can be explained in terms of the flag ("flag is now on by default").

[coffeescriptbot]

From @jashkenas on 2016-09-12 20:31

I think that a more appealing path than a --modern-style flag is to simply say that CoffeeScript 1.x is for ES3-era browsers, and CoffeeScript 2.x+ is for the evergreen browsers of today. Then you can make your breaking changes, release the new compiler, and folks that want to use it, can.

[coffeescriptbot]

From @mrmowgli on 2016-09-13 01:06

Coffeescript 2 should still end up being coffeescript. I think the problem a lot of these projects run into is that they don't become the defacto compiler. At some point the base coffeescript branch should merge the new changes from ES6, and you just use coffeescript. You've already integrated yield, which is an ES5+ feature. The real question is the best way to ease that transition, or have some procedure for transitioning, much like io.js -> node.js development. io.js is the unstable new feature set, node.js is the production ready version. @jashkenas, is there a reason you have been avoiding the use of flags?

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-13 02:04

I think CoffeeScript 2.0 is what would be produced by the --modern or --ecmascript flag. The only question is whether developing the ESNext-outputting compiler should happen in a 2.0 branch or in the current branch behind a flag.

The branch approach has been taken by Gulp in their long march toward version 4.0; for the last two years I’ve been putting "gulp": "gulpjs/gulp#4.0", in my package.json. It has its merits while developing in that you’re not looking at two versions of how features should be implemented; either you’re on the 2.0 branch or you’re not. The codebase is simpler.

Where it gets tricky, though, is if we want to add a new feature like await that could be added as both legacy and ESNext versions. The only way to do such a thing is to open two pull requests, one to merge into 2.0 and the other to merge into master. As the PRs would get reviewed, there would be duplication of effort for revisions that need to be applied to both PRs.

We could certainly call the --modern-flagged output “CoffeeScript 2.0,” and eventually release a 2.0.0 module to NPM that has --modern enabled by default. @jashkenas do you really care if this work happens behind a flag versus on a branch? I think this decision might best be made by whoever takes on this work (not that that will be me).

[coffeescriptbot]

From @jashkenas on 2016-09-13 14:37

@jashkenas do you really care if this work happens behind a flag versus on a branch?

I don't care terribly much per se, but my strong advice would be that having two code generation paths in an already complex codebase will make things that much more difficult to work with and continue to develop.

For development, maintainability, for marketing, and for user friendliness — I think it would be smart to work and release from a 2.0 branch. And to tell people that CoffeeScript 2+ targets evergreen JavaScript.

[coffeescriptbot]

From @JimPanic on 2016-09-13 15:00

I like that idea. Although as long as we don't introduce anything breaking backwards compatibility, semver kind of dictates to follow along the path of 1.x. (Has CS even been using semver so far?)

But given classes are a top priority, together with getters and setters, this is not too far away either.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-13 17:10

@jashkenas I’m not opposed to a new branch, but I think any “new” compiler is likely to include falling back to the “old” one, so we’re going to have a complicated codebase regardless. Like for example maybe class gets handled by the new compiler, and all other nodes get handled by the old. Until the new compiler is feature-complete with the old, we’ll have both littering up the codebase. But I doubt the new compiler would ever be feature-complete; more likely, it would only process features that we want to output differently than the old compiler (like =>) and leave everything else to the current compiler.

This assumes that the new compiler takes a different approach than the old one, for example what we’ve been discussing about ASTs in coffeescript6/discuss#25. If CoffeeScript 2.0 outputs ESNext syntax using same string-generation compiler that we have now, then sure, it should be in a new branch and there aren’t two code generation paths.

[coffeescriptbot]

From @jashkenas on 2016-09-13 17:25

Maybe I'm not being clear — there isn't any "new" compiler yet in existence. I'm talking about the ESNext-targeting features y'all are adding to the current compiler happening in a "2.x" branch, and being released as 2.0.0 soon.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-14 03:56

The thing about 2.0.0 is that we’re not likely to know what all of our breaking changes will be. We know that transpiling CS class to ES class will be a major breaking change, so certainly when (if) that happens we should bump to 2.0.0 unless this change is hidden behind a flag. But what if after that we conform our splat operator with ES’ splat operator? That’s another breaking change; do we go to 3.0.0?

We also can’t just hold off on releasing 2.0.0 until “all” of the ES-related breaking changes are done. Not only is that not fair to the people eager for class support, but by the time we implement all of the breaking changes we can think of now, ES2016 or ES2017 will be finalized and introduce more potential breaking changes if we want to target new features introduced by those standards. I also don’t know how much motivation we have as a community; we might not do much more than classes.

I think we need to release support for classes ASAP, regardless if some or all of the rest of these features are ready. So that leaves us with a few options:

1. When classes land, assuming it’s a breaking change and not esclass, we bump to 2.0.0. We then just keep bumping major versions for each new ESNext feature that’s a breaking change, regardless of how “big” the feature is.
2. Same as 1, but we state in the documentation that we’re not strictly following semver: major breaking changes will bump the major number, minor breaking changes (splat operator) the minor number.
3. We introduce a --modern or --ecmascript flag and keep all ESNext output behind there, aside from features like generators or modules that are opt-in by using the feature. We stay in 1.x, and the docs would state that the flag output could have breaking changes at any time. We eventually release 2.0.0 as the modern output and retire the flag.

I honestly don’t have a preference; my only strong opinion is that we don’t commit to a plan that causes some features to get delayed releases because we’re waiting for a bigger breaking-change release. Modules should be released as soon as they’re ready, as should classes and tagged template literals and await.

The other decision that should be made in concert with this is whether we release ES5-shim versions of any new features we build. This might only apply to await; we already have a PR with shimmed await. Do we release that in the 1.x branch, and then update it to ESNext in the 2.x branch/flag? Or just release only the ESNext version, and it’s opt-in like generators?

@jashkenas, @lydell, other maintainers?

[coffeescriptbot]

From @JimPanic on 2016-09-14 04:56

But what if after that we conform our splat operator with ES’ splat operator? That’s another breaking change; do we go to 3.0.0?

That's the basic idea, yes. But other feature releases (like async/await, let/const, etc) can happen in between. Any bug fixes or even additional features that are easily back portable can still also be released as new versions of previous major versions - so long as it's applicable because the code in question didn't change with the major versions.

In any case: we will require some form of release management process and proper regression coverage for this to work. (But we'll need that anyway :) )

[coffeescriptbot]

From @JimPanic on 2016-09-14 06:13

Actually most of the features are just additional syntax afaict. Classes are the only thing breaking source compatibility completely. import/export, async/await, let/const, template literals, fat arrows - they all do not break compatibility as they are planned right now. We might discover something during implementation, of course. But then again this might be a bigger change than originally anticipated thus we might also want to bump the major version.

I see this as a good path forward that also communicates progress to the community. (And I don't mean the major bump, just that there are actual bumps in any kind of version publicly released.)

[coffeescriptbot]

From @jashkenas on 2016-09-14 14:29

The thing about 2.0.0 is that we’re not likely to know what all of our breaking changes will be.

We need to figure that out. I'd put classes, arrow functions, spread operator, default parameters, destructuring assignment, and for..of loops on the breaking changes list. Perhaps there are more. We can ship a 2.0-pre as soon as you like with one or more of them. As soon as more those things are compiling into ES, they can be added to 2.0-pre. As soon as all of the breaking changes are done, it can be released as 2.0.0.

What else?

• Modules can be released immediately under 1.x and also under 2.0-pre.
• Let's not ES5-shim things. Let's generally have CS 1.x target ES3, and CS 2.x target evergreen JS.
• Any fixes that are backwards compatible can of course still be merged and released on the 1.x line.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-14 15:18

I like the idea of 2.0.0-pre. Would we release to NPM as that?

Where does this leave await? Should we accept the current ES5-generating PR as part of 1.x, and redo it as part of 2? (Leave aside the debate of whether it's worth incorporating at all. For the purposes of this question, let's just assume we want to support it.)

[coffeescriptbot]

From @jashkenas on 2016-09-14 15:26

I like the idea of 2.0.0-pre. Would we release to NPM as that?

Yes. And anyone who wants to try it out, can.

Where does this leave await? Should we accept the current ES5-generating PR as part of 1.x?

No. See: http://kangax.github.io/compat-table/es2016plus/#test-async_functions. We should wait until runtimes begin to implement it, and then compile it straight through.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-14 16:11

My point with asking about await was to try to get a general rule for how to handle new features we add between now and 2, that could be output as either shimmed ES3 or as straight ESNext. Perhaps tagged template literals are a better example; they’re supported in all evergreen browsers: http://kangax.github.io/compat-table/es6/#test-template_literals_tagged_template_literals. It would also be very straightforward how to shim them for ES3, if we wanted to make the effort to implement two outputs. (Which is a big “if”.)

So for tagged template literals or a feature like them, are we skipping the ES3 version and going straight to outputting ESNext only? Or do we want to output ES3 for the 1.x branch?

[coffeescriptbot]

From @rattrayalex on 2016-09-14 16:30

@jashkenas it sounds like you'd be opposed to CS2.0 including features that require further transpilation (eg; await).

Do you predict the population of folks who are both willing to upgrade to possibly-backwards-incompatible 2.0, but unwilling to use a second transpiler (or simply avoid using features like await) is likely to be significant?

await et al should be shipping in Node/Chrome quite soon – it might be very nice to have some forward momentum (eg; behind 2.0-pre) before the announcement hits.

[coffeescriptbot]

From @rattrayalex on 2016-09-14 16:31

Separately, I personally quite like the plan @jashkenas outlined above – @GeoffreyBooth , if you agree, would you like to incorporate it into the README (presumably paraphrased)?

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-14 19:10

@rattrayalex Yes, once we settle the ES3/ESNext question I’ll happily write this up.

Another option is that 2.0.0 includes Babel, perhaps behind a flag like --compatibility. So the CS2 compiler produces ESNext, and if --compatibility is enabled then CS2 compiler output is piped through Babel as part of the coffee CLI build tool. The people that want one-stop shopping with one build tool to ES3 still get it, but we get to strip out all shims and polyfills from our compiler and leave that work to Babel.

[coffeescriptbot]

From @jashkenas on 2016-09-14 19:47

So for tagged template literals or a feature like them, are we skipping the ES3 version and going straight to outputting ESNext only? Or do we want to output ES3 for the 1.x branch?

Let's skip the ES3 version, and ship them outputting to ESNext on the 2.0 lineage ASAP.

Do you predict the population of folks who are both willing to upgrade to possibly-backwards-incompatible 2.0, but unwilling to use a second transpiler (or simply avoid using features like await) is likely to be significant?

Not at all. I just think that it's generally foolish to begin chasing JS features that haven't started shipping yet — as soon as they start shipping, great. But before, you have things like this: https://gist.github.com/Rich-Harris/0b6f317657f5167663b493c722647221

await et al should be shipping in Node/Chrome quite soon – it might be very nice to have some forward momentum (eg; behind 2.0-pre) before the announcement hits.

Totally.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-14 20:42

Let's skip the ES3 version, and ship them outputting to ESNext on the 2.0 lineage ASAP.

Sorry to keep beating a dead horse, but is there a reason not to output ESNext in the 1.x branch if the feature is opt-in by usage? Like what we’re doing already for modules and generators. When tagged template literals are implemented, it will use a syntax that throws an error today; so it’s not a breaking change. Anyone who uses the new syntax would be opting in to the new feature, which would output ESNext.

[coffeescriptbot]

From @jashkenas on 2016-09-14 21:15

but is there a reason not to output ESNext in the 1.x branch if the feature is opt-in by usage?

We caaaaaan ... but it muddles the message. Honestly, modules and generators would be better shipped on the 2.x line than the 1.x line to keep things nice and simple. But that ship has already sailed (generators wise).

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-14 21:43

I think the message is pretty simple: ESNext output that is a breaking change comes in 2. ESNext that’s opt-in and therefore doesn’t break anything comes in 1.

[coffeescriptbot]

From @GeoffreyBooth on 2016-09-16 04:55

It seems like the desire for a new flag is low. Closing this issue unless someone wants to argue why a flag is better than a new branch/version.