CS2 Discussion: Features: Tagged template literals

[greghuc]

Current Coffeescript is incompatible with template literals, specifically tagged template literals. As such, direct support for template literals should be considered for coffeescript nextgen.

Specifically, calling a tagged template literal looks this:

var a = 5;
var b = 10;

function tag(strings, ...values) {
  ...
  return "Bazinga!";
}

tag`Hello ${ a + b } world ${ a * b }`;

Coffeescript already uses backticks to embed javascript:

hi = `function() {
  return [document.title, "Hello JavaScript"].join(": ");
}`

Unless I've missed a trick, it's currently impossible to use a tagged template literal in Coffeescript by embedding the relevant ES6 javascript.

FYI, I ran into this problem when exploring the newish bel DOM templating library. Since it uses tagged template strings, I believe it's inoperable with current Coffeescript. This is the first time I've seen incompatibility between Coffeescript and Javascript..

[JimPanic]

Good catch! This is definitely something that needs work.

Would it make sense to replace the current literal-JS syntax? Or should we rather introduce a different syntax for template strings?

[lydell]

See also: https://github.com/jashkenas/coffeescript/issues/1504#issuecomment-187814583

[greghuc]

I also submitted a bug report for this on the Coffeescript repo: https://github.com/jashkenas/coffeescript/issues/4301

[greghuc]

FYI, I got tagged template literals working in my browser-side Coffeescript project by encapsulating use of this es6 functionality in its own javascript class. So - for the first time - my project contains both coffeescript and javascript files, which are all brought together in a browserify + broccoli build step.

# views/test-view.js:

module.exports = function (bel) {

    var TestView = function () {
        ...
    };

    TestView.prototype.render = function (name) {
        return bel`<div>Hello ${name}</div>`;
    };

    return TestView;

};

# And in Coffeescript-land:

bel = require 'bel'
TestView = require('views/test-view')(bel)
element = (new TestView()).render 'John'

[rattrayalex]

CoffeeScript interpolates with " instead of ```.

Currently, code such as someTag"hello #{world}" throws a syntax error in coffeescript, so presumably this syntax could be supported (how hard it would be, I'm not yet sure).

@greghuc is that the feature you're suggesting we build, or do I misunderstand?

[greghuc]

@rattrayalex yes, I'm imagining that someTag"hello #{world}" should eventually work Coffeescript. That, or there needs to be an alternative to escaping Javascript using backticks, so that this doesn't break:

`tag`Hello ${ a + b } world ${ a * b }`;`

You could argue that alternative escaping is the better solution, given escaping is the root cause of the break.

I haven't done a deep dive on this, but the issue is really with ES6 tagged template literals, not template literals. ES6 template literals are equivalent to CS string interpolation. But tagged template literals are a new kind of beast. They involve all 'arguments' to the interpolated string being passed to the supplied function, which then processes them. Example from MDN:

var a = 5;
var b = 10;

function tag(strings, ...values) {
  console.log(strings[0]); // "Hello "
  console.log(strings[1]); // " world "
  console.log(strings[2]); // ""
  console.log(values[0]);  // 15
  console.log(values[1]);  // 50

  return "Bazinga!";
}

tag`Hello ${ a + b } world ${ a * b }`;
// "Bazinga!"

Might be worth backing away from this, and treating this as an issue with Javascript escaping in Coffeescript.

[rattrayalex]

Sorry, what does this have to do with escaping? Is it just a question of the syntax? I don't see why use of the backtick is required for this feature.

[greghuc]

To restate, it is currently not possible to use ES6 template literals or tagged template literals inside Coffeescript (in an escaped javascript block). This is because ES6 template laterals use backticks in their syntax, and Coffeescript already uses backticks to define the start and end of the javascript block. As such, the Coffeescript compiler breaks.

So see this, copy and paste the following into "Coffee to JS" http://js2coffee.thomaskalka.de, and view the error:

`
tag`Hello ${ a + b } world ${ a * b }`;
`

So current Coffeescript is broken for use of this ES6 feature.

[rattrayalex]

Ah, gotcha. So there are two things we could do (non-exclusively):

1. Add the "tagged" feature to coffeescript string templates. eg, myTag"hello #{'wo'+'rld'}"
2. Allow backtick-escaped JavaScript to include template literals, perhaps by allowing triple-backtick blocks to contain single-backticks.

They both sound fairly straightforward to me, though I don't know enough about the relevant parts of the coffeescript compiler to say.

[GeoffreyBooth]

The escaping thing is just an outright bug in CoffeeScript, that’s been open since 2011: https://github.com/jashkenas/coffeescript/issues/1504 But there’s more interest in it lately since the advent of template literals. Basically it seems like there’s a consensus desire to be able to escape backticks, as well as add a triple-backtick delimiter for backticked blocks. That seems like a worthy improvement.

But fixing escaped backticks just enables a hacky workaround to using tagged template literals; it’s a far cry from supporting the feature itself in CoffeeScript syntax. Theoretically, the efforts could proceed in parallel; we could build support for myTag"hello #{'wo'+'rld'}" without necessarily fixing backticks.

@greghuc, would you mind posting a proposal for what the CoffeeScript syntax should be for tagged template literals? Including what the expected JavaScript output would be. Would we support tagged template literal blocks, e.g. myTag"""some multiline string...?

Also, would you like to take the lead in implementing this feature in the current compiler? It’s one of only three items in the Top Priority tier of our features list, as the only ES2015+ features we’ve found so far that imperil interoperability.

[greghuc]

@GeoffreyBooth I'll start off by posting a proposal for tagged template literals. I'll aim to have this done by next weekend (by Sunday Sept 18th). Initial thoughts:

• Syntax is myTag"hello #{'wo'+'rld'}" or myTag"""some multiline string"""
• Works for single-line and multi-line strings
• Is semantically equivalent to ES6 tagged template literals, enabling interoperability with newer templating libraries that depend on this feature.
• But compiles to "polyfill" Javascript, not a straight call to ES6 tagged template literals.

Regarding taking the lead on implementing this feature, I'm hypothetically up for it. I'll make a decision once it's clearer what the scope is (once we've agreed on the feature proposal), and how much spare time I have for open-source coding.

[GeoffreyBooth]

@greghuc I agree with everything except I’m neutral on the “compiles to ‘polyfill’ JavaScript” part. I think we need to make a broader philosophical decision on whether newly-built features should still be compiling down to ES5, or should just output ESNext and leave the shimming to a tool like Babel. For modules and classes it hasn’t been an issue, since shimming modules is way beyond our capabilities and the current class keyword is already essentially an ES5 shim; but for tagged template literals we’ll need to make a choice.

I think as a group in this repo we’ve already reached a consensus that a future version of the CoffeeScript compiler should output as much ESNext syntax as possible, leaving the shimming to other tools. I proposed a flag for enabling such output, so it can be built gradually over time and not break backward compatibility. Assuming @jashkenas signs off on such a plan, it would then be CoffeeScript’s stated goal to output modern ECMAScript whenever possible, which implies that new features we add now should just go ahead and output ESNext. Ideally they would output both, an ESNext version if the --ecmascript flag is set and a shimmed version otherwise, if we have the time and motivation to implement both versions. But I would think that the ESNext version would be the default, since we know we need that for future versions of CoffeeScript. Building the shimmed version merely saves people from needing to attach Babel to their build chains, and I’m not sure such a savings is worth the added burden on us to build and support an alternate (much more complicated) output for a new feature.

So I guess would you be okay will building tagged template literals that output at least as ESNext? And if you want to take on the added challenge of outputting ES5, that’s awesome.

[greghuc]

@GeoffreyBooth I think outputting tagged template literals in ESNext is actually the nicest option:

• Far less work
• Is semantically equivalent to ES6 tagged template literals, by definition

I considered the polyfill approach, as I didn't think outputting ESNext would be an option. So I'd be happy aiming for ESNext output for first implementation.

But I'll start with the proposal first.

[GeoffreyBooth]

@greghuc that sounds great. I don’t think this feature even needs a flag; like generators or modules, using the new syntax opts into getting ESNext output. We only need to worry about flags if you wanted to build both possible outputs.

[jashkenas]

Assuming @jashkenas signs off on such a plan, it would then be CoffeeScript’s stated goal to output modern ECMAScript whenever possible, which implies that new features we add now should just go ahead and output ESNext.

My preferred plan would be for a CoffeeScript 2.0 to implement and output modern ECMAScript — as soon as those features are shipping in real browsers and Node. I think that to output them earlier than that point (to rely on chains of transpilers, and promises to implement the spec as its written but before its implemented) would be foolish.

[nilskp]

Triple quotes is used in a lot of languages to allow not escaping the single quote. Couldn't something be used here? I.e. triple backticks?

[rattrayalex]

@nilskp yes, see above.

[nilskp]

@rattrayalex Ah, yes. I didn't read the thread carefully enough.

[DomVinyard]

@jashkenas

to rely on chains of transpilers, and promises to implement the spec as its written but before its implemented would be foolish

Why foolish? That seems like an odd indictment of this entire effort.

[jcollum]

I think what he's saying is that CS2 shouldn't try to confirm to the spec for ES2016 before ES2016 is actually implemented in Node 7 (which would give time for the kinks to be worked out? make sure the feature will actually make it to the language?).

[mrmowgli]

You can also put such things behind a different branch, ie esnext. I get it, the idea is really about only putting things into the language that will actually be adopted. However we can "Anticipate" future features by placing it in an unstable or experimental branch.

Perhaps we should do a formal proposal for the release process.

CS2(CS3, CS4 etc.) branches would include breaking changes (for instance classes) and any currently existing (Implemented in Babel?) high priority items that are already shipping in browsers. The ESNext branch would implement soft targets or features that are nearly complete in browsers. The experimental branch would be implementations of low priority ES6/7 targets that aren't fully defined.

I also wouldn't mind having tagged "Stable" versions.

[GeoffreyBooth]

Based on some of @jashkenas’ other comments, I think that what he means by waiting for browser implementation is not that he feels that people should rely on the coffee command-line tool as their one and only transpiler, but rather that a spec is just words on paper until browsers have implemented it. So if we support a spec before it’s widely supported, we risk supporting something that’s not really final.

Getting back to template literals, support is widespread in evergreen browsers:

So I think this feature is safe to implement. Per this comment, unless anyone objects to @jashkenas, we’re going to skip the ES3 version of template literals and go straight to ESNext.

@greghuc are you going to implement both tagged template literals and regular template literals? You’re welcome to do both if you’re up to the challenge, though if we compile "foo#{bar}" to foo${bar} we would need to release this update as part of a proposed 2.0.0-beta release.

[greghuc]

As discussed, I've written up a proposal for adding ES6 'tagged template literal' support to Coffeescript. Executive summary follows, followed by more detail (code examples, etc). Opinions welcome.

Executive summary

Template literals and tagged template literals are a new feature in ES6:

• Template literals enable Coffeescript-like string interpolation. Input strings can be multi-line, but the output string is slightly different to a Coffeescript block-string (in terms of spacing).
• Tagged template literals enable custom string interpolation, with the input text and expression parts being passed to a given function for assembly into an output string. The function output can actually be anything, not just a String. The yo-yo library outputs DOM elements.

The current situation with Coffeescript:

1. ES6 template literals (in an embedded Javascript block) break the Coffeescript compiler, so cannot be used in a Coffeescript file. This is because the template literal syntax uses backticks, which Coffeescript already uses to define the start and end of the Javascript block.
2. Coffeescript already offers string-interpolation capabilities matching ES6 template literals (barring spacing differences between a CS block-string and a ES6 multi-line template literal).
3. Coffeescript has nothing equivalent to tagged template literals.

Proposals for Coffeescript to interoperate/adopt ES6 template literals:

1. Coffeescript should allow embedded Javascript blocks to be delineated with 3-backticks markdown-style, and not escape backticks in the Javascript block. This is the quickest/easiest way for ES6 template literals to be used in a Coffeescript file. Full proposal below.
2. Coffeescript should not 'natively' adopt tagged literals at present, since CS already offers equivalent string interpolation. In future, the CS compiler could directly output interpolated-strings as ES6 tagged literals where appropriate.
3. Coffeescript should 'natively' adopt tagged template literals, as it has no equivalent functionality. The CS compiler will output ES6 code. This will enable ES6 functions expecting tagged template literals as input to be called directly from Coffeescript. The implementation will augment existing Coffeescript behaviour for single-line, multi-line and block strings. As such, the implementation will slightly differ from ES6 for multi-line block-strings, due to the spacing differences. Full proposal below.

Three final observations:

1. To completely align Coffeescript string behaviour with ES6 template literals, the spacing semantics of a Coffeescript block-string would need changing to match a multi-line ES6 template literal (example below). This change would be backwards incompatible, so has not been proposed.
2. I found an example where ES6 tagged template literals are nested inside one another. To support this, Coffeescript must also be able to nested interpolated strings. I did some testing, and discovered that CS cannot nest block strings (example below). Bug?
3. There needs to be clarification where tagged template literal support should be added. Should support be added to the current Coffeescript compiler? Or should support only exist for a new 2.0 compiler? This is a purely ES6 feature after all.

Executive summary ends.

Bug-fix: ensure ES6 template literals can be used in CS embedded Javascript blocks

ES6 template literals cannot be used in an embedded Javascript block, since the new syntax uses backticks. Backticks are already used by Coffeescript to define the start and end of the JS block. This example blows up:

`
`3 + 2 = ${3 + 2}`
`

The proposed fix is for Coffeescript to allow embedded Javascript blocks to be delineated with 3-backticks markdown-style, and not escape backticks in the Javascript block. E.g.:

`` (should be 3 backticks, but github can't handle)
`3 + 2 = ${3 + 2}`
`` (should be 3 backticks, but github can't handle)

Reasoning:

• Was the solution commonly proposed in the Coffeescript issue first raised about backticks in 2011.
• Matches the Markdown code syntax.
• Echos the Coffeescript block-string syntax, which uses three double-quotes

New feature: adopt ES6 tagged template literals in CS

Summary of ES6 template literals

An ES6 'template literal' is a string literal that can include interpolated expressions, and can be multi-line. Example:

`3 + 2
 = ${3 + 2}`

//Output string
"3 + 2
 = 5"

An ES6 'tagged template literal' is a template literal, but prefixed with a function reference. The function is called to generate the output string, being supplied with the template literal in the form of expressions and the text between them. Example:

//Uppercase the given expressions
function upperExpr(text, ...expressions) {
  return text.reduce((accumulator, textPart, i) => {
    return accumulator + expressions[i - 1].toUpperCase() + textPart
  })
}

var name = 'Greg'
var food = 'sushi'
upperExpr`Hi ${name}
     Do you like ${food}?`

//Output string
"Hi GREG
     Do you like SUSHI?"

So the simpler 'template literal' can be seen as a tagged template literal with an invisible 'normal' function that just concatenates the given text-parts and expressions together.

Coffeescript proposal

The proposal is for Coffeescript to 'natively' adopt tagged template literals. The implementation will augment existing Coffeescript behaviour for single-line, multi-line and block strings. As such, the implementation will slightly differ from ES6 for multi-line block-strings, due to the spacing differences.

Single-line string:

upperExpr = (text, expressions...) ->
  text.reduce (accumulator, textPart, i) ->
    accumulator + expressions[i - 1].toUpperCase() + textPart

name = 'Greg'
food = 'sushi'

//CS input
upperExpr"Hi #{name} Do you like #{food}?"

//ES6 output
upperExpr`Hi ${name} Do you like ${food}?`;

Multiline string:

upperExpr = (text, expressions...) ->
  text.reduce (accumulator, textPart, i) ->
    accumulator + expressions[i - 1].toUpperCase() + textPart

name = 'Ismael'
food = 'whales'

//CS input
upperExpr"Hi #{name}. Do
 you like #{food}?"

//ES6 output
upperExpr`Hi ${name}. Do you like ${food}?`;

Block string:

upperExpr = (text, expressions...) ->
  text.reduce (accumulator, textPart, i) ->
    accumulator + expressions[i - 1].toUpperCase() + textPart

language = 'coffeescript'

//CS input
upperExpr"""
         <strong>
           cup of #{language}
         </strong>
         """

//ES6 output
upperExpr`<strong>\n  cup of ${language}\n</strong>`;

Differences between Coffeescript strings and ES6 string literals

Coffeescript already supported multi-line strings with 'block strings'. In ES6, multi-line strings are now supported with string literals. But the two differ in terms of spacing. Example:

CS block string:

"""
  <strong>
         cup of coffeescript
       </strong>
       """

outputs JS string of

"<strong>
       cup of coffeescript
     </strong>"

ES6 multi-line string literal:

`
  <strong>
         cup of coffeescript
       </strong>
       `
outputs JS string of

"
  <strong>
         cup of coffeescript
       </strong>
       "

To completely align Coffeescript string behaviour with ES6 template literals, the spacing semantics of a Coffeescript block-string would need changing to match a multi-line ES6 template literal. This change would be backwards incompatible, so has not been proposed.

Coffeescript bug? Can't nest block-strings inside block strings

I found an example where ES6 tagged template literals are nested inside one another. To support this, Coffeescript must also be able to nested interpolated strings. I did some testing, and discovered that CS cannot nest block strings. Bug?

Busted:

# Nested block strings
""" A test of #{"nesting" + func("""
quotes
""")} 
inside each other
"""

Works:

"A test of #{"nesting" + func("quotes")} inside each other"

"A test of #{"nesting" + func("""
quotes
quotes
""")} inside each other"

""" A test of #{"nesting" + func("2")} 
inside each other
"""

[GeoffreyBooth]

@greghuc this is heroic. In the interest of organizing our discussion, would you mind splitting some of these out into separate issues? I’m not sure template literals and tagged template literals should really be discussed separately, so we could keep those here; but the block-backticks operator could get its own issue, as could the bug about nested block strings (that should perhaps be a bug issue on the main coffeescript repo).

BTW if you want to type three backticks in a code block in GitHub, you can use its other code block syntax, which is indenting with four spaces:

console.log(I’m a JS code block embedded in CoffeeScript! The time is ${Date.now()});

[lydell]

A few little things here:

1. I see no value in aligning the spacing semantics with ES2015+. In my opinion, CoffeeScript’s spacing semantics are far superior over ES2015+.
2. ${} is invalid JavaScript, while "#{}" is valid CoffeeScript. (CoffeeScript allows “empty” interpolations.)
3. I can’t reproduce the nesting-of-block-string bug:

$ coffee --version
CoffeeScript version 1.10.0
$ coffee -bpe '""" A test of #{"nesting" + func("""
  quotes
  """)} 
  inside each other
  """'
" A test of " + ("nesting" + func("quotes")) + " \ninside each other";

[GeoffreyBooth]

@greghuc To try to answer your questions:

• If it doesn’t make sense to implement template literals and tagged template literals separately, and I can imagine that it doesn’t, then both features would have to ship together with the breaking change CoffeeScript 2.
• If tagged template literals can be implemented on their own, then we can squeeze that feature into 1.x; but I wouldn’t expend the effort if I were you if this is any extra work. In my opinion we could release 2.0.0-beta1 as soon as the first meaningful feature is ready, whether that’s classes or template literals/tagged template literals.
• Adding the block-backtick operator could happen anytime, in either 1.x or 2. That effort should get its own issue, branch and pull request.
• Ditto with any bugs you find.
• Aside from CoffeeScript’s spacing semantics being better than ECMAScript’s in @lydell’s and my opinion, aligning with ES here would probably make template literals incompatible with Literate CoffeeScript. I definitely see no benefit in that. CoffeeScript has significant whitespace and ES doesn’t, so differences in spacing/indentation semantics are to be expected.

[greghuc]

@lydell thanks for checking the bug - I can't reproduce it either on 1.10.0. I wasn't near a terminal, so was running Coffeescript with: http://js2coffee.thomaskalka.de/

[lydell]

@greghuc Use http://coffeescript.org/#try: next time ;)

[GeoffreyBooth]

@greghuc I invited you to become a collaborator on https://github.com/GeoffreyBooth/coffeescript. You’re welcome to use a new branch on that repo to implement this, if you want me or @lydell or @JimPanic to have write access to your branch. Or you’re more than welcome to work in your own fork.

When the branch is ready for a PR, please submit it against the soon-to-be-created 2 branch on https://github.com/jashkenas/coffeescript, unless you implement tagged template literals separately from template literals. If you submit the two separately, tagged template literals could go into master (which is the 1.x, non-breaking-change branch) and template literals would go into 2.

[greghuc]

@GeoffreyBooth I've spun out 2 new issues from the tagged template literals work:

• Output ES6 template literals for Coffeescript interpolated strings: https://github.com/coffeescript6/discuss/issues/41
• Allow backticks to be used inside embedded Javascript blocks: https://github.com/coffeescript6/discuss/issues/42

[greghuc]

So I'll make a start on adding tagged template literals to Coffeescript next week (Sept 26 onwards), as per the proposal in this thread. My time for open-source commits is constrained, so I'm not providing an ETA for completion right now.

This will also be my first time changing Coffeescript, so there'll be a ramp-up period whilst I understand how it works.

Are there any pointers for getting an overview of the Coffeescript compiler?

[GeoffreyBooth]

Hi @greghuc, I put up https://github.com/GeoffreyBooth/coffeescript/wiki/How-parsing-works which was consolidated from a few helpful comments left for me while I was working on the modules PR. I plan to remove the specific references to modules and move it to the regular coffeescript repo wiki soon. There are other good pages there too: https://github.com/jashkenas/coffeescript/wiki

I also created https://github.com/GeoffreyBooth/coffeescript-gulp to automate compiling and testing.

[jcollum]

Hey I don't know where else to put this so:

I'm working on a new Hapi.js project the last few days. Decided to do it in ES2015 since a couple of junior devs are helping me. HOLY CRAP I MISS COFFEESCRIPT. All these damned extra braces and semicolons and such, clutterin' up my code. And the lack of easy key/value list comprehension. Etc.

So, thanks so much for participating here. Know that your work is appreciated.

[GeoffreyBooth]

Thanks @jcollum 😄 I think #32 is the issue you want 😉

[GeoffreyBooth]

Released in CoffeeScript 1.12.0.

[coffeescriptbot]

Migrated to https://github.com/jashkenas/coffeescript/issues/4925