Tokenize Template Literals

[chharvey]

Tokenize template literals, which are immutable but dynamic.

TemplateFull   ::= "`"  TemplateCharsEndDelim ? "`"
TemplateHead   ::= "`"  TemplateCharsEndInterp? "{{"
TempalteMiddle ::= "}}" TemplateCharsEndInterp? "{{"
TempalteTail   ::= "}}" TemplateCharsEndDelim ? "`"

TemplateCharsEndDelim  ::=
          [^`{\#x03]                          TemplateCharsEndDelim ?   |
    "{" (([^`{\#x03] | "\"  ([^`#x03] | "`")) TemplateCharsEndDelim ?)? |
                       "\"  ([^`#x03] | "`")  TemplateCharsEndDelim ?
TemplateCharsEndInterp ::=
          [^`{\#x03]                          TemplateCharsEndInterp?   |
    "{"  ([^`{\#x03] | "\"  ([^`#x03] | "`")) TemplateCharsEndInterp?   |
                       "\" (([^`#x03] | "`")  TemplateCharsEndInterp?)?

Template literals are sequences of characters delimited with back-ticks ` `. They may contain line breaks, but do not have escapable characters (except the back-tick). Most importantly, template literals may contain interpolation, which breaks up the template into several tokens.

`a template
may contain line breaks
and {{ 'most importantly' }},
interpolated expressions`

There are 4 types of template literal tokens: full, head, middle, and tail.

`3 times 4 is {{ 3 * 4 }}, that is,{{ ' ' }}twelve.`

The tokenizer should split the code above into seven tokens:

1. a template head: `3 times 4 is {{
2. an integer: 3
3. an operator: *
4. an integer: 4
5. a template middle: }}, that is,{{
6. a string literal: ' '
7. a template tail: }}twelve.`

As with string literals, the tokenizer does not “cook” the string or compute the values of the interpolated expressions. But at runtime, the expressions’ values will be computed and injected into the template at their respective locations.

Expressions are optional in the interpolation syntax.

`The quick brown fox {{}} jumps over {{  }} the lazy dog.`

Only the back-tick is escapable in template literals.

• \` will produce U+0060, a back-tick, the string template delimiter (technically, “grave accent”)

Otherwise, backslashes do not escape anything; they are literal characters.

[chharvey]

commits 4860a41 , 8d0a33a close this