Interpolated strings

[ItsDeltin]

Interpolated strings (aka template literals) allow you to insert values into strings with a format like $'Hello {x}!'

Possible ways to implement:

Implementation 1

Tokens have a new field (ex PreceedingCharacters) in the Token class which contains the preceding whitespace characters leading to the previous token. For example:

String myString = $"Hello {x}!";

This will generate the tokens:	Type	Text	PreceedingCharacters
Identifier	String	Empty
Identifier	myString	' '
Equals	=	' '
InterpStart	$	' '
DoubleQuote	"	Empty
Identifier	Hello	Empty
CurlyBracketOpen	{	' '
Identifier	x	Empty
CurlyBracketClose	}	Empty
Exclamation	!	Empty
DoubleQuote	"	Empty

When the InterpStart token is encountered, the string will be built using the proceeding tokens until the curly bracket is reached.

Pros:

• Easy to implement.

Cons:

• Each token requires more memory:
  • Every token needs to track the preceding whitespace.
  • More tokens need to be added. This can be egregious with lot of strings, since a token needs to be added for each symbol.

Implementation 2

Curly brackets are counted during the lexing stage.

For lexing the code below, when the first { is encountered in the string, value starts. Every { after that will add to a counter, } will subtract from it. When } is encountered when the counter is 0, the interpolation head is matched.

String myString = $"Hello {func(() => { })}!";

Pros:

• Easy to implement.
• Less memory usage compared to implementation 1, something like $'Hello {x}!' requires only 3 tokens rather than 8.

Cons:

• There can not be any valid syntax that only has either the opening or closing curly bracket (outside of strings).
  • As of writing this, every opening curly bracket in the syntax requires a closing curly bracket, so this is only a potential issue for future possible syntax.

Implementation 3

The lexer is changed to only get the tokens when needed, so lexing is done during parsing rather than before. This is how the Typescript compiler handles interpolated strings and parsing in general.

Pros:

• Has identical memory usage to implementation 2.

Cons:

• Harder to implement compared to the other versions.

[ItsDeltin]

Added with v2.0-beta.8

[ItsDeltin]

I went with 3 because 2 did not work well with the incremental lexer.