search

rusty-ecma / RESS

Rusty EcmaScript Scanner
MIT License
93 stars 10 forks source link

Implement TC39 JSON-Superset proposal #51

Open FreeMasen opened 5 years ago

FreeMasen commented 5 years ago

Allow line separator and paragraph separator in string literals.

proposal

FreeMasen commented 5 years ago

The changes to allow for these new characters need to be made in the tokenizer's string method.

Currently there is a check for "is_new_line_not_cr" that will need to be updated to only check for '\n' as the other two characters in that method are now allowed with the JSON superset proposal.

A good test for this would include the following strings:

"\"line\u{2028}seperator\"" "'paragraph\u{2029}seperator'"

The test should correctly evaluate these to the appropriate RawToken and after parsing each the buffer should be empty. A good reference for this is the tokenizer_strings test in the same module.