[AST] Unicode Support

[LuoZijun]

Hi, the parser is not ready for parse unicode, right ?

The master branch:

• https://github.com/ratel-rust/ratel-core/blob/master/core/src/tokenizer.rs#L39-L57
• https://github.com/ratel-rust/ratel-core/blob/master/core/src/tokenizer.rs#L978-L980

The rewrite branch:

• https://github.com/ratel-rust/ratel-core/blob/rewrite/ratel/src/lexer/util.rs#L12-L30
• https://github.com/ratel-rust/ratel-core/blob/rewrite/ratel/src/lexer/mod.rs#L60-L78
• https://github.com/ratel-rust/ratel-core/blob/rewrite/ratel/src/lexer/labels.rs#L4-L31

JavaScript Unicode Name Example:

这是一个名称 = "世界 ( World )！";

console.log(这是一个名称.length);
console.log(`Hello, ${这是一个名称}`);

[maciejhirsz]

Hey! You can try the above snippet in http://maciej.codes/ratel-wasm/ and see that it works.

It's not really correct, the lexer reads any sequence of non-ASCII bytes as identifiers, so technically any set of unicode characters not separated by permitted ASCII in identifiers, will produce an identifier token. To be really compliant with the specs we would have to make sure that the unicode we accept as identifiers are letter characters (the rewrite branch as it is checks it for the first character).

[LuoZijun]

I'm so sorry for that, i will close this.