Unicode text is not correctly parsed

danipen / TextMateSharp

A port of tm4e to bring TextMate grammars to dotnet ecosystem

MIT License

87 stars 15 forks source link

Closed danipen closed 3 years ago

danipen commented 3 years ago

Wen trying to parse the following line using the csharp grammar:

string s = "chars: 安定させる";

The IToken::StartIndex and IToken::EndIndex for the string token are not correct.

danipen commented 3 years ago

I added a test in TokenizeUnicodeCommentsTest to check that case.

Remember to uncomment the Unicode string when there is a fix for it.

danipen commented 3 years ago