Throwing tokenizer errors when string doesn’t end

[xDGameStudios]

When I tokenize a QuotedString.CStyle and the file ends mid way without closing quotes how can I send an error message? (I’m using a custom tokenizer) it matched an empty string... should it?

[nblumhardt]

Hi! Checking out how TokenizerBuilder behaves might help, here. The example JSON parser uses it and has the following behavior:

PS ...\superpower\sample\JsonParser> dotnet run
json> {"x": "fdsjlkfdsa
            ^
Syntax error (line 1, column 7): incomplete string, unexpected end of input, expected `"`.

[xDGameStudios]

Hi! Checking out how TokenizerBuilder behaves might help, here. The example JSON parser uses it and has the following behavior:

PS ...\superpower\sample\JsonParser> dotnet run
json> {"x": "fdsjlkfdsa
            ^
Syntax error (line 1, column 7): incomplete string, unexpected end of input, expected `"`.

Hmmmm Okay! I now see what I was doing wrong.. my mistake. I'm having some problems editing the JsonStringToken to allow for matching VerbatimStyle strings.

@"this is a verbatim string where all the character are escaped \t and \n are not special chars"

I'm doing it like this (adding the @ check at the begining of the parse but it is not working) I must be doing something wrong.. can give me some help?

public static TextParser<Unit> JsonStringToken { get; } =
            from at in Character.EqualTo('@')
            from open in Character.EqualTo('"')
            from content in Span.EqualTo("\\\"").Value(Unit.Value).Try()
                .Or(Character.Except('"').Value(Unit.Value))
                .IgnoreMany()
            from close in Character.EqualTo('"')
            select Unit.Value;