Closed vk-coder closed 7 months ago
In the following code the parser incorrectly report "1." as a float.
parser := tokenizer.New() parser.AllowNumbersInKeyword() parser.DefineTokens(1, []string{".."}) stream := parser.ParseString("1..2")
output
0: { Id: 0 Key: -3 <<<<<< Value: 1. <<<<<< Position: 0 Indent: 0 bytes Line: 1 } 1: { Id: 1 Key: -6 Value: . Position: 2 Indent: 0 bytes Line: 1 } 2: { Id: 2 Key: -2 Value: 2 Position: 3 Indent: 0 bytes Line: 1 }
Expectation is as follows. Note: If ".." is not explicitly defined as a token then second token in the stream can be marked as TokenUnknown
0: { Id: 0 Key: -2 <<<<<< Value: 1 <<<<<< Position: 0 Indent: 0 bytes Line: 1 } 1: { Id: 1 Key: 1 <<<< Value: .. <<<< Position: 2 Indent: 0 bytes Line: 1 } 2: { Id: 2 Key: -2 Value: 2 Position: 3 Indent: 0 bytes Line: 1 }
Fixed in 1.4.2
In the following code the parser incorrectly report "1." as a float.
output
Expectation is as follows. Note: If ".." is not explicitly defined as a token then second token in the stream can be marked as TokenUnknown