bzick / tokenizer

Tokenizer (lexer) for golang
MIT License
98 stars 6 forks source link

parser incorrectly report float token for multiple "." #11

Closed vk-coder closed 7 months ago

vk-coder commented 1 year ago

In the following code the parser incorrectly report "1." as a float.

    parser := tokenizer.New()
    parser.AllowNumbersInKeyword()
    parser.DefineTokens(1, []string{".."})

    stream := parser.ParseString("1..2")

output

0: {
        Id: 0
        Key: -3                            <<<<<<
        Value: 1.                          <<<<<<
        Position: 0
        Indent: 0 bytes
        Line: 1
}
1: {
        Id: 1
        Key: -6
        Value: .
        Position: 2
        Indent: 0 bytes
        Line: 1
}
2: {
        Id: 2
        Key: -2
        Value: 2
        Position: 3
        Indent: 0 bytes
        Line: 1
}

Expectation is as follows. Note: If ".." is not explicitly defined as a token then second token in the stream can be marked as TokenUnknown

0: {
        Id: 0
        Key: -2                           <<<<<<
        Value: 1                          <<<<<<
        Position: 0
        Indent: 0 bytes
        Line: 1
}
1: {
        Id: 1
        Key: 1                             <<<<
        Value: ..                          <<<<
        Position: 2
        Indent: 0 bytes
        Line: 1
}
2: {
        Id: 2
        Key: -2
        Value: 2
        Position: 3
        Indent: 0 bytes
        Line: 1
}
bzick commented 7 months ago

Fixed in 1.4.2