bzick / tokenizer

Tokenizer (lexer) for golang
MIT License
98 stars 6 forks source link

Parsing closing parenthesis/brackets/curly without a token for them causes panic after parsing TokenString #9

Closed DSVDiniz closed 7 months ago

DSVDiniz commented 1 year ago

How to reproduce:

main.go

package main

import (
    "os"
    "github.com/bzick/tokenizer"
)

func main() {
    var parser = tokenizer.New()

    fp, err := os.Open("test.txt")
    if err != nil {
        panic(err)
    }
    defer fp.Close()

    stream := parser.ParseStream(fp, 4096).SetHistorySize(10)
    defer stream.Close()
    for stream.IsValid() {
        switch stream.CurrentToken().Key() {
        default:
            println(stream.CurrentToken().ValueString())
            stream.GoNext()
        }
    }
}

test.txt

a)

This also happens with "a]" and "a}", but does not occur when using 'ParseString()', only when using "ParseStream()" Can be mitigated by simply creating a token for ")" or other closing brackets. Edit: actually, any symbol after a TokenString causes a panic.

bzick commented 7 months ago

fixed in 1.4.3