bzick / tokenizer

Tokenizer (lexer) for golang
MIT License
92 stars 6 forks source link

Parsing an empty string causes odd behavior #7

Closed dpedu closed 5 months ago

dpedu commented 1 year ago

This code panics on the marked line. It seems like IsValid() incorrectly returns true when it should return false since there are no more tokens available.

p := tokenizer.New()

stream := p.ParseString("")
defer stream.Close()

for stream.IsValid() {
    token := stream.CurrentToken()
    if token.Is(tokenizer.TokenInteger) { // <--- panic
        fmt.Println(token.ValueInt())
    }
    stream.GoNext()
}

I'm using 1.3.0 and the panic stack trace tells me the problem is here:

github.com/bzick/tokenizer.(*Token).Is(...)
    /xxx/go/pkg/mod/github.com/bzick/tokenizer@v1.3.0/token.go:220
bzick commented 1 year ago

I'll fix that

bzick commented 5 months ago

Fixed in 1.4.4