ianlewis / lexparse

Experimental lexer/parser library written in Go
Apache License 2.0
0 stars 1 forks source link

Update Parser to not use Lexer instead of lexemes channel #12

Open superfrink opened 10 months ago

superfrink commented 10 months ago

Currently Parser reads lexemes from a channel and does not have a reference to the Lexer. This is non-optimal because Parser is unaware of errors that occur during lexing.

Lexer.Run() is already able to return an error. Lexer.Lex() will store the error in the Lexer and close the lexeme channel. Parser does not know why the channel was closed. The input might be done or if might be an error. Parser is unable to call Lexer.Err() to check for an error.

It would be better if Parser had a reference to Lexer. Then Parser could call Lexer.Err().

Parser could call something like Lexer.NextLexeme() and which returns both the lexeme and an error. This would return both values in one call instead of reading a lexeme from a channel and then calling Lexer.Err().

This would change creating a parser from:

    l := lexparse.NewLexer(runeio.NewReader(inReader), &lexState{})
    lexemes := l.Lex(ctx)
    p := lexparse.NewParser[calcToken](lexemes)

to:

    l := lexparse.NewLexer(runeio.NewReader(inReader), &lexState{})
    p := lexparse.NewParser[calcToken](l)

And change Parser.Peek() from:

    l, ok := <-p.lexemes
    if !ok {
        return nil
    }

to

    l, err := p.lex.NextLexeme()
    if err != nil {
        return fmt.Errorf("lex error: %w", err)
    }
ianlewis commented 3 months ago

I suppose I wanted for the caller to have control over the goroutines that get created (e.g. by Lex). The caller can pass a context to Lex to allow them to cancel the goroutine, impose timeouts etc.

I also thought the caller should have control over error handling for the lexer instead of having Parser handle it. In that case the lexer would return an error and the channel given to the parser would just be closed. I agree this is a bit harder to reason about so that's why the LexParse function exists.

I suppose your idea is to have the Parser pass through errors but then it's hard to know whether the errors came from the lexing or from parsing.