Check next token without consuming it

[alastairzotos]

I want to make a recursive descent parser, which requires the ability to check the next token and act accordingly. Ideally, there would also be a function to consume a token of a certain type, and to throw an error if the type of the token doesn't match.

Example:

const left = parseExpression();

if (lexer.check('operator')) {
    const operator = lexer.next(); // Now we actually consume it
    const right = parseExpression();

    return left + right; // Obviously we would return according to the operator type here
}

return left;

Is something like this possible?

[nathan]

const peekable = lexer => ({
  here: lexer.next(),
  next() {
    const old = this.here
    this.here = lexer.next()
    return old
  }
})

const lexer = moo.compile({ws: /[ \t]+/, id: /\w+/})
const tokens = peekable(lexer.reset(`abc def ghi`))
tokens.here
tokens.next()

[alastairzotos]

That's nice but it's not very good with Typescript. Notable, this.here is just implicitly any.

I have this for now but I'd rather not have to write it:

let lastPeekedToken: moo.Token | null = null;

export const peek = (lexer: moo.Lexer, type?: ITokenType): moo.Token | null => {
    if (!lastPeekedToken) {
        lastPeekedToken = lexer.next();
    }

    if (type && (!lastPeekedToken || lastPeekedToken.type !== type)) {
        return null;
    }

    return lastPeekedToken;
}

export const consume = (lexer: moo.Lexer, type?: ITokenType): moo.Token => {
    const token = lastPeekedToken || lexer.next();
    if (lastPeekedToken) {
        lastPeekedToken = null;
    }

    if (type && (!token || token.type !== type)) {
        throw new Error(`Expected '${type}', got ${token ? token.type : 'eos'}`);
    }

    return token;
}

[tjvr]

I agree with Nathan; you can write this yourself :-) We’d rather keep Moo’s core simple.

For what it’s worth, I don’t see a reason you couldn’t add types to Nathan’s version.