How to peek tokens and access extras?

[irh]

I'm making use of Extras to keep track of current line information, and in my parser I also want to peek the next token.

The trouble is that if I wrap the Lexer using Peekable then I no longer have a way to access the .extras field.

lexer.token is mentioned as a solution in #82, but that doesn't seem to be available anymore.

I assume I can make my own peekable iterator that will give me access to .extras, but before I go down that path I'm wondering if I'm missing something?

Thanks for the great library!

[maciejhirsz]

Hey! You are not missing anything I'm afraid. I think it might be useful for Lexer to implement a custom peekable method that returns a custom peekable iterator which also derefs to Lexer itself so you can use all the methods (ditto for SpannedIter). That would probably also be a good first issue :).

[irh]

OK thanks, I'll have a go at implementing it then =)

[irh]

I took a quick look, and it seems like there's a potential issue with the idea. For a Peekable iterator we need to implement next() to either return the peeked value, or the next token from Lexer::next().

By auto-dereferencing Peekable to Lexer we're then making both the Peekable::next() and Lexer::next() methods available in the same iterator (i.e. via fully qualified syntax), and mixing the use of the two with peek() would cause odd behaviour.

[maciejhirsz]

Ah, right. Well, in that case it will just need to be boring and add a proxy method forspan, and one for extras that returns &mut T.