Handling end of file in the lexer

[SeanDS]

PLY has t_eof token support so you can detect things like open brackets. Is it possible to do this in SLY? If not, the code from PLY seems (at least at a glance) to be quite straightforward - is this something you'd accept a PR for?

[dabeaz]

The tokenize() method is a generator. Can this not be accomplished by wrapping it in some way? For example:

class MyLexer(Lexer):
    def tokenize(self, text):
           yield from super().tokenize(text)
           # Do whatever special EOF handling needs to take place here
           ...

[SeanDS]

Good idea.