Change lexer state according to parser state on the fly

[brchiu]

If we want to change lexer states according to parser states, e.g. according to certain token combination patterns, it is not possible to do so because parse method of Parser class uses tokens that be produced from Lexer's tokenize method.

To be exact, I would like to implement similar behavior in asn2wrs.py that changes lexer state after WITH SYNTAX or in specific states, e.g. p_ObjectSet, p_ParameterList... .

Is it any alternative way to do it ?

Thanks.

[brchiu]

I record the initial lexer of Lexer1 inside MyParser class, then to switch to Lexer2 in specific states to do the trick.

Thank you for this wonderful project.

class Lexer1(Lexer):
    ...

class Lexer2(Lexer):
    ...

class MyParser(Paser):
    def __init__(self, lexer):
          self.lexer = lexer

    @_("TOK_WITH TOK_SYNTAX lbraceignore SyntaxList rbraceignore")
    def WithSyntaxSpec(self, p):
        return p[3]

    @_("braceignorebegin TOK_LBRACE")
    def lbraceignore(self, p):
        return p[0]
    @_("")
    def braceignorebegin(self, p):
        self.lexer.begin(Lexer2)

    @_("braceignoreend TOK_RBRACE")
    def rbraceignore(self, p):
        return p[0]

    @_("")
    def braceignoreend(self, p):
        self.lexer.begin(Lexer1)

if __name__ == "__main__":
    lexer1 = Lexer1()
    lexer2 = Lexer2()
    parser = MyParser(lexer1)
    parser.parse(lexer.tokenize(data))