Conditional lexing

[domingoUnican]

Dear all,

I am having problems trying to do conditional lexing: Here is a sample code:

import sly

class CalcLexer(sly.Lexer):
    tokens = { NUMBER, PLUS, MINUS, TIMES, DIVIDE, LBRACE }
    ignore = ' \t\n'

    NUMBER = r'\d+'
    PLUS = r'\+'
    TIMES = r'\*'
    MINUS = r'-'
    DIVIDE = r'/'
    LBRACE = r'\{'

    def LBRACE(self, t):
        raise sly.LexerStateChange(BlockLexer, t)

class BlockLexer(sly.Lexer):
    tokens = { RBRACE, NAME, VALUE }
    ignore = ' \t\n'

    NAME = r'[a-zA-Z_][a-zA-Z0-9_]+'
    VALUE = r'\d+'
    RBRACE = r'\}'

    def RBRACE(self, t):
        raise sly.LexerStateChange(CalcLexer, t)

if __name__ == '__main__':
    lexer = CalcLexer()
    for tok in lexer.tokenize('3 + 4 { foo bar 1234 } * 6'):
        print(tok)

The expected output should be:

Token(type='NUMBER', value='3', lineno=1, index=0)
Token(type='PLUS', value='+', lineno=1, index=2)
Token(type='NUMBER', value='4', lineno=1, index=4)

However, the result is: Token(type='NUMBER', value='3', lineno=1, index=0) Token(type='PLUS', value='+', lineno=1, index=2) Token(type='NUMBER', value='4', lineno=1, index=4) Traceback (most recent call last): File "prueba.py", line 31, in for tok in lexer.tokenize('3 + 4 { foo bar 1234 } * 6'): File "/home/user/Git_Repositories/sly/sly/lex.py", line 400, in tokenize tok = _token_funcs[tok.type](self, tok) File "prueba.py", line 15, in LBRACE raise sly.LexerStateChange(BlockLexer, t) sly.lex.LexerStateChange: (<class 'main.BlockLexer'>, Token(type='LBRACE', value='{', lineno=1, index=6))`

[dabeaz]

Although LexerStateChange appears to be defined in the code, it doesn't seem to be used anywhere nor is it documented (an abandoned idea at one time perhaps). The proper way to invoke a state change is to use the lexer.begin() method. See the CHANGES file, dated 4/1/2018 for an example of usage.

[domingoUnican]

Works! Thank you!