Closed michal-pawlik closed 4 years ago
Seems there's edge case not covered here:
def test_error_reset_column_number(self):
lg = LexerGenerator()
lg.add("NEW_LINE", r"\n")
lg.add("NUMBER", r"\d+")
l = lg.build()
stream = l.lex("12\nfail")
stream.next()
stream.next()
with raises(LexingError) as excinfo:
stream.next()
assert excinfo.value.source_pos.colno == 1 # it fails with 3 instead
The same assertion works if added in test_error_line_number.
FYI here's a fix for colno when there's lexing error: https://github.com/nobodxbodon/rply/commit/4afbd7f12f200790d0466c23f1e0660b770e5ef6 , covering the edge case above.
When lexer is unable to find matching token,
LexingError
raises. This error contains source position for further debugging, but in current implementation it only points valid index (character number), while the line and colum numbers equal-1
.This PR fixes this issue by tracking column number in
next()
method ofLexerStream
and providing bothself._lineno
andself._colno
toSourcePosition
inLexingError
.