search

alex / rply

An attempt to port David Beazley's PLY to RPython, and give it a cooler API.
BSD 3-Clause "New" or "Revised" License
381 stars 60 forks source link

Correct line and column number in source position when raising LexingError #95

Closed michal-pawlik closed 4 years ago

michal-pawlik commented 4 years ago

When lexer is unable to find matching token, LexingError raises. This error contains source position for further debugging, but in current implementation it only points valid index (character number), while the line and colum numbers equal -1.

This PR fixes this issue by tracking column number in next() method of LexerStream and providing both self._lineno and self._colno to SourcePosition in LexingError.

nobodxbodon commented 3 years ago

Seems there's edge case not covered here:

    def test_error_reset_column_number(self):
        lg = LexerGenerator()
        lg.add("NEW_LINE", r"\n")
        lg.add("NUMBER", r"\d+")
        l = lg.build()
        stream = l.lex("12\nfail")
        stream.next()
        stream.next()
        with raises(LexingError) as excinfo:
            stream.next()

        assert excinfo.value.source_pos.colno == 1   # it fails with 3 instead

The same assertion works if added in test_error_line_number.

nobodxbodon commented 3 years ago

FYI here's a fix for colno when there's lexing error: https://github.com/nobodxbodon/rply/commit/4afbd7f12f200790d0466c23f1e0660b770e5ef6 , covering the edge case above.