Exception on $end Token

[FrankC01]

I'm getting this exception:

/x/parser.py:53: ParserGeneratorWarning: 4 shift/reduce conflicts
  return self.pg.build()
Traceback (most recent call last):
  File "core.py", line 33, in <module>
    parser.parse(tokens).eval()
  File "/usr/local/lib/python3.7/site-packages/rply/parser.py", line 60, in parse
    self.error_handler(lookahead)
  File "/x/parser.py", line 50, in error_handle
    "Ran into a %s where it wasn't expected" % token.gettokentype())
ValueError: Ran into a $end where it wasn't expected

When attempting to parse:

text_input = "print(4 + 4 - 2);"

Given this token list:

Token('PRINT', 'print')
Token('OPEN_PAREN', '(')
Token('NUMBER', '4')
Token('SUM', '+')
Token('NUMBER', '4')
Token('SUB', '-')
Token('NUMBER', '2')
Token('CLOSE_PAREN', ')')
Token('SEMI_COLON', ';')

And this parser:

from rply import ParserGenerator
from ast import Number, Sum, Sub, Print

class Parser():
    def __init__(self, mlex):
        # A list of all token names accepted by the parser.
        self.pg = ParserGenerator(mlex.get_tokens())

    def parse(self):
        @self.pg.production(
            'program : PRINT OPEN_PAREN expression CLOSE_PAREN SEMI_COLON')
        def program(p):
            return Print(p[2])

        @self.pg.production('expression : expression SUM expression')
        @self.pg.production('expression : expression SUB expression')
        def expression(p):
            left = p[0]
            right = p[2]
            operator = p[1]
            if operator.gettokentype() == 'SUM':
                return Sum(left, right)
            elif operator.gettokentype() == 'SUB':
                return Sub(left, right)

        @self.pg.production('expression : NUMBER')
        def number(p):
            return Number(p[0].value)

        @self.pg.error
        def error_handle(token):
            raise ValueError(
                "Ran into a %s where it wasn't expected" % token.gettokentype())

    def get_parser(self):
        return self.pg.build()

[alex]

I don't have time to super dig it ATM, but you may want to try resolving your shift/reduce conflicts. S/R conflicts can produce unexpected parsers.

[FrankC01]

Thanks for the quick reply... any reference to where I can go learn shift/reduce conflicts to know where that may be occuring?

[alex]

It's happening because of the expression: expression SUB expression/expression: expression SUM expression rules. Because these rules have equal precedence, either order-of-operations is valid from the parser's perspective (don't let your grade school math teacher find out!)

If you work through the example in https://rply.readthedocs.io/en/latest/users-guide/parsers.html#generating-parsers you'll see how precedence is used.

[FrankC01]

I was able to remove the shift/reduce by basically just commenting it out and changing the main production to "PRINT SEMI_COLON" and passing in "print ;" but the exception still occurs. I will look at your reference but it feels as though there should be an 'end' production to handle the lookahead?

UPDATE I added the precedence which took care of the shift/reduce but it didn't make a difference. Could it be using Python 3.7?

[FrankC01]

found the issue

prior to the parse I was exhausting the tokens

[mprofinder]

Hai, Can i know were you made the corrections. I am also facing a similar problem: /home/jm/Documents/Compiler Project/parser.py:38: ParserGeneratorWarning: 4 shift/reduce conflicts return self.pg.build() Traceback (most recent call last): File "main.py", line 14, in parser.parse(tokens).eval() File "/home/jm/miniconda3/lib/python3.7/site-packages/rply/parser.py", line 50, in parse t, symstack, statestack, state File "/home/jm/miniconda3/lib/python3.7/site-packages/rply/parser.py", line 80, in _reduce_production value = p.func(targ) File "/home/jm/Documents/Compiler Project/parser.py", line 27, in expression return Sub(left, right) TypeError: init() takes 1 positional argument but 3 were given