if else - both statements always executing

[travisstaloch]

I'm trying to parse a language called pinescript which has if-then-else statements. I've extended the calc example to support many of its features. I'm currently stuck on this one. It appears that the else block is always executing no matter the result of the expr. Here is the output from the terminal when I parse the offending statement:

WARNING: 34 shift/reduce conflicts
WARNING: 11 reduce/reduce conflicts
calc > if true x = 10 else x = -1
DEBUG: ID ASSIGN expr x 10
DEBUG: ID ASSIGN expr x -1
DEBUG: {'x': -1} True 10 -1
10
calc > x
-1

As you can see I've added a cexpr (compound expression) along with expr to the grammar.

    @_('cexpr')
    def statement(self, p):
        return p.cexpr

    @_('ID ASSIGN expr')
    def statement(self, p):
        print('DEBUG:', 'ID ASSIGN expr', p.ID, p.expr)
        self.names[p.ID] = p.expr
        return p.expr
...
    @_('IF expr statement ELSE statement')
    def cexpr(self, p):
        print('DEBUG:', names, p.expr, p.statement0, p.statement1)
        return p.statement0 if p.expr else p.statement1

    @_('IF expr expr ELSE expr')
    def cexpr(self, p):
        # print(p, p.expr0, p.expr1, p.expr2)
        return p.expr1 if p.expr0 else p.expr2

Any idea how I can make only the correct branch statement evaluate? Please let me know if any additional info is needed to understand whats happening.

[dabeaz]

This is not so much a SLY issue as it relates to the overall strategy of parsing expressions. The calc example in PLY/SLY is performing immediate evaluation of the expression as it goes. In order to make this work, you need to change all of this into some kind of delayed evaluation. One way to do this would be to build an abstract syntax tree of the expression and to create some sort of evaluation procedure for obtaining its value when you need it. This is a skeleton:

def eval_expression(ast):
       ... evaluate the supplied expression ...

@_('IF expr expr ELSE expr')
def cexpr(self, p):
      if eval_expression(p.expr0):
          return eval_expression(p.expr1)
      else:
          return eval_expression(p.expr2)

The overall idea is that each branch of the conditional will be presented as an unevaluated expression. Based on the outcome of the test, you'll evaluate only one of the branches and throw the other one away.

[travisstaloch]

Thanks so much for the advice! Using delayed ast evaluation worked well for me. After I read your message, I looked again at the 'AST Construction' section of https://github.com/dabeaz/sly/blob/master/docs/sly.rst which led me to these changes.

Its working now. Basically, just returning ast tuple instead of doing assignment and using an eval_ast() method from my parser's parse() method and the offending if-then-else handler. Maybe I'll convert other sections to use ast as needed.

    @_('ID ASSIGN expr')
    def statement(self, p):
        return ('assign', p.ID, p.expr)
...
    @_('IF expr statement ELSE statement')
    def cexpr(self, p):
        if p.expr:
            return self.eval_ast(p.statement0) if type(p.statement0) == tuple else p.statement0
        else:
            return self.eval_ast(p.statement1) if type(p.statement1) == tuple else p.statement1
...

    def eval_ast(self, ast):
        if(ast[0].split('-')[0] == 'assign'):
            return self._assign(*ast[1:])

    def parse(self, tokens):
        result = super(TvParser, self).parse(tokens)
        if type(result) == tuple:
            return self.eval_ast(result)
        else:
            return result

I learned about this project after watching your talk at pycon 2018. Really enjoy watching and learning from your presentations on youtube. Keep at it!

Now, to fix another bug thats popped up after these changes and then attempt to implement functions. I think functions are going to force me to use the ast approach throughout... Thanks again.

[travisstaloch]

I have been tinkering with my sly parser and lexer and trying to create an interpreter. I found a project that uses LLVM for this by one of your students. It looks very promising. I just wanted to share in case anyone else is looking for similar: gone lang

[dabeaz]

The code you've linked is from a past version of the compilers project that I teach from time to time. It's not really an "interpreter" in the traditional sense. However, many of the issues surrounding the evaluation of if-else, functions, and other things will still apply.

[travisstaloch]

I guess you would call the gone lang project a compiler. As is, it uses clang to create an executable when run with: python -m gone.compile examples/mandel.g. I'm trying to figure out how I can make it compile my .pine files to llvm ir ( .ll files ).

I basically want to make them into modules that I can use from python, passing pandas dataframes or numpy arrays along with other standard python data types. Making the .ll files is easy as its already being done, just saved to a tempfile. Not sure how to pass in and process something like a dataframe or np array though...