Help Required for developing expression evaluation parser!

[aamironline]

Hello,

Hope you are doing great! Thank you very much for developing such a wonderful library.

For one of our project (https://github.com/maniartech/x/), we are developing expression evaluator in Golang. We have used expr library for now. But would like to replace with our own evaluator/parser. We do not want to write from the scratch. The pigeon seems to provide perfect solution for our requirements. This seems to be easy. I started with changing calculator parser provided with the pigeon library. However, it is not working out.

I am new to PEG ecosystem, can anyone help me develop this parser! I am open to start a new MIT licensed open source project for this expression evaluator.

Thanks and Regards.

The expression evaluator has got following syntax!

-a-----a-----b-----b----------c----d------------l-------c----e---
SUM(AVERAGE(varA, obj.key, 10 + (2 + 3), [1, 23, 4]), 10 - 2, x)
-------------e-----e----------e--------------e-----------e----e---

-a---o-k---v----k---v----k------v--------
KEYS({ a: true, b: false, c: [1, 2, 3] })
------------t---------t-------l------------

- a: Function Name
- b: Variables, it should be JSON like variable with dot notation support
- c: Expression, Could be mathematical (+ - * /), comparison(== != > etc..), boolean (| & ~) etc...
- d: Bracketed Expression
- e: Comma separated function parameters
- o: JSON like object with nested object and array support
- k: The object key
- v: The object value
- l: The list (array) value in object 
- t: The boolean value

[breml]

Hi @aamironline

Thanks for your kind words. I am the current maintainer of pigeon and I am happy to provide you with some hints and links you might find helpful in your endeavor.

• Start by focusing on the parsing part and ignore the evaluation / processing part in the first step. In this regard, the calculator example is not the best / easiest, because it does combine the parsing and the evaluation into a single processing step. Me personally prefer to separate these two steps so that the parsing just returns an abstract syntax tree (AST), which is then used for the further processing. If performance matters, you can always decide to optimize it later and again combine the two steps (guarded by the unit tests, you have developed a long the way).
• I normally use https://pegjs.org/online for the first few steps (just to get going). All the basic constructs are the same as in pigeon (but there are differences for the more complex grammars).
• I propose you define the necessary types for an abstract syntax tree for your expressions. You could have a look at the logstash-config example linked below, which does also support minimal expressions and conditions.
• Start with the very basic building blocks and add more features over time. E.g. start with the definition of what kind of white spaces are allowed, what characters are allowed for function names, what kind of values (e.g. strings in double quotes, integers, floats, booleans) are supported.
• Use go test to test your current state of the grammar parsing (again see the linked examples below) to prevent regressions, when every you alter the grammar.
• I usually try move as much of the Go source code as possible out of the .peg. Therefore I usually have a file ending on _helper.go, where all the helper functions are located, that are called from the code generated by pigeon. In the .peg file, I normally only have short statements like return xyz(arg1, arg2). The return values of xyz match the expectation of pigeon, that is they return an interface{} and an error.

Over all, I feel like you might find a lot of inspiration in https://github.com/breml/logstash-config/blob/master/logstash_config.peg

An other good example is https://github.com/bytesparadise/libasciidoc/blob/master/pkg/parser/parser.peg

Use the two links in the Wiki to find more examples.

[aamironline]

Wow, what a detailed response! Thank you so much. I'll try your suggestions and provide my progress in this thread. Thanks again 👍🏻