Generate grammar at runtime

[codeyash]

Hello

Like http://pegjs.org/online

I want to define rule in plain files and parse accordingly.

Is it possible? If not please add this.

[d-frey]

The link you gave is not a plain grammar file, it also contains code. We usually keep these layers separate, but we do have a tool in the examples section that might be what you are looking for: abnf2pegtl. It can convert an ABNF grammar (slightly adapted to PEGs instead of CFGs) into PEGTL source. abnf.abnf is an example input that shows the grammar of the ABNF files we accept.

Is this what you are looking for or were you asking for something else?

[codeyash]

ok Please let me try to explain more. Its new to me.

This is plain grammar file https://github.com/igordejanovic/Arpeggio/blob/master/examples/csv/csv.peg

Corresponding code file which parses above peg file https://github.com/igordejanovic/Arpeggio/blob/master/examples/csv/csv_peg.py

This is PEG parser library which in written in python. I'm looking same in C++.

[d-frey]

csv.peg is a plain grammar file, but there is no standard - and that's the real problem here. Each library/toolset defines its own grammar to parse grammar files. This means you can not simply take "a grammar file" from one library/tool and use it with another library/tool.

We decided to use ABNF as a starting point because at least that is defined in RFC 5234, so it is the closest we could come to a standard. But an ABNF assumes a CFG, not a PEG so we simply had to adapt it. We kept the adaptions as minimal as possible, if you look at the source code at examples/abnf2pegtl.cc, you'll find a long comment at the beginning documenting all differences to an ABNF.

I think that abnf2pegtl is the best we can currently offer and it should do what you need. You can #include the source from another file where you can add namespaces, other include, rules, etc. as you like and, of course, you'll have to write actions to make use of the grammar.

Looking at Arpeggio, you might also look for more than just a way to convert a grammar into the tool's source format. If you are looking for an automatically generated parse tree (or an AST) that you can use immediately, we do not have something in this area (yet).

[codeyash]

Exactly yes If you are looking for an automatically generated parse tree (or an AST) that you can use immediately, we do not have something in this area (yet)

ok I will explore more. Thanks.

[d-frey]

OK, I see. I adapted the title accordingly as the main issue is the generation of a parse tree / AST. Enhancing abnf2pegtl (or creating a new version for that task) should then be easy enough.

I talked to Colin and we have some ideas that we'll explore in the next days on how to create the actual parse tree / AST structure, hence I'll keep this issue open for now and update when we have more information.

[codeyash]

Problem is not only AST but to generate AST based on plain grammar file.

So parsing and generating AST will need two things from user i.e. grammar file and targeted code file. After that I have to take care all other aspects. Grammar must be dynamic although rules may(should) follow same pattern.

That python library is excellent but I want it in c++ and your library looks solid although I failed to find my main AIM i.e. AST

I will look forward for your opinions.

[ColinH]

Building an AST is relatively easy with the PEGTL, however the one thing that it very deliberately does not support are dynamic grammars, i.e. building parsers from grammars loaded at run-time. The PEGTL uses C++ compile-time mechanisms to build a grammar from individual parsing rules, which helps make the library small and simple, and is very often sufficient.

For dynamic grammars the PEGTL is simply not the right tool, and it probably never will be since it would change, well, basically everything, creating a fundamentally new library. The only way would be to dynamically use a C++-compiler to compile PEGTL grammars at run-time. Otherwise you are beyond the design-space of the PEGTL.

Good luck with your project, we hope you find something more suitable to your requirements!

PS: Perhaps it would be easy to make a parser library following the same structure as the PEGTL but with interface classes and virtual functions instead of templates, allowing for dynamic grammars. It might also be difficult or impossible, we haven't tried it yet, but please let us know if you do :-)

[codeyash]

Ok Thanks for info. I've limited knowledge to create full parser at the moment.

I found one https://github.com/yhirose/cpp-peglib. It may solve my problem not sure yet.

Thank you!