miho / VMF-Text

Powerful Grammar-based Language Modeling Framework
Apache License 2.0
10 stars 0 forks source link

allow grammar rewriting for advanced features #7

Open miho opened 6 years ago

miho commented 6 years ago

To support all planned features we need grammar rewriting for automatically adding additional actions.

We refused to rewrite the user-defined grammar since it is difficult to preserve all features provided by ANTLR4 and compatibility is a high-priority feature. But we found a way to preserve the full grammar spec even though we introduce actions into the grammar for improved LP (#2).

For the rewriting we reuse the UPModel but instead of using the full VMF-Text API we do the rewriting with ANTLR4s TokenStreamRewriter. This simplifies the implementation dramatically.