Building parse trees from very large files (e.g.> 200MB data file) can occupy considerable memory. If the transform is applied immediately at the point that the syntactical construct is recognised, memory can be economised, particularly if only certain information is relevant. This would also allow streaming data. Lark already has this capability via the transform option to Lark, which in Lerche is simply executed following the completion of parsing.
Considerations:
If an item for the parse tree is processed by user-supplied functions, the type restraints for nodes of {Tree,Token} are no longer true. Some form of type parameterisation is needed to avoid type instability. This should be under the control of the user.
Building parse trees from very large files (e.g.> 200MB data file) can occupy considerable memory. If the transform is applied immediately at the point that the syntactical construct is recognised, memory can be economised, particularly if only certain information is relevant. This would also allow streaming data. Lark already has this capability via the
transformoption toLark, which inLercheis simply executed following the completion of parsing.Considerations:
{Tree,Token}are no longer true. Some form of type parameterisation is needed to avoid type instability. This should be under the control of the user.