search

cenotelie / hime

Apache License 2.0
27 stars 4 forks source link

Separator tokens in Abstract Syntax Tree #31

Closed woutersl closed 8 years ago

woutersl commented 8 years ago

Original report by Anonymous.

By default separator tokens are discarded during parsing. Is there a way to have them in AST structure?

woutersl commented 8 years ago

Original comment by Laurent Wouters (Bitbucket: 557058:675792b6-d731-4823-9f7d-c6dfcb2df2b5, ).

There is no easy way to do this. Separator tokens should not be syntactically significant (like spaces in the majority of the programming languages). A workaround would be to not define a separator terminal (Separator option) in the grammar and add the terminal where appropriate in the grammar rules.

woutersl commented 8 years ago

Original comment by Laurent Wouters (Bitbucket: 557058:675792b6-d731-4823-9f7d-c6dfcb2df2b5, ).

Separator tokens are designed to be automatically dropped. This goes against the original intended design. A workaround is to not define a separator token and handle it manually in the syntactic rules.