search

kevinushey / sourcetools

Tools for reading, tokenizing, and parsing R code.
MIT License
77 stars 3 forks source link

Partial parse/tokenisation? #2

Open hadley opened 8 years ago

hadley commented 8 years ago

Not sure if you have this already, but if you do, it's not documented.

Could be own function, or partial argument to existing functions.

kevinushey commented 8 years ago

What would the partial argument do in this case? For tokenization it's at least it's (mostly) stateless so this should be easy. The exception being [ and [[, e.g. in tokenizing

foo[bar[[1]]]
foo[[bar[1]]]

We need to maintain a token stack to disambiguate when to tokenize ] vs ]].

hadley commented 8 years ago

partial would change it from returning an error to returning something with a partial flag.