Improve some of the combinators as to give better tokens thus better highlight results.
I tried to explain the whys on the commit messages but if they are not that clear ping me so I can come up with a better description for this PR. 😄
Before After
Things that I already mapped and we will have to do (but I was thinking about doing it in other PRs, wdyt?):
Fix the combinator for strings for some cases (for example, the "V +" is being tokenized as something like [:punctuation, :variable, :whitespace, :operator]) and it should be a single unit
Fix the combinator for the interpolation characters, that as far as I can tell are not being tokenized at all
Add the string token to the binary combinator (<< >>)
Fix the combinator for module attributes (e.g. -module, -import)
The overall combianators organization, maybe split some into smaller units and compose from there? (for example, have a erlang_letter = ascii_char([?A..?Z, ?a..?z) and compose it on larger combinators)
Have some kind of pattern for what is a final token combinator and what can be composed
But that can be discussed waaaay later. 😄
I've been based all my changes on the "semi-official"erlang/spec, mostly the Grammar Appendix.
Improve some of the combinators as to give better tokens thus better highlight results.
I tried to explain the whys on the commit messages but if they are not that clear ping me so I can come up with a better description for this PR. 😄
Before
After
Things that I already mapped and we will have to do (but I was thinking about doing it in other PRs, wdyt?):
"V +"
is being tokenized as something like[:punctuation, :variable, :whitespace, :operator]
) and it should be a single unit<< >>
)-module
,-import
)erlang_letter = ascii_char([?A..?Z, ?a..?z)
and compose it on larger combinators)token
combinator and what can be composedBut that can be discussed waaaay later. 😄
I've been based all my changes on the "semi-official" erlang/spec, mostly the Grammar Appendix.