search

soutaro / rbs-inline

Inline RBS type declaration
MIT License
231 stars 7 forks source link

Fix trivia tokens handling #67

Closed soutaro closed 3 months ago

soutaro commented 3 months ago

The tokenizer allows two tokens lookahead. This PR is to skip trivia tokens between lookahead tokens.

The skip parameter type annotation requires second lookahead token : to confirm it's not a skip annotation. This works.

# @rbs skip: untyped

But there may be some trivia tokens between skip and :.

# @rbs skip : untyped

And, the tokenizer didn't handle this well -- the second lookahead token was :tWHITERSPACE, not :kCOLON.

The issue is fixed with this PR.