Tokenizer trips up around range and rangeincl if no spaces

JSAbrahams / mamba

🐍 The Mamba programming language, because we care about safety

MIT License

88 stars 4 forks source link

Tokenizer trips up around range and rangeincl if no spaces #254

Closed JSAbrahams closed 2 years ago

JSAbrahams commented 2 years ago

The following produces what we expect: 0 ..= 2 A Int, RangeIncl, and Int token.

Whereas the following: 0..=2 when there are no spaces, produces Int, Range, Equals, and Int token. (Not verified in test, but something like this).

Unit test this to make sure tokenizer deals with this situation.

Expected behavior

The former.

JSAbrahams commented 2 years ago

Issue was solved in #257