darklang / dark

Darklang main repo, including language, backend, and infra
https://darklang.com
Other
1.65k stars 90 forks source link

Add semantic tokenization tests #5381

Closed OceanOak closed 2 months ago

OceanOak commented 2 months ago

This PR fixes semantic tokenization, and adds tests for it

OceanOak commented 2 months ago

Todos and things to consider for a follow-up PR:

Notes in case we decide to capture all symbols some day:

currently, we are not tokenizing the following:

StachuDotNet commented 2 months ago

Those are great comments - I added a link in #5259 to that comment so we don't lose track. Quick thoughts: