search

lezer-parser / lezer

Dev utils and issues for the Lezer core packages
33 stars 1 forks source link

Incremental parsing of large document breaks when changing something after a skip node #53

Closed GerwinBosch closed 2 months ago

GerwinBosch commented 3 months ago

Hey, @TriplyDB we're working with CM and lezer for our SPARQL editor. However, we've encountered an issue when we introduce a lot of comments within large queries.

Our issue is that when, within a large document, tokens after/within an uncertain amount of skip-tokens when you uncomment a syntactically valid line, the parsing seems to fail (indicated by the validation and highlighting not working). Uncommenting the first comment token doesn't cause this issue. Any below do.

cm6-comment-parse-issue.webm

Luckily, I encountered #24 and its minimal reproduction was also able to reproduce our current problem. I've forked, updated the dependencies and tweaked the description to match our issue. The fork can be found here

marijnh commented 2 months ago

That reproduction repo is using ancient 0.x versions of @lezer and @codemirror packages. Does the problem still occur if you use current versions?

GerwinBosch commented 2 months ago

Oops, good point (I missed that -.-). I've updated the dependencies in the fork and can confirm the issue is still reproducible.

marijnh commented 2 months ago

Thanks for the comprehensive reproduction code! I was able to isolate it and found a fix, which is implemented in attached patch.