Incremental Highlighting

[thecoolwinter]

Description

This PR adds incremental highlighting to the text editor. When an edit occurs, it applies the edit in tree-sitter and calculates what other indices may need re-highlighting. This lets us take full advantage of tree-sitter and make our editor much more efficient for large code files.

Todo

• [x] Apply Edits to tree-sitter
• [x] Calculate changed ranges due to an edit
• [x] Invalidate ranges & queue highlights after an edit
• [x] Query tree-sitter for highlights for a range
• [x] Restrict all highlights to visible text
• [x] Highlight newly visible text on scroll
• [x] ~Fix memory bug~

Related Issues

27 🚀 Performance: Incremental Highlighting

45

55

21

Checklist

• [x] I read and understood the contributing guide as well as the code of conduct
• [x] My changes generate no new warnings
• [x] My code builds and runs on my machine
• [x] I documented my code
• [x] Review requested

[thecoolwinter]

This isn't quite ready for review, I've noticed that opening files leaks ~30-50Mb of memory and want to make sure it's not due to this code. The app will also allocate massive amounts of memory when files are opened, and it scales linearly with the size of the file. Opening

• a package-lock.json (133KB) allocates ~500MB of memory
• another JS file (3KB) allocates ~100MB After closing either of these files 30-50MB of memory stays allocated and stacks consistently each time the file is opened.

[austincondiff]

@thecoolwinter if the memory leak is coming from STTextView, can this be reviewed and merged?

[thecoolwinter]

@austincondiff working with Martin it looks like I was mistaken about there being a mem leak. It turns out the textView.addAttributes method takes longer with longer texts, and that was a red herring when combined with the memory usage of the editor. Looking at other syntax editors it looks like the memory usage is normal and we can ignore it as it does get released quickly soon after it's allocated.

What I'm struggling with now is finding a way to apply attributes that doesn't scale with document length and is fast enough to let us scroll the editor smoothly while applying edits. Tree-Sitter can absolutely keep up with scrolling, it's just TextKit that's slowing it all down right now.

That being said it's perfectly functional for smaller documents.

(I've also got to fix those lint errors)

[thecoolwinter]

This PR is ready for review. I replaced textView.addAttributes by modifying the textStorage object directly. When measuring highlight speed on a package-lock.json file this led to a speedup from ~100ms to ~10ms for each new line highlighted when scrolling. This is still not the best performance, but feels just fine for editing and can be improved in the future.

[austincondiff]

A 10x performance increase, not bad! 👏

I will review this soon.