stop lexing/parsing so much

[gavinking]

Currently, we completely parse the current source file three times on almost every keypress.

• Once, for the autoedit strategy,
• then again for the syntax highlighter, and
• finally, to typecheck the code (ok, this one doesn't happen on quite every keypress, and it is asynchronous).

Now, I have not noticed all this stuff causing any perceptible lag in the editor, but it's still wasteful, and we should try and at least make the first two reuse the same parse tree, since they anyway happen synchronously I believe.

[ncorai]

Right now, the IDE is killing me when editing a 700-line ceylon file. It's madly stuttering while typing comments (!), because it insists on parsing everything after each new character I type, even if I haven't stopped typing yet. Besides reducing the number of passes, I would highly suggest:

1. Waiting for input pauses (like 0.8s or maybe user-configurable) before parsing.
2. Only reparsing local changes. If I'm editing a comment or inside a method, nothing else will be impacted. I'm sure it's doable to map the file's language structures to character indices in the file and thus to immediately identify where the changes are ocurring, where the boundaries of the surrounding entity are, and then to reparse that entity, compare the generated result with the previous version and assess impact to the other entities.

[gavinking]

Right now, the IDE is killing me when editing a 700-line ceylon file. It's madly stuttering while typing comments (!),

Well, I just copy/pasted Iterable (660 lines) into a new file and tried typing in comments in various places in the file. It's true that I experience a pretty minor slowdown right at the top of the file, but nothing really serious.

1. Waiting for input pauses (like 0.8s or maybe user-configurable) before parsing.

This implies that syntax highlighting would be an async process. I tried that, it's terrible. Doesn't work. It has to be synchronous, which means it also has to be every keypress.

1. Only reparsing local changes. If I'm editing a comment or inside a method, nothing else will be impacted.

We already do this. Within-token changes don't result in presentation updates. See PresentationDamageRepairer:66.

[gavinking]

One thing to note: I guess we do re-lex and re-parse the source on every keypress, even in the case of a within-token change. In theory, for an in-token change, we could just update all the tokens, adjusting their offsets. This would certainly save some work. However, given the constraints of the APIs we're working with, that might be very difficult to implement.

[ncorai]

My machine has an i7-2760QM (4 physical cores, 2.4GHz/boosts to 2.6GHz) with 8GB of memory, of which 2GB are allocated to Eclipse's heap. I restart Eclipse when I get north of 1.5GB (which happens daily, even though my project is still pretty small), to prevent GC interruptions. So Ceylon IDE is not (or should not be) disk-limited, or memory-bound, or thread-starved, and my CPU speed is at least average.

Now, here's a test I just did: I fiddled with Ceylon IDE and JDT while watching CPU utilization:

I. Ceylon IDE

a) Typing

At the time the editor stutters, CPU utilization does jump briefly to 100% for one core, while the other cores remain idle .

b) Saving

The build process seems single-threaded too. Not only that but, while that one core is pegged, I can type all I want: nothing gets displayed until the CPU core is available again, which seems to imply that the building is done by the UI thread.

II. JDT

a) Typing

Now I switch to JDT and start editing a comment (in a file with multiple classes, 1,400 lines): CPU stays completely flat. I edit a method content: no CPU usage until I pause and then a very small bump. Absolutely no slowdown, of course.

b) Saving

2 cores are utilized although among those two are no more pegged than one core would be under ceylon. Even while this is happening my editor remains completely fluid, which means that whatever is hitting those two cores is not a UI thread.

Notes: a) In practice, because of the windows scheduler, what I'm reporting above as a peak was sometimes distributed between 2 and 4 cores, but only one was really pegged at any given time, with the others instantly dropping back to almost idle. When I'm reporting 2-core use, really 4 or 5 cores were active at different levels but again no more than 2 would be significantly high at any point in time. b) In JDT my disk use never went past 5% in either operation and even in Ceylon IDE it stayed at 10% max, so obviously we're not IO-bound here.

Conclusion?

Based on the test, my problem seems related to Ceylon IDE using the UI-thread for lexing and building. Could that be the issue?

[gavinking]

After #1116, we're now doing another lex.

[gavinking]

Not going to happen in 1.1.