Investigate DFA compilation times

[pokey]

They are up over 700ms for both me and @AndreasArvidsson. We should do another git bisect to see what caused it

[pokey]

@AndreasArvidsson suggested that https://github.com/cursorless-dev/cursorless/pull/2457 is the likely culprit, and I'm inclined to agree

[AndreasArvidsson]

I wonder if we could claw back some of the compilation time by removing cursorless_surrounding_pair_force_direction

[AndreasArvidsson]

@phillco

I've done some benchmarking. All figures are in milliseconds and performed with a restart in between in a python file.

• main: 700
• without any modifiers: 180
• Without any marks: 500
• without cursorless_surrounding_pair_force_direction: 660
• Without cursorless_surrounding_pair_scope_type: 560
• Without cursorless_surrounding_pair_scope_type_plural: 560
• Without cursorless_surrounding_pair_scope_type & plural: 450
• Without cursorless_glyph_scope_type & plural: 580

It's obvious that surrounding pair is dfa intensive. A reason for this is that surrounding pairs is actually multiple list and captures. We have a list for pairs that are only selectable, but not wrapable, we have a list for pairs that are both selectable and wrapable, and we have a list for pairs that are just wrapable, but not selectable.

I think we could increase dfa performance quite a bit if we just did a single list with all the scope types. This means that when we read our csv files instead of creating a bunch of different lists we would basically create a single list with all the scope types. Give the value as something unique so we can distinguish them. eg surroundingPair.parentheses and then when the user speaks that scope we take the identifier and construct the actual scope. This would get rid of multiple Talon list captures and make it just a bag of words.

The only scope that we can't do this with and that still needs to be its separate capture is cursorless_glyph_scope_type. The reason for this is that it has an argument <user.any_alphanumeric_key>. In theory we could listen for changes to the registry and pull this out and construct a single list, but that feels a bit to hacky?

What I'm proposing is that we make a single list for all the simple scope types(line, funk, etc), all the surrounding pairs(round, diamond), and all the custom regexes. This hopefully would decrease dfa compilation time quite a lot. I can't of course know how much until I actually try it and this is quite an invasive rewrite. We have of course thousands of grammar test so I'm not worry that I won't get it right. It's more a matter of if this is a direction we prefer? Unfortunately I don't think we have much of a choice if we want to keep compilation time down.

[phillco]

Re cursorless_surrounding_pair_force_direction unfortunately we found these are still referenced in the documentation: https://github.com/cursorless-dev/cursorless/issues/2625