operator-pending mode interrupted by redraw or job/event

[justinmk]

Followup of https://github.com/neovim/neovim/pull/3246#issuecomment-136294721 (cannot find an existing issue for this).

To reproduce, use :term to run a program that refreshes often (such as top):

:term top
<c-\><c-n>
<c-w>       <-- operator-pending mode is canceled by the next redraw

Solution suggested by @tarruda :

we need to handle K_EVENT in every function that reads another character for the purpose of finishing an operator pending mode. It's not an absurd amount of work, but it requires a search through normal.c and ops.c

[ZyX-I]

<C-w> does not start an operator-pending mode. It starts “control-w mode” (really just a function which calls Neovim-specific getchar, and acts based on this). This is also the case for absolutely any “compound” command like gu in normal mode: g starts “g mode”, u is a character inside it.

[tarruda]

really just a function which calls Neovim-specific getchar, and acts based on this

True, but it is simpler to theorize about the program structure when you consider every character-driven event loop as a separate mode, or simply a state where the program is waiting for events.

While working on #3413 I was forced to read and understand every major mode function and how character translations(mapping and even internal translations such as "stuffing") work, and I strongly believe that a major contributor to Vim internal complexity is exactly how these parts are implemented.

Consider how #3413 generalized the idea about modes as separate states of a pushdown automaton. Can we improve on that and do something similar for the internal modes such as "ctrl+w mode" or "g mode"? It turns out we can, and using a very old idea: By modeling these internal modes as states of a NFA.

What I mean is that we can use the same algorithm for parsing regular expressions to match Vim commands with callbacks, and this is possible because vim internal modes are just simple regular languages. We can also reuse the same NFA algorithm to reimplement the mapping engine, since mapping is basically string recognition and translation.

So with a simple NFA algorithm we can remove a lot of complexity from getchar.c and basically describe the major modes as a set of string->callback associations. This is something I have planned for after the first release and will bring a number of improvements to Neovim:

• Greatly simplified code
• Allow events to be processed in the middle of mappings
• Allow events to be processed in the middle of internal modes

This refactoring will be a follow up of #3413 which blocked events in mappings/internal modes to avoid interrupting user key sequences.

[ZyX-I]

@tarruda In site of this refactoring does it make sense to reopen #421?

[tarruda]

@tarruda In site of this refactoring does it make sense to reopen #421?

That seems very aligned with what I envisioned with this NFA refactoring, except that each mode is its own "language"(eg: each mode has its own bnf/grammar).

I'm just not sure if it would make sense to expose Vim modes outside editing context. We could allow programs to create their own modes and expose a state machine to parse input and invoke callbacks, but mode transition seems very application-specific.

For example, Vim starts in normal mode, but it can very well be configured to start in insert mode and jump to normal mode when ctrl+o is pressed. In this scenario, the pushdown automaton will have insert mode at the top of the stack with normal mode being pushed when ctrl+o is pressed.

In any case, this is pretty much what most parser or state machine generators do: Associate the matching of strings with application-provided callbacks. If you think about it, switching modes is nothing but switching parsers. The only thing I can see a standard parser generator won't provide is the ability to parse mappings, since parser generators usually operate during compilation time and mappings are dynamic.(though parsing libraries like lpeg also work with dynamic languages)