Parser and interpreter API for implementing own editor as separate process

I thought a lot about making extensions for the shell. I think, one of the most powerful things would be to simply expose a JSON API for the editor. This could make possible to implement a fully custom editor. There are languages that are more powerful than Go. For example, I want to develop an editor in Crystal or D.
It should be able to:

get the content of right/left prompt from the shell
get values of variables from the shell (for example edit:max-height)
get completions from the shell
get syntax highlighting, and compilation check

It would manage on its own how to:

handle keybindings
draw to the screen
switch modes

It could also implement completion in a compiled language, which is faster, and is more comfortable to write than Elvish in most cases when dealing with completion processes. For example, I want completion mode to open automatically, and typing would automatically filter the candidates (see how the command line works of the Kakoune editor). This would only be possible in a compiled language.

It could implement custom modes. For example, it would only query the directory history for the histlist mode, and use its own matching and sorting engine (even very complex ones) with blazing speed.

It could simply query the command history and give autosuggestions based on it; using its own, complex, and fast matching engine.

How it would work?
When the command exits, and the shell's editor activates, it wouldn't draw anything to the screen, instead it would launch a new process (the custom editor), and use its input/output for communication. (The editor could open /dev/tty for drawing to the terminal.)
It would send the command to the shell (using the API), and exit, when the command line is ready for evaluation.
If the editor process exits with nonzero, the shell's built-in editor would take the control over the drawing.

This would be a very big step forwards. I think it is easier to implement (a JSON conversion tool is available in the shell), than all the proposed editor features. These features then could be implemented in another language which is easier to write and runs faster than Go.

What do you think about it?
(I know I can't describe my ideas in the most easily understandable way, so ask me if you don't understand something.)

Correct me if I am wrong, but you are essentially proposing:

The Elvish process runs the Elvish interpreter and some of editor core, and exposes an API to manipulate them via IPC
Another process that implements the editor communicates with the Elvish process.

While this can definitely work (it is similar to how Emacs's C/S mode works, for instance), the primary problem is that the API surface area will be very large. Large API surface area makes changes more costly and limits the development of both producer (Elvish) and consumer (the external editor).

It is much simpler if Elvish simply exposes the interpreter via an API. The basic API of an interpreter is very simple; written in Go:

interface Interpreter {
  interprete(code string) error
}

This definition is schematic because Go interfaces are not suitable for defining IPC API, and the return value should probably be more sophisticated; but it illustrates the point. If you wrap Elvish's interpreter in an IPC API, you can start implementing an external editor right away. In fact, this is exactly what elvish -web does: it exposes an interpreter API on an HTTP path /execute:

https://github.com/elves/elvish/blob/a8d8e6405eb14891c6b1e5d8aa44a1d0ca61b25d/program/web/web.go#L48

The implementation of that handler is essentially "eval the HTTP request body and return results in the HTTP response body":

https://github.com/elves/elvish/blob/a8d8e6405eb14891c6b1e5d8aa44a1d0ca61b25d/program/web/web.go#L64-L87

@xiaq The shell would not use the editor core, so it wouldn't need to expose anything from the build-in editor. The buffer, cursor movement, insert/delete, modes, history, etc. is handled by the external editor process.

What the shell would expose:

simple parser api for syntax highlighting e.g. the editor requests syntax highlighting on a string, and the shell returns a hash with ranges and styles
simple parser api for compilation error checking, e.g. the shell would return a range where the error is, and the error message, or return OK if it's correct
parser api for completions command: exa ~; kak $var/code/crystal/ the value of the $var variable is /home/username, and the cursor is at the end of the line It would return and array with kak and /home/username/code/crystal (so the current command is parsed). If the cursor is on a variable name, then it would return completion candidates instead of the parsed arguments.

The editor could:

evaluate commands in the shell while running (e.g. cd - from directory history mode)
return its buffer as the accepted command for evaluation (it must be the last message, and return with 0 after it)

What it would simplify:

command history, directory history, editor hooks, and text manipulation commands could be implemented entirely on the editor side
this would allow to disable the history daemon in the shell, and enable the editor to use its own (more powerful) history storage

The web editor could also use this API.

OK, I see what you mean now; your proposal is kind of complementary to the interpreter API I formulated. You are basically saying that Elvish should expose something like the language server protocol (in addition to the interpreter API). If you have not heard of LSP, you should visit that website now.

I don't think that Go is fundamentally a bad language for implementing editors, especially for implementing a terminal-oriented one; I am interested in this "external editor" approach, although I don't yet see how any editor feature can be dramatically simplified by switching to another language. However, this point is not very important. Exposing an LSP API from Elvish has wider benefits than just enabling an alternative terminal-based editor: it also enables integration with other editors like Vim, VS Code.

elves / elvish