Migrate Cursorless to language server

[pokey]

Overview

One possible approach to #435 is to just turn the core of Cursorless into a language server. On the surface, it seems like it may not be a great fit, given that Cursorless doesn't do pretty much any of the things expected by a language server (eg jump to definition, find references, rename, list workspace symbols, autocomplete, etc). However, the lsp protocol is actually quite general, and Cursorless could just define custom requests and notifications for its own purposes. Here are the benefits of this approach:

• We'd easily be able to switch VSCode Cursorless to use this server, so that it also uses the same code path as other editors. The VSCode lsp example should actually work pretty well for us
• We get file synchronisation for free on the server side, as well as on the VSCode client side, using the node client and server lsp libraries
• We get file synchronisation for free in many other editors, using existing lsp client libraries (eg https://github.com/ballerina-platform/lsp4intellij or https://github.com/gtache/intellij-lsp for IntelliJ)
• We are using a well-defined and widely used protocol
• Many editors have built-in support for LSP, so we could possibly leverage that, or at least borrow code

The approach

Phase I: preliminaries

There are two main directions at the start, that can happen somewhat in parallel

Abstract away references to vscode in Cursorless

We will gradually move everything VSCode-specific behind the ide abstraction. We will make some of these function calls asynchronous. Those async function calls will eventually become round-trips to the lsp client.

Change sidecar extension to be an LSP server

The sidecar extension will use the node LSP server library to implement the LSP protocol. This library will keep an in-memory version of documents. We can listen to these changes and propagate the changes to VSCode. We could either propagate these changes by clobbering editor text, or support incremental changes. Incremental changes probably won't actually be too hard to support from client to server to VSCode, as the change notifications are very similar in format to the edits that we'd apply to VSCode. See server.ts from the VSCode extension sample for example code, keeping in mind that in that repo the server runs as a node subprocess, whereas here we're running it directly in the sidecar extension. Note that the work of getting the sidecar to implement this synchronisation from lsp server in-memory state to VSCode editor state is throw-away work.

The sidecar extension will need to start supporting our custom LSP messages, such as cursorless/command, selection / cursor synchronisation, hat updates, etc. This work is not throw-away work, as we will want to fold it into Cursorless eventually.

The editor clients will also need to be changed to be LSP clients. The document synchronisation should come for free from an LSP client library. They will need to handle / send our custom messages for cursor synchronisation, hats, and Cursorless command messages.

None of the code to do this LSP server implementation should require any changes to Cursorless itself in the short term.

Here are some things that will need to happen along the way:

• [ ] Define a custom message for Cursorless commands, maybe cursorless/command. This will be used to send cursorless command from editor client to server
• [ ] Add new synchronisation code for cursor positions, as today they only sync document contents
• [ ] Add new sync code for visible ranges
• [ ] We could use a custom notification to send hats from server to editor client, calling it eg cursorless/updateHats. This would happen in response to document changes sent from the client. The message would be a bit like workspace/applyEdit, where the document references include version numbers, enabling the client editor to reject the update if the version is old. Note that we prob want version number to incorporate cursor positions and visible ranges, unlike applyEdit

Initial work in Cursorless extension

In addition to the above two directions, we will create an implementation of the ide abstraction that defines how Cursorless will interact with a client using the lsp protocol. At the outset, it will just directly call a method on the sidecar to ask it to send an lsp message to the client, but that method call should be easy to replace with something that directly ends the lsp message in the future. This implementation of the ide abstraction will be used in place of the VSCode implementation when we're using the sidecar. Not sure how this will work where we need functionality from both, because the sidecar setup is still running in VSCode. We can see how this works out in the code. It might need to keep its own VSCode ide object and forward some things to it.

We'll first use this setup to implement the "follow" action and hat synchronisation

Phase II: proper language server

Once all of the references to VSCode are behind the ide abstraction, and we have https://github.com/cursorless-dev/cursorless/issues/945, we can

• [ ] move everything in the Cursorless extension that is outside the VSCode ide implementation into a language server
• [ ] migrate the code from the sidecar lsp server extension directly into the lsp ide implementation, rather than having it call out to the sidecar extension
• [ ] move the VSCode implementation of ide into a new VSCode lsp client extension that will replace today's Cursorless extension. All of the functions on this ide will become new LSP messages
• [ ] Add methods to the lsp ide implementation that will talk to the client
• [ ] Implement these new LSP messages on other editor clients instead of clobbering their state from VSCode

References

• https://github.com/volarjs/language-tools-starter might be useful

Old stuff

Actions

We'd need to look at our actions to figure out how to handle the fact that they sometimes go back and forth multiple times with the editor. Here are a few examples:

• for "bring", cursorless wants to highlight the newly inserted item
• For "chuck", Cursorless wants to highlight the item before it is deleted
• For "pour", we may perform multiple actions in sequence if there are multiple targets, to pour each one
• For all targets, we want a that mark that reflects the target after it has been modified. In the case of "bring" with multiple sources, we may end up inserting multiple things at a single location (eg "bring line air and bat and cap")

Possible approaches

• Return a list of things to do, some of which could trigger going back to the server. We'd potentially need a barrier to ensure synchronisation completes between steps
• Have intermediate messages sent from server to client. Could result in easier procedural code in server, as these intermediate steps could be transparent

For updating the "that" mark, could possibly have Cursorless set intermediate "that" mark and then keep it up to date with document changes.

We probably want a barrier no matter what, so that commands can linearise properly. Maybe look into hooking into synchronisation code so we can wait for it to happen. Possibly could use custom notifications or requests

[pokey]

update from meet-up 2023-10-31:

• we'll start by focusing on jetbrains, rather than thinking about how to support arbitrary editors
• to that end, @phillco will do a bit of research on which of the following options would be easiest from a jetbrains client perspective:
  • using Cursorless as an npm library by figuring out how to run a JS interpreter
  • using Cursorless bundled as a wasm
  • using Cursorless as an lsp server