mainej
commented
2 years ago Thanks for starting this thread @ericdallo. I've been thinking about some of the details. Here's an enumeration of the features that lsp4j provides, or at least the features I'm aware of. I'll discuss each one below, adding commentary about problems with lsp4j and notes about re-implementing.
- JSON-RPC protocol handling
- De-serialization of JSON into Java class instances
- Distribution of messages to a threadpool of workers
- Processing of messages via Service class instances
- Serialization of Java class instances into JSON
- Bi-directional client->server and server->client communication
- Sequencing of server->client requests
- Request cancellation
JSON-RPC protocol handling
JSON-RPC itself is quite simple, fitting into a page-long spec.
The LSP flavor of JSON-RPC is more elaborate, specifying details of the message headers, and how the messages are to be separated from each other by newline characters.
lsp4j handles this layer, reading and writing messages in this format. It can read and write from a pair of stdio streams, or through a socket. lsp4clj servers only uses the stdio streams (I think), although #1 requests that we support sockets.
I'm not sure whether lsp4j validates the messages' format. That is, if it were waiting to receive a header and received something else, does it discard the invalid line, throw an error, or something else? What if a header is partially valid (for example, if it specifies an encoding of utf-16, which isn't allowed by the LSP spec)?
commentary
The fact that LSP specifies this layer is a severe limitation. It means that we can't send messages back and forth using standard HTTP servers and clients, because they all expect to speak the HTTP protocol. That is, they expect to have the HTTP/1.1 line, among other things, which LSP doesn't support.
So, I think we'd have to do our own reading/writing of messages as lsp4j does, straight off of sockets or stdio streams. Maybe this isn't too terribly difficult, but it's not something I've ever tried.
De-serialization of JSON
lsp4j de-serializes the JSON. It captures the JSON-RPC method, params and id (if present).
It turns the params into an instance of a Params class, which in turn contains references to other instances. The structure of this instance tree mirrors the structure of the data defined in the LSP spec.
lsp4j then invokes a method on a Service object corresponding to the JSON-RPC method, passing the Params instance.
It keeps track of the JSON-RPC id. See Request cancellation and Response Serialization below for how the id is used.
commentary
This is the one of the worst parts of lsp4j, from a lsp4clj perspective. It is probably useful if you have a lot of tooling that needs static classes. But it gets in our way. The Clojure community has developed much nicer idioms for these responsibilities.
As @ericdallo described, we spend unneeded time casting to and from the Params and Response classes. Mistakes in configuring the reflection cause weird problems in Graal. Worse, because it's easy to cause errors by accidentally "debeaning" too deeply, we only ever debean to the level we need to. This leaves parts of deeply nested structures (like client capabilities) half Clojure and half Java. When one of the servers needs some Java data to be coerced to Clojure, we have to bump lsp4clj.
In my vision, de-serialization would be done by Cheshire (or charred if we want to remove the Jackson dep). camel-snake-kebab would convert all the keys to kebab-case keywords. This would get us Clojure primitives and associative data structures nested all the way down, with hashmaps whose keys are Clojure-friendly. More importantly the LSP spec would be the authority on what keys were available, and we wouldn't have to spend any time adding new keys when the spec was updated (unless we wanted to clojure.spec them to check that clients were implementing the LSP spec correctly) or when one server needed different data. We could remove a bunch of coercion, and not fear that keys or values would be slightly different from the LSP spec in Graal because of reflection.
Distribution of messages
When lsp4j invokes the Service method, it does so in a thread pool. If many messages come in simultaneously, they each get assigned to their own worker thread.
I don't know how many threads it uses.
commentary
This is another low-level responsibility that might be trickier than we think. We could certainly distribute method processing to a thread pool. But what happens if some request starts taking a long time and the thread pool fills up? What if a thread blocks too long? Do we need timeouts? Might be good to study how lsp4j does this.
Processing of messages
Processing is mostly out of lsp4j's hands.
lsp4clj steps in at this point. It provides implementations of the Service interfaces (though clojure-lsp and other servers provide much of the details). lsp4clj coerces the Params objects into Clojure hashmaps, validating that they have the format specified by the LSP spec.
The servers (e.g. clojure-lsp) then do the work of generating responses, returning their responses as hashmaps. lsp4clj then coerces those hashmaps back into Java instances, and passes them back to lsp4j.
commentary
A big reason that lsp4clj exists is simply to implement the Service interfaces. But if we didn't have to do that, there'd be much less for lsp4clj to do.
After de-serializing, we'd have an id, method name, and params map, according to the JSON-RPC spec. We'd need to dispatch based on the method name. Clojure has good tools for this.
(defmulti handle-notification (fn [method _params] (->kebab-case-keyword method)))
(defmethod handle-notification :default [method _] (error-responses/undefined-method method))We could leave it up to the individual servers to implement each method. This would let each server grow to implement the methods it felt were important for its users. For example, at the moment lsp4clj doesn't support textDocument/willSave because clojure-lsp doesn't need it. But with defmethod extension, any server could write the following and have it work.
(defmethod lsp4clj/handle-notification :text-document/will-save ...)`Less ideally, lsp4clj could do what it does now—provide a protocol which the servers must implement and call into this protocol based on the method string. This would force updates to lsp4clj whenever one server needed to implement a new method.
Response Serialization
lsp4j takes this response data and reattaches the request id, as required by JSON-RPC. It then serializes the full response into a JSON message. Then it generates the appropriate headers and writes this message back to the client.
If there was an error, I believe lsp4clj notices that and adjust the response as appropriate, but perhaps that's an lsp4j responsibility.
commentary
It's a good idea to have a helper grab the id and re-attach it after getting a response, instead of trying to thread the id through the method processor. But Clojure is good at that kind of thing—a ring-style handler is a good model. Grab an id out of a request, pass the request to something else which will generate a response, then reattach the request id to the response.
As with de-serialization, serialization of the messages would be easy, using one of the Clojure JSON libraries. We would probably want to do a little conformation, as we do now, to ensure the responses are in the right format.
Writing the response back out to the stream with the right headers would be a little trickier, as with reading from the stream, but nothing too hard.
Bi-directionality
JSON-RPC is typically uni-directional. A client sends requests and notifications to a server, which responds appropriately.
But LSP is bi-directional. Both the client and the server can initiate requests and notifications, and both respond.
lsp4j allows the server to participate in this bi-directionality, making it able to both receive and send messages.
commentary
lsp4j makes a big deal out of modeling bi-directionality. Because of the symmetry they model both the client and the server the same way, as Endpoints. This may be useful if you need to write both clients and servers, but we don't. We're writing servers.
Yes, both directions speak JSON-RPC. We might want some shared helper methods related to JSON-RPC. But we don't need that to be a central feature of our data model.
In particular, when acting as senders of requests, it may be useful to servers to have these requests modeled as CompletableFutures so they can easily be sequenced (see below). But as receivers of requests, servers should be free to use whatever concurrency constructs they like (keeping in mind caveats about message ordering).
They shouldn't have to use CompletableFutures just because that was good for the symmetrical use-case.
Sequencing of requests
When the server sends requests, it sometimes needs to know that the client has processed one request before it sends a second request.
lsp4j models the requests as CompletableFutures. Servers can take advantage of this to sequence their requests. A server can wait on a future (deref it) before initiating a second request. But, it's up to clojure-lsp (or another server) to arrange to funnel all of its requests through a single thread, so that the derefs stack up behind each other.
commentary
Clojure-based HTTP clients often model requests as derefable objects too, so users can decide whether to await a response. lsp4clj might not be able to use an HTTP client, but we can use the same basic idea.
Request cancellation
lsp4j also models incoming requests as CompletableFutures. It keeps a cache of these futures, keyed by id. When the server returns a response, lsp4clj signals this to lsp4j by fulfilling the future. At this point lsp4j takes the future out of its cache.
If at some point while the server is processing the request, the client asks lsp4j to cancel it (by providing a request id), lsp4j looks up the future in the cache, and if it exists, tries to cancel it.
commentary
The lsp4j futures are deceptively named CompletableFutures, but they are not java.util.concurrent.CompletableFuture instances. They are similar, but have special machinery which allows them to be canceled. The method processing code has to participate in this, by periodically checking whether it has been cancelled.
None of our message processors check whether they've been canceled, so we don't take advantage of lsp4j's cancellation.
But it's interesting to think about what we'd need to do to support this feature at the lsp4clj layer. We'd need something similar—a stateful cache of requests indexed by request id. We may have to require that servers participate in the java.util.concurrent.Future interface, so that requests can be cancelled by future-cancel. I don't understand why lsp4j didn't take this same approach, since the java.util.concurrent.Future interface (and therefore java.util.concurrent.CompletableFuture) includes a cancel method.
Maybe there's a reason, for example so that method processors have control over when they are cancelled, and don't get cancelled when they are halfway done with something. We'd have to think about that if we added cancellation support.
Final thoughts
I realize I'm describing a lot of breaking changes. Servers would have to re-write sections of their initialization and method processing logic. They'd have to rip out anything that used the lsp4j classes, or worked around incompletely de-serialized data. But, I think after the re-implementation, the servers would be simpler. And we'd have a more attractive base for writing LSP servers in Clojure.
This may be more of a second project than a "v2" of lsp4clj. In fact, it may be easier to start this work in clojure-lsp, implementing it only there at first. If we did it that way, we'd want to keep in mind the possibility of extracting it, as happened with lsp4clj.
ericdallo
Discussing with @mainej, we realized that lsp4j is not helping that much lsp4clj or clojure-lsp.
The main reason we use it IMO, it's because lsp4j implements the json parsing and stdout/stderr LSP jsonrpc communication layer, making the client just need to implement an interface handling the request methods and notifications easily. But lsp4j has some drawbacks like:
I think we could think about how we can achieve the same lsp4j offers to us using only Clojure