Understanding the Slynk protocol?

[HiPhish]

Hello,

I'm a Neovim user who wants something akin to Sly. Obviously I will have to do the client side myself, but I want to use Slynk as the server component. The first step is understanding how the protocol works so that I can connect the editor to Slynk. I know how to start a standalone Slynk process, but I don't know what format the messages need to be in.

(slynk:create-server :port 0)

This will cause Slynk to listen for TCP messages on some port. Next I would like to feed messages to that port using netcat. So here are my questions:

• What format need the messages be in?
• How are data types serialized? If an expression like (+ 2 3) is sent to the server for evaluation, is it simple encoded as a string?
• What about security? How does Sly authenticate itself towards Slynk?
• Is TCP the only communication channel? No stdio?

I think once I know how to exchange messages I can start chipping away at the actual Neovim plugin. At the moment I am using Vlime, which is nice, but it is very clunky. There are so many nice Neovim improvements that could be made and I would like if both communities could share the same backend tooling. I don't know Emacs Lisp, but if someone can point me to specific functions I might be able to make sense of them.

[joaotavora]

Hello Alejandro,

• What format need the messages be in?
• How are data types serialized? If an expression like (+ 2 3) is sent to the server for evaluation, is it simple encoded as a string?

To answer most of your questions, the quickest way is to see the protocol happening live in front of you. If you can start up Sly (for emacs), pull up the *sly-events* buffer and follow along as you do things. Here's an example bit of a transcript:

(:emacs-channel-send 1
                     (:process "(+ 2 3)"))
(:channel-send 1
               (:write-values
                (("5 (3 bits, #x5, #o5, #b101)" 5 "5"))))
(:channel-send 1
               (:prompt "COMMON-LISP-USER" "CL-USER" 0 6))
(:emacs-channel-send 1
                     (:process "(princ 'hello)"))
(:channel-send 1
               (:write-values
                (("HELLO" 6 "'hello"))))
(:channel-send 1
               (:prompt "COMMON-LISP-USER" "CL-USER" 0 7))

This is plain UTF-8 encoded text over TCP. Whitespace not very relevant.

• What about security? How does Sly authenticate itself towards Slynk?

Sly authenticates with a sly-secret thing. I don't know the details, to be honest, as I never use it. SLY is meant to connect to locally established servers on single user systems. It also allows you to connect through the network to other systems, in which case you have to sort out security using some other program (for example, only allowing connections on a port that's gated through an SSH tunnel). There was some talk about this some years ago in the SLIME mailing list.

• Is TCP the only communication channel? No stdio?

The way M-x sly (and M-x slime this part hasn't changed) works is it starts up an inferior process and briefly talks to it via stdio to convince it to start a Slynk server (over TCP). Then SLY connects to this server. You can also M-x sly-connect if the server has been started already. This approach is very flexible as it allows many clients per server, persistent servers, and also multiple communication channels between the same client and server (i.e. as in the REPL "dedicated stream"). It also allows you to snoop traffic with a tool like Wireshark.

[HiPhish]

That is not quite what I wanted to know. Let's say you wanted to test Slynk without Emacs, you would need to build a mock client that lets you send raw bytes. So what format would the messages need to be at a byte level? Through experimentation and poking in the Slynk code I have found that I can enter the following into netcat

2c    (:emacs-rex (slynk:connection-info) nil t 1)

The message consists of a header which is always 6 bytes and gives the length of the message in hexadecimal notation, followed by the actual message. Slynk then gave me back a giant s-expression which I would have to process client-side. It looks like this format of size followed by body is used by all messages. The next question is what messages exist. What is the deal with the arguments nil, t and 1 above? Some messages require a channel, what is a channel? A worker thread?

It also allows you to connect through the network to other systems, in which case you have to sort out security using some other program (for example, only allowing connections on a port that's gated through an SSH tunnel). There was some talk about this some years ago in the SLIME mailing list.

Makes sense, if my system is already compromised to the point where one process can just write to arbitrary ports I am already done for. And for networking it makes sense to rely on proven tools like SSH.

[joaotavora]

That is not quite what I wanted to know.

I'm sorry, I did my best. If I was in your position, I would still open Emacs and see how it works, because Emacs contains the reference implementation of a Slynk client.

If you want to build a Slynk client for Neovim, it's going to be much harder if not downright impossible if you don't want to look at this reference implementation, because there is really no better description, no real documentation besides the code. I can give you a pointer or two once in a while, of course.

Besides if you're interested in doing this, you must surely be interested in Common Lisp, and if you're interested in Common Lisp, then Emacs Lisp isn't that far off a dialect.

The message consists of a header which is always 6 bytes and gives the length of the message in hexadecimal notation, followed by the actual message.

Oh right, that's indeed 6 UTF-8 chars of indicating the length in hex. That's slightly silly but it was an inheritance from SLIME. And then the corresponding number of UTF-8 characters.

I had forgotten about this detail. How did I look it up? I looked at the Elisp functions sly-net-send and sly-net-encode-length.

What is the deal with the arguments nil, t and 1 above? Some messages require a channel, what is a channel? A worker thread?

Read CONTRIBUTING.md for some hints. But most of all read the Elisp in SLY to understand what it means. Threads only exist Lisp side, by the way, Emacs has no threads. It's close to JavaScript (but not horrible) in that regard.