RPC Interface

[rychipman]

This has been on my mind a lot lately, and I think it is especially relevant as vis continues to mature -- I think vis needs an RPC interface.

As I mentioned in the discussion on PR #33, I think vis is really valuable and great because of how lean it is. Marc did some good work a while back on keeping the UI decoupled from the text manipulation logic, but, as features get requested and added, it will get more and more difficult to improve the vis experience while keeping it clean and uncluttered.

Aside from completely separating UIs from the vis core, the RPC interface would allow "plugins" to be written for vis in any language. This would allow lots of auxiliary functionality (indentation, formatting, autocompletion, etc.) to be added to vis without cluttering up the core.

So I wanted to get the discussion started on the best way to approach this problem. For reference, I have been looking at the Neovim API and the Neovim Plugin Architecture. While these APIs are much bigger than we would likely want for vis, they are nonetheless a good reference point to start from.

[rychipman]

Some options for serialization:

• BSON
• Protobuf
• MessagePack

[greduan]

I'd vote for Msgpack, followed by BSON. Simply because they are the simplest, they're just smaller versions of JSON, which is very widespread and found on almost every language at this point. At least that's the case for Msgpack and maybe for BSON (not sure).

[rychipman]

Between the two, I think Msgpack is preferable. It's designed from the ground up for RPCs, whereas BSON is really designed for storage & traversal (the serialization format of choice for MongoDB). This is the c lib advertised on the msgpack website.

The downside is that msgpack-rpc doesn't seem to have a c implementation.

[greduan]

Well the C++ version can be used, or it can be implemented from zero. Dunno which would be more troublesome.

[rychipman]

Between the two, I think doing a C implementation from scratch would be better than dealing with C++ in the code base (I know Marc has expressed a similar sentiment in the past).

But when protobuf has a working c rpc implementation (which, for what it's worth, is being actively developed as well), is Msgpack still that much better?

[rychipman]

Alternatively, there are things like gRPC (still in alpha), or just using ZeroMQ. However, I think a RPC library is a must because it makes client creation for different languages so much easier.

[greduan]

The thing with Protobuf, from my limited knowledge, is that it is not plain JSON. It uses schemas (it seems) and stuff like that.

[rychipman]

Defining a schema is not a bad thing though, as it allows code generation in multiple languages. Even though msgpack doesn't require schemas, we would likely use them anyways.

[erf]

have you checked out https://capnproto.org/

[martanne]

Yes I agree, a RPC interface would be useful. The library we are going to use should ideally be

1. C only (not C++) and have minimal dependencies
2. support static linking
3. non-copyleft licensed i.e. ISC/BSD/MIT/Apache style

Last time I investigated this 1. was already a slight problem. If I remember correctly most libraries written in C seem to provide the serialization part, but do not cover the actual RPC marshaling.

I think introducing a proper (libuv based?) mainloop, which could be used for the :! filter implementation, would make sense to handle the RPC clients.

Then there is the question of what kind of API to expose. Only the low level text_* interface? That is should the display logic resides within the server or on the client side etc.

In short: yes an RPC interface would be useful, but unfortunately I will most likely not have time to actually work on it in the near future. Instead I will focus on getting the current single process design to a releasable state. That shouldn't discourage others to work on it, though.

[pentlander]

Another option for the RPC library is nanomsg. It's written in pure C and has an MIT license.

[erf]

There is a pure c implementation of capnproto.

[martanne]

There is a pure c implementation of capnproto

I didn't really take a closer look but: number of contributors: 2, last commit: 2 years ago, does not seem that promising.

Besides the points listed above other desired properties include:

• actively maintained, widespread community support
• bindings to popular scripting languages available

[erf]

The main c++ lib seems active, there are also support for other languages.

[anatol]

Another option for serialization/RPC is thrift https://thrift.apache.org/ It has bindings support for many languages https://github.com/apache/thrift/tree/master/lib

[critiqjo]

I don't think there should be any built-in IPC/RPC support. Since there's in-process extension through lua, let the plugins do IPC on their own. You should never expose built-in API to a separate process like Neovim did. Anybody can connect to the TCP port, and access all files accessible through Neovim, remotely. Leave the pain of security to plugins.

And for embed-ability, instead of RPC, I would suggest providing the engine as a library. If I am creating an IDE, I'll just import the Vis module and run an event loop natively, which will notify me of state changes together with (abstract) details of how UI looks like (e.g. cursor position(s), an array of lines, starting and ending positions of various highlight groups, etc.).

[rychipman]

Yeah, this issue was initially opened before the lua plugin API, so I think this is now no longer needed!

[martanne]

I believe the Lua API and a language agnostic RPC interface serve different purposes. The latter would facilitate a client/server architecture.

[ii8]

CBOR should be considered as a serialization format if and when a decision is made: http://cbor.io/ Similar to msgpack but more consistent allowing for smaller implementations.

[mcepl]

The same question: what is this issue good for? Does anybody even plan to work on it?