More Communication Protocols

[djungelorm]

The underlying server communication protocol should be moddable, to allow communication via other means than just google's protocol buffers.

Then we can support, for example:

• http://en.wikipedia.org/wiki/SOAP to make it easy to interact with kRPC from web apps (e.g. JavaScript)
• CCSDS protocol???

[thesamprice]

Send me an email, I could probably help with the javascript / CCSDS protocol stuff. My C# skills are pretty bad, but I know javascript,c, python.

[djungelorm]

This is fairly low on my priority list right now, but when I do get round to it I'll put together a prototype and if you could try out the SOAP interface from JS that'd be great (I don't know JS very well...)

I was thinking we could allow additional DLLs to be placed in kRPC's plugin directory, which would provide additional communication layers. The in-game server interface could then allow you to choose which communication protocol you want the server to use.

[kleiram]

Quick suggestion: REST instead of SOAP, since it's also HTTP but a lot easier to interface with!

[Outurnate]

GRPC might be cheap to generate code for

[sgayda2]

GRPC would solve a lot of your issues, you are already doing what grpc does, proto messages over ipc but it will do a lot more as well, and make it more reliable and stable.

[gitanat]

Seconding GRPC, it seems well maintained and solves the problem of having to write client libraries.

Unfortunately it won't help for getting data in the browser, as GRPC has no defined mechanism for interacting with it (like websockets).

[djungelorm]

I've only had a quick look at GRPC, but here are two issues that came to mind compared to the current server protocol:

• I'm not sure GRPC would allow for kRPC "streams". GRPC has "streams" but they sound a bit different as "the client completes once it has all the server's responses."
• I don't think dynamic clients are possible with GRPC? The kRPC Lua and Python clients currently get a list of available RPCs from the server and create the stubs dynamically. This keeps their implementation nicely separated from the server.

Also, I've started working on making the server protocol used by kRPC extensible. My vision is that the user will be able to start one or more servers, and each server can be of a different 'type'. I see no reason why GRPC couldn't be included as one of these 'server types'. Then GRPC can even run alongside the current server protocol (with its streams and dynamic client support). A REST 'server type' is also planned.

[djungelorm]

In case you're interested, the changes are on this branch: https://github.com/krpc/krpc/tree/feature/server-refactoring

[sgayda2]

I have used GRPC streaming in a project and it worked by having the client connect to the server to get data. The server would never finish sending data so the stream was maintained until the client decided to end the connection.

[gitanat]

I don't think dynamic clients are possible with GRPC? The kRPC Lua and Python clients currently get a list of available RPCs from the server and create the stubs dynamically. This keeps their implementation nicely separated from the server.

Indeed, GRPC does force you to have a well defined interface, and compiling protobufs at runtime is not easy.

Is that added flexibility useful in practice, though? Doesn't the current code have to agree on the protobufs ahead of time anyway?

If we do get gRPC, REST would come for free, if we are willing to run a external program

[djungelorm]

Yes the current code does have to agree on the protobufs, but not the RPCs. The protobufs just define what a request/response message looks like, not what the RPCs are. So you can add/change RPCs on the server and the python/lua clients automatically pick up the changes.

The existing C++, C# and Java clients have generated stubs though, so these could be just be replaced with gRPC.

[Lokaltog]

Let me know if you need help testing a websockets/REST API, I'll definitely port KeRD to kRPC as soon as this is ready.

[djungelorm]

I've now got the server refactored into a state where I can start adding additional communication protocols, and I'm going to start by adding ReST+websockets. Here my thoughts on a potential design. Comments welcome!

RPC Server

Uses ReST over HTTP. Requests are sent to the server as GET requests.

• Requests to call static methods are of the form: /service-name/method?arg1=value&arg2=value
• Requests to call object methods are the form: /service-name/object-<objectid>/method?arg1=value

Some examples:

• Get the active vessel: /space-center/get-active-vessel (Returns a response containing the vessels object id number)
• Get a vessels position: /space-center/vessel-4/get-position (where 4 is the object id of the vessel in question)
• Get the control object for a vessel: /space-center/vessel-4/control
• Enable RCS: /space-center/control-3/set-rcs?value=true using the id returned by the previous request
• Create a maneuver node: /space-center/control-3/add-node?ut=12345&prograde=100

Responses are sent as JSON formatted strings, following the JSON API specification: http://jsonapi.org/

Stream server

For this, I don't think ReST over HTTP is enough, as the server needs to send data to the client continuously. I think websockets would be more suitable? The client could connect to the websocket, send a request (formatted in the same manner as above, but just putting the URI in a text frame?) and the server will then send repeated messages containing the return value of the request. If the client sends additional requests, their results could be added to the response messages sent by the server, so that clients can have multiple streams over the same TCP connection.

Now I'm going to go ponder how to fit gRPC into all of this!

[Lokaltog]

I'm personally not sure if a separate REST API is required for this plugin, or at least if it should be prioritized over WS. After playing around with the Python API, I'd prefer a very similar RPC API for JS using only websockets. The reason for this is that there's much more overhead requesting stuff via HTTP, so polling some resource at e.g. 10hz would be unreasonably resource intensive with a REST-based API, but would work just fine with websockets.

A REST API would in my opinion be better suited for applications where the data doesn't change as frequently, or in this case maybe limited to actions instead of data retrieval - e.g. for RCS like you mentioned PUT /space-center/control/3/rcs with data enabled=true or something like that.

Just a point regarding REST APIs: they're usually implemented using existing HTTP methods instead of having method names in the URI, so data retrieval would use e.g. GET /space-center/active-vessel as opposed to GET /space-center/get-active-vessel, changing data would use the PUT method, creating data the POST method and deleting data the DELETE method.

I was looking around for a simple temporary workaround to start playing around with websockets, and noticed that the websockify project might be able to accomplish what I think would work great as a JS API. Basically it would expose the gRPC/protobuf(?) API directly to JS via websockets, then something like protobuf.js could possibly be used to decode and encode communications with kRPC. The JS API would then expose actions and data through objects similar to e.g. the Python API. By doing it this way you'll also be able to keep the JS API similar to the other RPC APIs which is a plus.

What do you think?

[djungelorm]

You make a good point about performance. Having to create a new tcp connection for every request would be painful!

I guess what we are really after then is a JS client with some code to wrap server-side object ids up as JS objects. This is separate to the actual transport layer underneath, so could be done via gRPC (or the current protocol based on protobuf, but would take more work to implement the client). Then there's no need for websockets or REST over HTTP at all.

I don't know if gRPC can provide this object id wrapping though?

[Lokaltog]

You'd still need websockets support in kRPC (currently no browser can open regular TCP connections), but I'm sure there's some library or wrapper for C# that can expose the kRPC socket as a websocket (Telemachus has a simple WS server implementation, and here's more info related to writing WS servers). If you're able to implement a way to connect to kRPC via websockets I can definitely help writing the JS bindings if needed.

[djungelorm]

Ah I see. Sorry my knowledge of browser based programming is rather lacking. Didn't know you can't just open a TCP connection!

So I guess option 1 is to add a websockets server to kRPC, and use the same protobuf based serialization over it just like the current TCP based protocol. Then implement a JS client (just like the current clients).

Or option 2: we add a gRPC server to kRPC and just use their JS client. But I guess this would not work in a browser if it needs to open a TCP connection? Or can we do gRPC over websockets?

Also, some other thoughts on gRPC: I've been having a read of the gRPC documentation, and it doesn't look like it can provide the same "remote objects" that the current kRPC clients do. It follows the design philosophy of "services not objects and messages not references". While adding a gRPC server would give us a JS and many other clients for free (which is a good reason to add support for it) those clients would feel very different to the current clients. You would need to pass object ids around as plain old integers in the gRPC messages, rather than being able to make method calls on objects.

So option 2b is to add a gRPC server to kRPC, then build a JS client on top of the gRPC JS client that provides these "remote objects". Writing such a client would probably be less implementation with than writing a JS client for the current protocol, plus we get all the gRPC clients for free :)

I think I prefer option 2b if gRPC can run in a browser - less implementaion work and lots of free clients :)

[djungelorm]

To answer my own question, gRPC doesn't work in the browser. I guess we need option 1 for KeRD to work with kRPC.

[djungelorm]

There is also an official JS protobuf library we could use (instead of protobuf.js) for the encoding and decoding and it works in the browser: https://github.com/google/protobuf/tree/master/js

[Lokaltog]

Yep, option 1 sounds like the only real option (at least if you want to support browser based kRPC clients), even if it means more work implementing the JS bindings. Google's protobuf library sounds good! I can start working on JS bindings as soon as a websocket server is added to kRPC, do you know if it will require a lot of work implementing a WS server?

[djungelorm]

No, it should be fairly straight forward. I imagine I'll be able to get a prototype done over the weekend for you to try out.

[Lokaltog]

Awesome, I can't wait to check it out.

[gitanat]

This is looking great!

I just want to point out that, although browser-based clients can't possibly support gRPC (because of the lack of TCP support), there are some projects around that act as bridges (https://github.com/gengo/grpc-gateway), so we could get REST for free, if there's a way of running that from inside Kerbal (can you launch processes within unity?).

Technically, we could also get gRPC support from websockets using this project (https://github.com/kanaka/websockify), but we'd have to re-implement gRPC in javascript...

Having to create a new tcp connection for every request would be painful!

With HTTP (1.1), you can reuse the same connection. The overhead in HTTP really comes from the fact that it's request/response based - streams of data are not well supported, so the client has to constantly issue new requests.

[djungelorm]

@Lokaltog I've got a websockets server ready for you to play with :D

You can download it from here: https://krpc.s3.amazonaws.com/deploy/feature/websockets/375.1/krpc-0.3.0-30-g65ca751.zip

To use it, just install the plugin as usual, fire up KSP, choose a port number and hit start server. Then send KRPC.Request messages to it and receive KRPC.Response protobuf mesages from it using websockets frames with opcode = binary. Note that you don't need to send the size of the protobuf message like you do with the other protocol. The size is include in the ws frame header, so you just need to send the protobuf message data.

Here's an example script (in python) that I used to test it: https://gist.github.com/djungelorm/6372d31acc515fc63f18c889a8f5f631#file-ws-client-py

I haven't done much testing so expect bugs. I'll be doing some more testing tomorrow, so let me know if you have any issues!

If your interested in the server code it's here: 856f819b7fed55064e2094ee05d30ed0d48a6665

[Lokaltog]

Sweet! I just downloaded and installed it, and your test script works fine. I'll start working on some proof-of-concept JS code right away.

[Lokaltog]

I've reimplemented the basic example using webpack and proto-loader which uses protobuf.js to compile .proto files to JSON and handle the message encoding and decoding. Here's the example script in JS:

var ProtoBuf = require('protobufjs')
var proto = ProtoBuf.loadJson(require('krpc.proto')).build()
var sock = new WebSocket('ws://localhost:50000')
sock.binaryType = 'arraybuffer'

sock.onopen = (ev) => {
    let req = new proto.krpc.schema.Request('KRPC', 'GetStatus')
    sock.send(req.toArrayBuffer())
}

sock.onmessage = (ev) => {
    let resp = proto.krpc.schema.Response.decode(ev.data)
    let status = proto.krpc.schema.Status.decode(resp.return_value)
    console.log(status)
}

This prints the same status object as the Python script. I'll start reading up on how the kRPC Python bindings are actually implemented, then I'll start working on the JS bindings. I'll use protobuf.js for now, as Google's JS implementation is currently in a poorly documented beta.

[Lokaltog]

I'm currently attempting to set the client name and receive the client identifier like this, but I'm not receiving any response from kRPC when sending the hello message and client ID. The connection is also left open so I think some error may be occuring in kRPC. Is this feature not implemented yet?

[Lokaltog]

I also can't request the services with KRPC.GetServices. Here's the stack trace from the debug log:

[EXC 14:29:15.759] IndexOutOfRangeException: Array index is out of range.
    KRPC.Server.WebSockets.Header.ToBytes ()
    KRPC.Server.WebSockets.RPCStream.Write (KRPC.Service.Messages.Response value)
    KRPC.KRPCCore.ExecuteContinuation (KRPC.Service.RequestContinuation continuation)
    KRPC.KRPCCore.RPCServerUpdate ()
    KRPC.KRPCCore.Update ()
    KRPC.KRPCAddon.FixedUpdate ()

I don't receive any errors when I attempt to send the hello message and the client name.

[djungelorm]

I'm working on a fix the IndexOutOfRangeException.

Regarding setting the client name, that's not to do it with websockets - sorry I should have clarified. You can set the client name by setting the "Origin" header in the HTTP request used to establish the connection. Once the connection is established, use the protocol as described here: http://krpc.github.io/krpc/communication-protocol.html#invoking-remote-procedures i.e. all the stuff that happens after the bit of python code you quoted.

Also, the stream server isn't working yet, so just ignore that.

[Lokaltog]

Ah, I see. I noticed the Origin header code, but unfortunately it doesn't appear you can set custom headers when establishing a websocket connection in a browser. Maybe it would be possible to accept a client name as a GET argument too? I.e. set the client name like this: new WebSocket('ws://localhost:50000/?name=MyClient')

[djungelorm]

Ah OK, I just assumed you could! That looks link a reasonably clean way to do it. And if it isn't set, we could default to using the browser provided Origin header. And if that's not available either just use "unknown".

[djungelorm]

Here's an updated version with the IndexOutOfRangeException fixed: https://krpc.s3.amazonaws.com/deploy/feature/websockets/376.1/krpc-0.3.0-31-g71aee06.zip

The websocket frame header decoding/encoding had a few bugs when dealing with large frames.

[Lokaltog]

Perfect, it appears to work now!

[Lokaltog]

Just a quick question: are responses from kRPC guaranteed to be returned in the same order they were submitted? I.e. if the client rapidly submits requests 1, 2 and 3, will responses always be returned in order 1, 2, 3 (FIFO) or could they possibly be returned in a different order?

[djungelorm]

In order. See paragraph 3 of this section of the docs: http://krpc.github.io/krpc/communication-protocol.html#invoking-remote-procedures

[djungelorm]

I've been working on the websockets implementation and it should be relatively stable now. Here's the latest version (based on kRPC 0.3.2, which supports KSP 1.1.2): https://krpc.s3.amazonaws.com/deploy/feature/websockets/409.1/krpc-0.3.2-73-gc086569.zip

The stream server is also accessible over a websockets connection, although I've not documented it yet - give me a shout if you want details on how to connect to it. I've also added much more thorough unit tests, and it also now passes the tests in the Autobahn TestSuite.

All that remains is to update the UI so you can choose protobuf over plain TCP or websockets. Once that's done I'll try implementing gRPC support.

[gitanat]

Thanks a lot for your efforts djungelorm, it's great to have websocket support! Hopefully nice UIs will follow, now that people can develop them :)

[unstoppablecarl]

@djungelorm I am very interested in setting up the websockets. I would be happy to help with testing just let me know how to set it up.

[Lokaltog]

@djungelorm Just wanted to let you know that I'm currently very busy with work and unfortunately haven't had any time to work at the JS bindings lately. I do have some basic proof-of-concept client code if anyone wants to check it out or continue working on it until I have more time to work on it myself.

[unstoppablecarl]

I would be happy to give it a try. Just tell me how to set it up.

[djungelorm]

Thanks for the update @Lokaltog The latest version with the websockets implementation is still the link the last comment I made. I'll bring the dev branch up to date with v0.3.4 tonight so you can try with the latest changes.

[djungelorm]

Here's an updated websockets server build, based off of v0.3.4: https://krpc.s3.amazonaws.com/deploy/feature/websockets/484.1/krpc-0.3.4-26-g0fc1eec.zip

[unstoppablecarl]

I got it up and connected! using the following code adapted @Lokaltog 's example.

'use strict';

var WebSocket = require('ws');
var ProtoBuf = require('protobufjs')

var builder = ProtoBuf.loadProtoFile('./krpc.proto'),
    proto = builder.build();

var sock = new WebSocket('ws://127.0.0.1:50000')
sock.binaryType = 'arraybuffer'

sock.onopen = (ev) => {
    let req = new proto.krpc.schema.Request('KRPC', 'GetStatus')
    sock.send(req.toArrayBuffer())
}

sock.onmessage = (ev) => {
    let resp = proto.krpc.schema.Response.decode(ev.data)
    let status = proto.krpc.schema.Status.decode(resp.return_value)
    console.log(status)
}

[eXigentCoder]

I tried adapting the above code to work with the proposed 0.4.0 release, specifically the build here however I get an error ("Error on socket { Error: Parse Error ..." as soon as the websocket attempts to connect.

I then looked in the krpc repo and realized the websockets release is a separate feature branch which explains why my code doesn't work. I then downloaded the latest websockets release here however trying to hit GetStatus with that causes the following error

Error: Illegal wire type for unknown field 6 in Message .krpc.schema.Response#decode: 6

Using the krpc.proto + GameData files in the release linked above (websockets server build, based off of v0.3.4) the code runs fine, has something changed In how the messages are sent/received or is my code just buggy?

Feel free to check out my GitHub code repo, was hoping to build it out into an npm package that people can use as a node client. Any help would be greatly appreciated, and thanks for all the hard work!

[djungelorm]

I made some changes on the v0.4.0 branch that hadn't been merged into the feature/websockets branch. Specifically, I implemented the features described in sections 2 and 6 of the pull request. The changes in section 6 shouldn't affect the websockets server, but the ones in section 2 do.

I've merged these changes into the websockets branch so it's now up to date, and a build is available here: http://krpc.s3-website-us-east-1.amazonaws.com/deploy/feature/websockets/684.1/

The format of request and response messages have changed (to allow you to send multiple RPCs in a single request), so you'll need to modify the above code for it to work. I've not tested it, but you'll probably want something like this:

sock.onopen = (ev) => {
    let call = new proto.krpc.schema.ProcedureCall('KRPC', 'GetStatus')
    let req = new proto.krpc.schema.Request([call])
    sock.send(req.toArrayBuffer())
}

sock.onmessage = (ev) => {
    let resp = proto.krpc.schema.Response.decode(ev.data)
    let status = proto.krpc.schema.Status.decode(resp.results[0].value)
    console.log(status)
}

Also, FYI, I plan to release the websockets server in v0.4.0. Hopefully very soon!

[djungelorm]

Also, quick update on gRPC. A few weeks ago I had a go at getting the C# version of gRPC to work with Unity's version of Mono - which is a right pain, as it uses async which isn't supported. However, I have managed to get a modified DLL that appears to work (although I only did some basic testing).

[eXigentCoder]

Awesome, thanks so much. Got it working with the updated release, your non tested code above worked great Will play around with it a bit more when I get back from brunch and see if I can't get a basic working version out for people to experiment with.

[eXigentCoder]

I have managed to get a very basic node client library up and running if anyone feels like playing around, there are still a bunch of things to do like:

• Adding more tests
• Documentation on how to use the api
• Examples of how to use the api
• Decoders for return types of [double, float, sInt32, sInt64, uInt32, uInt64, bool, bytes, enum] see ./decoders.js file
• Check if adding and removing streams actually works
• Code clean up

I started off building code manually (see the apis/krpc.js file) but then realised I could generate it all from the getServices call, so switched to that (Code in the ./utilities/generate-service.js file).

So far I have tested:

• krpc.getStatus
• krpc.getServices
• krpc.getClients
• krpc.getCurrentGameScene
• spaceCenter.clearTarget

[gitanat]

Great stuff, thanks, ExigentCoder! @djungelorm Would it be possible to have this javascript api be part of krpc - much like the existing python and C++ clients?

[djungelorm]

Yeah it's possible. Although for now I think it would be easier for @eXigentCoder to work on it in its own repo?

Also, having it in the main kRPC release wouldn't really gain much from a users perspective. They would just be doing npm install krpc-node anyway, which is unaffected by where the source code is actually held. However, it would be good to have it documented on http://krpc.github.io/krpc