ymeine
commented
10 years ago In fact I would be in favor to implement as many as relevant solutions, from lower level to higher level.
Lower level generally means faster but with higher amount of work to put in place a tailored solution.
Higher level on the other hand generally means easier to implement, more reliable, more flexible and sometimes with side-effects features that can be handy, but also slower.
Anyway, there are still distinct domains to improve:
- the transport: this is from the IPC domain, there are only a few solutions that are usually cross-platform, not too complex and with enough features. Sockets for network, pipes (keeping in mind that sockets on localhost might be equivalent to a pipe) for simple stream of data.
- the protocol: this means how messages are encapsulated to be sure the other part receives it completely, without errors. For now we use HTTP, we could use TCP, or implement a completely custom one if we use pipe transport.
- the format of the messages: this is how messages are encoded and decoded, to actually mean something
- the API: in fact most messages mean in fact calls to services, and it was handy to use RPC, that is simply messages that identify the method to call and gives arguments to it.
Actually some points might be kind of merged together depending on the chosen solution. The more we want to be faster, the more we will be specific.
For instance for now using our stack, we can transport any other data through HTTP, and if ever we still send JSON encoded data, we could imagine giving objects with completely different properties.
While on the other end we could imagine sending special command codes through a pipe, with very specific arguments then, with order restrictions, maybe length restrictions too. A bit like processor instructions if you like. Whereas our current solution is more like a web stack (or imagine Python vs Assembler)
Anyway, we should provide the alternative solutions to the clients. They could choose the one that suits their needs by giving a proper option on the command line.
TL;DR: Reduce the overhead introduced by HTTP, JSON serialization and also RPC.
Here is a reminder schema about how the backend exposes the core library features to clients - to enable independent systems generic communication:
The JSON here is used to encode RPC requests, the HTTP just transports them. Of course, all components can be run through any system (JVM, Node.js, Python, ...), but the
Serverand theCoremust run into the same (here Node.js).Overheads identification
Keep in mind before all that any suggestion here should not go against this requirement: limit the amount of double maintenance, on client and backend. That means that if we build something more optimized but not standard, the development/maintenance overhead should be small, otherwise drop it.
HTTP
Too verbose to transport only RPC requests, otherwise only small HTTP requests like identification of the server, shutdown, ...
Removing the unnecessary HTTP headers would be great. Having a simple request format like: a command ID, followed by the data to send, if any.
Connection is not maintained, so there is overhead to set it up and to destroy it each time.
TCP?
JSON
A bit verbose too (still better that XML :wink: ): at least the required double quotes around object keys, not the case in JS.
At least you can minimize it, and enable short property names (one letter is fine in this case).
However I think JSON is already small and we need to keep something with available parsers for lots of systems.
Alternatives
There are also other standard solutions like CORBA (but I'm not sure there is an available mapping for JavaScript in this case).
I found recently (05 Jul 2013) an article talking about Thrift. The description at least corresponds exactly to what we want to do here: provide services to clients whatever the system they use, and automatically deal with remote procedure calls and so on.