Comms in binary

[AlistaiReid]

More than 50% of the profiled run-time of stateline-server is string packing and unpacking.

This will require changes to:

• communication between sampler thread and delagator thread
• writing chains (put in another thread? or use binary format?)
• comms between delegator and client process

[darrnshn]

This is pure speculation since I haven't profiled the code:

• In terms of big speed-ups, putting chain writing to CSV in a separate thread should be good since it's mainly IO bound. Something like a lock-free producer-consumer queue might be useful?
• (Probably most relevant to string packing/unpacking) One small thing I noticed was that in the messages library there was a lot of unnecessary dynamic allocation (primary because the messaging system was designed to be really flexible back in the day). Things like addresses are all std::vectors that only contain a couple of elements. Instead of copying strings, we can std::move them instead (especially for message body data which is usually quite big due to specs). Lots of copies and lots of std::vectors that we probably can remove!
• If the message body is way too big, then we can perhaps perform some compression before sending?
• Delegator::bestWorker is O(N). I reckon we can do some fancy stuff to reduce it to O(log N). Probably not important if it didn't show up in the profiler.
• Delegator is a socketrouter with bound message handlers at runtime. So every time there's an incoming message, we lookup a index in an array and dynamically dispatch the callback stored in the array (which is probably stored on the heap). This is sorta like a DIY vtable. If the message types that we get chages often, this might not be very cache friendly. Since we know at compile time exactly what methods should be called on the delegator when a message comes in, this runtime flexibility seems a bit overkill.
• Heartbeat messages don't have to be sent all the time. All messages can serve as heartbeats. A heartbeat is essentially a filler message that makes sure we don't timeout. If jobs are short (which is the case we're optimising for right?), then heartbeats probably don't even have to be sent. I think this is what RabbitMQ does, but at the cost of more complexity.
• Should probably double check this, but the worker time prediction algorithm predicts the running time of a job based on the worker. If we're running in the cloud, then workers are roughly equivalent. A more sensible way would be to predict the running time based on the job type, which is a better indicator of how long the job is going to take.

[lmccalman]

all good points =) I believe the worker time prediction is done on job type and worker at the moment-- but could well be wrong about that. This unnecessary dynamic allocation is a big thing we should fix soon.

[darrnshn]

We should do #32 first so we can measure the performance improvements.

[darrnshn]

Also, an old idea was to use something like protobuf/flatbuffers to remove most of the serialisation code and make it easier to communicate with other languages. However, most of these serialisation libaries have fancy features like backwards-compatibility which we don't really need ATM. So we may incur an unwanted performance overhead.

EDIT: We could go for a compromise, where the client facing communications are done in protobuf but the internal server communications is done with raw text.

[darrnshn]

Using binary protocol for the entire system. Using both is slow because we have to convert between protobuf and binary.