Control protocol

[BenediktBurger]

Ideas how a director (e.g. a measurement script) can control actors (e.g. controlling instrument drivers).

[BenediktBurger]

One idea is the majordomo protocol of the zmq manual.

Idea of that protocol

• Workers offer their services to a majordomo (central part)
• clients request some service to be done.
• several workers can offer the same service. Which worker will do it, does not matter
• Doing the same service in a worker will return the same result (idempotent)
• Workers and clients and the majordomo exchange heartbeats to show that they are still alive.

Pros

• hearbeating
• routing to services
• an already defined protocol (but only the transfer part, not the content)

Cons (for instruments)

• You want to talk to a specific multimeter / oscilloscope, not any
• There are no working and maintained python implementations I could find.

[BenediktBurger]

Based on the majordomo and some applications, I thought about a coordinator, which delivers messages to a recipient based on a string, similar to an email:

1. All participants contact regularly the coordinator, telling their own (str) name, such that the coordinator has a dictionary of names and corresponding zmq addresses.
2. A participant may send a message to any other participant, by sending it to the coordinator and stating the name of the recipient.
3. The coordinator sends the message to the destination without changing anything (neither header nor content)

Parts of the header:

1. Recipient name (that the router may deliver it, obligatory)
2. Sender address (that the recipient may respond, optional)
3. Subject (that you may sort an answer to the corresponding request, optional)

[BenediktBurger]

Another tool, yaq, uses Apache Avro RPC (remote program call) as serialization scheme, maybe we can orient us there for the content serialization.

[bilderbuchi]

To preserve the generality of the protocol, we should imo avoid sending pickled python objects around, but rather appropriately serialised payloads (e.g. json).

[LongnoseRob]

To preserve the generality of the protocol, we should imo avoid sending pickled python objects around, but rather appropriately serialised payloads (e.g. json).

this points to another question: Do we run our own error-detection or do we trust the "communciation" channel will take care of this?

[BenediktBurger]

The problem can be split into two:

• how to get some information to the destination (routing part)
• how to interpret the transmitted information (do this, do that)

[bklebel]

this points to another question: Do we run our own error-detection or do we trust the "communciation" channel will take care of this?

I think we will have to do at least a limited degree of error-detection ourselves. This might include uuids which tie answers to the questions which evoked them (or other metadata information which does the same job, maybe that is what @bmoneke calls "Subject" above), something which zmq does not do by itself, especially not if we start implementing asynchronous coordinators. I am not sure whether we would have to detect errors in the transmission of individual messages, like checking against a hash value after receipt of the message body, but I think this is properly handled by zmq (and I think I would indeed only want to consider such a "high-level" communication framework where we do not have to think about sth like this) I am not perfectly sure how reliable (in terms of dropped messages etc.) zmq really is, but so far in my implementation it works, as long as queried nodes are actually alive, the asking node will get the proper answer-message eventually. If we go for a heartbeat anyways, we immediately know about dead/unresponsive/not-yet-started nodes, and do not have to "catch that exception" after not having received an answer to a question until a timeout, but can return immediately that the requested node is offline. How a coordinator should react to that is then another question, and might depend on the intended use and application (possibly out of our scope)

[BenediktBurger]

Subjects / IDs

In my (testing) implementation, I have two "subjects" in the header: one, which the sender of the message may choose freely (for example a timestamp, message ID etc). The second subject is that of the sender of the original message, such that the original sender may attribute the response to the request.

An example of A requesting something from B: (format "name of recipient";"subject the recipient sent";"name of sender";"subject of sender")

A sends a request to B with the following header: B;;A;request 1 (it is a request, therefore no subject of B)

B responds and uses a timestamp as its sender subject: A;request 1;B;timestamp

Now A knows, that it is a response of its request 1 message. If a reply should be necessary, A could write this header: B;timestamp;A;reply_timestamp with its own reply_timestamp as a new subject. B in turn would know, that the reply belongs to the message it sent. And so it could go back and forth.

Reasons for the 2 header system:

• each message can have a unique ID ("subject of sender")
• you can attribute a response to the original request

Symmetry / Asymmetry

The majordomo protocol is asymmetric: it differs for workers and for clients.

The above mentioned protocol is symmetric: any node uses the same protocol to reach any other node. That keeps the protocol simple and versatile.

[BenediktBurger]

How a coordinator should react to that is then another question, and might depend on the intended use and application (possibly out of our scope)

I agree, that every user should handle that.

[bklebel]

This way of the two subjects seems quite nice to me - I am not sure about what use-cases would need a longer ping-pong back-and-forth communication in a consecutive string of messages though. In what I have in mind, when a director needs more information or wants to give more commands, I would rather send those independently and thus asynchronously, than waiting for the last reply of some actor.

The above mentioned protocol is symmetric: any node uses the same protocol to reach any other node. That keeps the protocol simple and versatile.

That idea is quite interesting, however, I guess your coordinator would look a bit different than what I have seen so far in zmq proxies, wouldn't it? I guess with this symmetric n-to-n communication it would be easier to build bigger and bigger systems in a LEGO fashion, than with an asymmetric system. And if we plug an automatic discovery of additional coordinators on the network to it, I imagine this could become very easy to use (although not necessarily easy to implement....I would not really know where to start in doing an all-python OS independent automatic discovery of additional coordinators/nodes). Possibly the automatic discovery is out of our scope for the start. And the more I think about it, the more it sounds like email, I am not sure we want to re-invent that one ;)

The problem can be split into two:

• how to get some information to the destination (routing part)
• how to interpret the transmitted information (do this, do that)

I agree. Also, I guess our main focus would be, in the beginning of our roadmap, the routing part - if we can't get the information where we need it in the first place, there is little use to try to decide about what to do with it.

[BenediktBurger]

That idea is quite interesting, however, I guess your coordinator would look a bit different than what I have seen so far in zmq proxies, wouldn't it?

Yes, but it is (in the core workings) quite simple.

And if we plug an automatic discovery of additional coordinators on the network to it, I imagine this could become very easy to use (although not necessarily easy to implement....I would not really know where to start in doing an all-python OS independent automatic discovery of additional coordinators/nodes).

Here again, I did mix an idea from majordomo: If a node writes a message via a coordinator, that coordinator stores the node's name (from the header) and its address. That information is updated with heartbeats (empty messages directed to the coordinator). That way, we don't have to discover the nodes, but the nodes tell themselves.

And the more I think about it, the more it sounds like email, I am not sure we want to re-invent that one ;)

The advantage is, that it is easier to setup (ok, I never tried to send email in a privately made network) than email.

[bklebel]

That way, we don't have to discover the nodes, but the nodes tell themselves.

Well, I was thinking about a bigger network, with potentially multiple coordinators, but I guess this is not necessary as long as we do not want to jump from one network/domain/area to another, and need to connect them, in a fashion which would be similar to email. Another instance in which we might want to have more than one coordinator on the same network would be if we had such a lot of traffic that one coordinator cannot keep up, but I guess our current scope does not extend to such network sizes.

That information is updated with heartbeats (empty messages directed to the coordinator).

Hmm, maybe we would rather like to give that a special subject too? keeping it simple, just "heartbeat", so that in case we consider such a message again we have no collisions?

[BenediktBurger]

Hmm, maybe we would rather like to give that a special subject too? keeping it simple, just "heartbeat", so that in case we consider such a message again we have no collisions?

Right now, the coordinator updates the address list (and also a timestamp of that last contact) with every message it receives, independent of the recipient. If a node wants to say explicitly, that it is still in the network, it has to send a message to anyone. The most simple recipient is the coordinator itself and without a content, as that is not necessary. This "most simple message" is used as a heartbeat 😁.

Well, I was thinking about a bigger network, with potentially multiple coordinators,

That should be no problem: Only the coordinators have to be upgraded to exchange address information. That way, we can reduce the multi coordinator question to one of content and not the transfer protocol. The Coordinator looks in its list of addresses where to send a message and does not care, whether that recipient is the final recipient or another coordinator, which in turn can send it further.

Name service

Another idea: A node may ask the coordinator for all names available.

[bklebel]

If a node wants to say explicitly, that it is still in the network, it has to send a message to anyone. The most simple recipient is the coordinator itself and without a content, as that is not necessary. This "most simple message" is used as a heartbeat 😁.

Ok, that is a nice solution!

Only the coordinators have to be upgraded to exchange address information.

Okay, true, then the coordinators need to start exchanging which nodes are connected to them. So if one coordinator (C1) has the sum X nodes, and the second one (C2) has the sum Y of nodes, and node x wants to talk to node y, then C2 needs to have told C1 his list of nodes (Y), and C1 can advertise in its name service that these are available through him too, node x sends the message for y to C1, C1 knows that y is from C2, and forwards it. If we need multiple hops, and every coordinator advertises (possibly a publishing socket) which nodes are available to talk to, and all coordinators listen to all other coordinators they are directly connected to, save those other nodes in their "proxy-list" and advertise them in turn, we can have a continuous forwarding. It's then a question to which coordinator C1 should forward a message if all C_i say "I could potentially forward it" (I think we are already a bit outside of the original scope here). I guess one more problem which we might encounter in that instance is the update rate for the name service in the whole network, if two nodes are "far away" from each other in this network, the heartbeat might have some delay...but again, maybe that is way out of scope.

I think the name service could be pretty easily done with a publisher, to which nodes can subscribe, which publishes the list of nodes regularly, and would surely be beneficial when directors want to discover actors and somebody wants to implement some automatic GUI generation for all the actors on the network - the director will know how they are named themselves and cancel themselves from the list of nodes which they received from the coordinator.

[BenediktBurger]

Here again, our scope / goal helps us to not over engineer the protocol.

With "name service" I just wanted to mark, that the coordinator keeps a record of active nodes.

We do not need to publish that information, it is sufficient that a node requests it from the coordinator. I think you rarely need that information (in a small lab, you know how you called that device over there). It's more a check: who's online?

[bilderbuchi]

And the more I think about it, the more it sounds like email, I am not sure we want to re-invent that one ;)

I'm actually more reminded of TCPI/IP (and DNS -- name services!). Please try not to reinvent the Internet! :tongue:

Some high-level loose remarks:

• I think timestamps should be on every message -- that probably makes it much easier to analyze logs (same goes for sender and recipient, I guess)
• Are the "subjects" you talk about the same as zmq "topics"?
• maybe we should try getting down the details with one coordinator, first, and iterate to multiple coordinators when the wrinkles have been ironed out? Or are you concerned that we could make some technical decision that makes multiple coordinators tedious further down the road? Maybe just keep the option in mind, but don't fully design it?

[BenediktBurger]

There exists no "Zmq topic" as such. The pub/sub sockets allow to filter messages if the begin of a message corresponds to some string. Therefore, you can use the first frame as a topic indicator, but you do not have to. You could subscribe to a publisher and receive everything.

The idea about topics is, that not every recipient wants all messages of a publisher.

In a one on one communication, you want everything, therefore we do not filter, but use the subject as identifier (timestamp or so) of a conversation.

I'm actually more reminded of TCPI/IP (and DNS -- name services!).

How difficult is it, to install a dns server for processes on the same computer? I'm not sure, that it is simpler. Yes, dns would be an option as Zmq uses tcp as underlying software.

maybe we should try getting down the details with one coordinator, first, and iterate to multiple coordinators when the wrinkles have been ironed out

I agree, but we should keep it in mind.

Funny fact: today I wanted to connect two of my (testing) coordinators on different computers and I wrote an additional piece of software, which can connect two coordinators: you just mix and match the individual building blocks as you need them. That connection piece cannot chain coordinators yet, but I have already some ideas.

[bklebel]

In all those thoughts, for me first and foremost is the ease of use for somebody adopting either this protocol or our planned reference implementation. To my knowledge, people reinvent the wheel as soon as it seems that reinventing it is more easy and quick (and probably dirty but this is often not cared for too much, I am afraid, especially as long as supervisors in experimental academia are mostly people who do not develop code themselves) than understanding this big thing somebody else built, in order to use it. To that end, I picked up the idea of the name service (we discussed about service discovery at some earlier point too, if I remember correctly) and explored it a bit, because, I think, it can make connecting new components to a growing system so much easier. However, since "ease of use" is a core issue for me not only in regard to a user who wants to set up a system as quickly and easily as possible, but also in regard to a user who might face a problem while trying, so that they are tempted to understand enough of the whole system to debug it, I think it is important to keep our system simple too.

In my implementation, when I first wanted to connect an actor on a different machine (RPi) to my coordinators on my main machine, it took me quite a while to get all the zmq/IP addresses/sockets/ports right, which is something that should ideally work out of the box.

Also, I am concerned with reliability. At our institute, some measurement runs last for a few weeks (working with microkelvin temperatures can be quite slow, while high-speed data acquisition is still needed for high-precision measurements), and I would like to avoid having a single point of catastrophic failure in the forced reduction to a single coordinator.

And last but not least, I am concerned with growing systems. When I started with my setup, I thought I would basically need to connect a temperature controller and a multimeter, now we have over 15 devices connected, some bought, some self-built, and we are planning on adding additional ones. If somebody chooses our protocol/system, I do not think that we want them to fall flat on their noses once they put their toe over the line of our scope. Not everything needs to be implemented "at the start", we know that this is a process, but we might want to explore those ideas (at least on the side).

So, TL;DR, in total I agree, this might not be the no. 1 priority for the protocol, while it should be kept in mind.

One more thing regarding "ease of use": if we go for one coordinator per IP address in our system (which I think is the case for Tango, more or less), this might force users to start (and possibly monitor) such a coordinator on every machine of the system, and whether we want this is a choice/protocol decision we should definitely keep in mind.

[BenediktBurger]

Thanks for your thoughts.

Two notes regarding one or multiple coordinators:

• you can install them as a OS service, which works great if they are stateless, as Zmq does reconnect automatically.
• in my current implementation you can connect to a coordinator on another system, but you do not need to. That makes the software setup simpler: just connect to localhost.

[bklebel]

in my current implementation you can connect to a coordinator on another system, but you do not need to. That makes the software setup simpler: just connect to localhost.

That is quite what I had in mind. Now, with a day distance, I think it might be good to avoid the coordinator-coordinator-coordination for the time being, and stick to one (per necessary type, if multiple types will be necessary), but I am not quite sure how to keep it in mind while avoiding it...especially if we define that in terms of a protocol