Proposal for Handling Long-Running Actions

[nsoblath]

Problem

In some cases actions started by a Dripline request (e.g. via an OP_CMD) are long-running, so the reply is not sent until long after the standard timeout for the client (10 seconds is typical). This situation requires the user to know that the command is long-running and set an appropriate custom timeout.

Proposed Solution

The basic idea is that we let the service, which is performing the action, inform the client that it's working, and that the client should wait longer for a response.

Here are the details:

• We define a second type of reply message, the heartbeat reply. Just like requests have the msgop field where an integer defines types of requests, we would define a field for replies that is an integer that defines the type of reply.
• The heartbeat reply can have detailed information about the status of the action in the retcode and return_msg, as well as in the payload.
• The heartbeat reply should also have a field that counts the number of heartbeats.
• The client should have a standard timeout time, just as it does now, e.g. 10 seconds.
• The service should have a heartbeat interval, which is by default less than the client timeout, e.g. 8 seconds. If an action takes shorter than the heartbeat interval, it just sends the standard reply about the action. If an action takes longer than the heartbeat interval, it sends a heartbeat reply every heartbeat-interval-seconds until the action is complete, at which point it sends a standard reply.
• Here's the sequence of events for a long-running action:
  1. Client sends a request to the service and starts waiting for a reply with a reply-timeout (RT) seconds
  2. Service receives the request and starts the action
  3. After heartbeat-interval (HBI) seconds, Service sends a heartbeat reply
  4. Client receives the heartbeat reply and resets its wait timer to RT seconds
  5. After another HBI seconds, Service sends a heartbeat reply
  6. Client receives the heartbeat reply and resets its wait timer to RT seconds
  7. After < HBI seconds the action is complete
  8. Service sends a standard reply to the Client
  9. Client receives the reply and handles it appropriately

[laroque]

A few notes, consistent with the above:

• A given request may produce any number of heartbeat replies, but should always produce exactly one non-heartbeat reply, which is always the last reply. (ie if a client receives a non-heartbeat, then no more replies are coming, if it receives a heartbeat, more information is always expected).
• I want to stress the second bullet. I'm imagining that a "generic" command will work in the sequence above, where most heartbeats just have retcode 0 and empty payload and return_msg. However, some processes will intermingle more detail (eg, a multi-get may send values from each target as they are received, in addition to the full set of values in the final reply, the ESR service may send out something indicating scan of coil <n>/<N>; <X>% complete. These messages may come at the HBI, or they may come when events trigger them (like changing coil) and reset the HBI.
• All of the replies should be sent to the exact routing key as is done currently. Also, we may want to change the consume pattern (maybe this is just a dripline-python statement) to not have auto-ack and to not ack the messages as they are seen. That way the sequence of replies remains available if the client disconnects and later reconnects, this would also require that the queues be made durable but with a lifetime. I'm considering the case of web clients in particular, where hitting a refresh button will result in things needing to reconnect.

Questions:

• Related to my last point above, should we consider a change to how reply-to is created by clients? I'd like to be able to start an ESR or RGA scan from a browser, close it or connect as another user, and then see this information. If the reply-to is generated via a UUID then a new connection has no way to figure out what that binding should be.