PVA message ordering not guaranteed

[mdavidsaver]

The behavior of PVA wrt. ordering of network operations is (intentionally) not well defined. eg. if the handling of a Put operation queues a Monitor before completing, it is not guaranteed that the client will see the Monitor update before the Put completion.

This is a direct result of the fair_queue behavior of the message send queue used by both client and server. This code was put in place to mitigation starvation issues like those seen with cagateway where a high bandwidth subscription will cause high latency on concurrent operations.

https://github.com/epics-base/pvAccessCPP/blob/d4f3abf461547393172e263f7a62b0669352a814/src/utils/pv/fairQueue.h#L34-L55

This surprised and inconvenienced @thomascobb.

[mdavidsaver]

I don't know an quick way to provide an ordering guarantee. The simply solution of removing/disabling the fair_queue risks introducing contention issues in large IOCs/gateways. It might be safe to disable fair_queue if/when per-subscription flow control was enabled by default (aka. default to "record[pipeline=true]"). Though this would not give @thomascobb the guarantee he wants as flow control might prevent a monitor update from being sent before the Put completion.

In thinking about this, I don't see any transparent solution. Either low level (eg. explicit sync. message) or high level (extra fields and a protocol between client/server) changes seem necessary.

[thomascobb]

The main use case for this is for a PV that contains a number of NTScalar structures and accepts RPC. An RPC could modify a number of these NTScalars atomically and then return when done. Ideally I would like to monitor the entire structure, see the initial update come in, do an RPC that modifies a number of sub-structures, then see a single update of these NTScalars, followed by the RPC return.