MichaelJCaruso
commented
6 years ago Summary and description of this issue complete. Cumulative pull request containing its fix and numerous Vxa enhancements supporting cross-language (js for now) scripting coming later today.
Disclaimer This issue does not affect current production deployments. By design, production deployments use pools to receive and dispatch queries to worker Vision (batchvision) processes. In those deployments, no worker is ever presented with more than one query at a time (i.e., no worker ever receives a burst of queries to process).
Query Processing Overview
batchvision presents two query processing interfaces to the world:
The read-eval-print interface is batchvision original query processing interface. For reasons of re-use and expediency, all queries, regardless of the interface through which they arrive, are processed using batchvision's legacy read-eval-print implementation. In the case of queries arriving via the message passing interface, a VEvaluatorPump provides the necessary stream mapping and coordination.
The life-cycle of every message-based query begins with the receipt by a VEvaluatorPump of an Evaluate message. The pump responds by immediately writing the request's query code and associated execution directives to a pipe that supplies input to the read-eval-print controller. That write is asynchronous along with the pump's subsequent gathering and eventual return of the request's response via an OnResult message to the client:
Client VEvaluatorPump VReadEvalPrintController | | | | Evaluate | | |------------------------>| | | | Code & Directives | | |- - - - - - - - - - - - - >| | | | | | | | | | | | | | | Prompts & Output | | |< - - - - - - - - - - - - -| | OnResult | | |<------------------------| | | | |The round trip that begins with an Evaluate message from a client and ends with an OnResult (or OnError) message back to that client forms the request-response envelope of all queries. The implementation of that envelope as depicted above is simple and robust, requiring no more than reliable asynchronous inter- or inter-process pipe pairs and careful bookkeeping of the flows through those pipes. No additional side-channels beyond those pipes are needed.
Message-passing interfaces are capable of supporting much more than the simple chained request-response pattern of a query's envelope. The one that prompted discovery of this issue involves client callbacks (more generally, message passing conversations between the client and the Vision code processing the client's request). The following stylized interaction illustrates:
Client VEvaluatorPump VReadEvalPrintController | | | | Evaluate | | |------------------------>| | | | Code & Directives | | |- - - - - - - - - - - - - >| | | | | | Who's asking? | | |<==========================| | |==========================>| | (theClient) | | | | |< + + + + + + + + + + + + + + + + + + + | | | | | | 'theClient.createObject' | |<----------------------------------------------------| | | | return (anObject) | |---------------------------------------------------->| | | | | | for (x,y,val) in (xs,ys,vals) | | anObject.addPoint(x,y,val) | |<----------------------------------------------------| | | | | | anObject.doAnalysis () | |<----------------------------------------------------| | | | | | return (analysisResult) | |---------------------------------------------------->| | | | | | | | | | Prompts & Output | | |< - - - - - - - - - - - - -| | OnResult | | |<------------------------| | | | |In the context of batchvision's current message processing implementation, enabling this conversation requires the introduction of a side-channel between the read-eval-print controller and the VEvaluatorPump mediating its connection to the client:
Client VEvaluatorPump VReadEvalPrintController . . . | | Who's asking? | | |<==========================| | |==========================>| | (theClient) | . . .Analysis
As its title stipulates, this issue only occurs while processing bursts of queries that make Vxa (side-channel) client callbacks. The operation of the side-channel was sketched above. With special attention to event timing and delays, the following annotated sketch illustrates query burst processig for a burst of three queries that do not make side channel callbacks:
VEvaluatorPump VReadEvalPrintController | | Evaluate-1 | | ------------------------>| Request-1 | Evaluate-2 |- - - - - - - - - - - - - >| R E-P: Start-1 ------------------------>| Request-2 | Evaluate-3 |- - - - - - - - - - - - - >| R-E-P: Queue-2 ------------------------>| Request-3 | |- - - - - - - - - - - - - >| R-E-P: Queue-3 | | | | | | (a) | (Response-1)< - - - | R-E-P: End-1, Start-2 | | (b) |< - - -(Response-1) | Pump: End-1, Start-2 OnResult-1 | | <------------------------| | | | | | | | (a) | (Response-2)< - - - | R-E-P: End-2, Start-3 | | (b) |< - - -(Response-2) | Pump: End-2, Start-3 OnResult-2 | | <------------------------| | | | | | | | (a) | (Response-3)< - - - | R-E-P: End-3 | | (b) |< - - -(Response-3) | Pump: End-3 OnResult-3 | | <------------------------| | | |Assuming that both the evaluator pump and read-eval-print loop are idle at the start of the burst, all three queries are re-formatted and written to the read-eval-print controller's input pipe in quick succession. As expected, the read-eval-print controller begins processing the first query immediately, leaving the second and third in the pipe. Noteworthy (and, in fact, critical to this issue) is pipe latency. While not illustrated for the request pipe, it is represented for the response pipe using interrupted horizontal lines to indicate that responses always arrive at the evaluator pump sometime after being written to the response pipe. That is particularly important because different state changes are associated with transmission and receipt. As noted in the side-bar annotations associated with response transmissions (a), the read-eval-print controller immediately considers itself done with the query it was processing and immediately begins processing the next query in its input pipe. The pump does the same the instant it receives a response to the request it was processing (b). Unsurprisingly, the pipe's timing latency will always create a timing window during which the pump and read-eval-print controller disagree over which request they are processing.
Given the discussion of the previous paragraph, the fact that the side channel always returns the evaluator pump's current client in answer to who's asking?, and the required experimental confirmation, what's going on is clear. Side-channel calls are simply happening during the window of disagreement for the second through final queries in a query burst:
VEvaluatorPump VReadEvalPrintController | | Evaluate-1 | | ------------------------>| Request-1 | Evaluate-2 |- - - - - - - - - - - - - >| R E-P: Start-1 ------------------------>| Request-2 | Evaluate-3 |- - - - - - - - - - - - - >| R-E-P: Queue-2 ------------------------>| Request-3 | |- - - - - - - - - - - - - >| R-E-P: Queue-3 | | | Who's asking? (1) | |<==========================| R-E-P: Running-1 |==========================>| Pump: Running-1, OK | client-1 | | | | | | | (a) | (Response-1)< - - - | R-E-P: End-1, Start-2 | | | Who's asking? (2) | |==========================>| R-E-P: Running-2 |==========================>| Pump: Running-1, WRONG | client-1 | | | (b) |< - - -(Response-1) | Pump: End-1, Start-2 OnResult-1 | | <------------------------| | | | | | | | (a) | (Response-2)< - - - | R-E-P: End-2, Start-3 | | | Who's asking? (3) | |==========================>| R-E-P: Running-3 |==========================>| Pump: Running-2, WRONG | client-2 | | | (b) |< - - -(Response-2) | Pump: End-2, Start-3 OnResult-2 | | <------------------------| | | | | | | | | (Response-3)< - - - | R-E-P: End-3 | | |< - - -(Response-3) | Pump: End-3 OnResult-3 | | <------------------------| | | |Repair
The easiest (and chosen) short term repair is to maintain request counters in both the evaluator pump and read-eval-print controllers, using the values of those counters in the side-channel to unambiguously identify the desired request's client. Longer term and for a number of other very good reasons, the better approach is to rewrite batchvision's query processing machinery to provide direct native support for the message-based interface.