Queue Server is using JSON RPC protocol for communication between the manager process and other processes. In this PR the communication code is modified so that the messages are not encoded as JSON string by the process sending the message and the decoded by the receiving process. Conversion data (e.g. a large array) to and from a JSON string is slow and completely unnecessary for interprocess communication within one application. Instead, the JSON RPC message is sent as a dictionary via multiprocessing.Pipe, which automatically pickles the outgoing message and then unpickles it at the receiving end. The change does not affect user-facing API, but is expected to make Queue Server more stable due to shorter interprocess communication delays.
Other changes: json-rpc package is replaced with custom implementation of JSON RPC handler, which supports the option to enable/disable JSON encoding (json-rpc is no longer a dependency); new parameter use_json in the constructors of PipeJsonRpcReceive and PipeJsonRpcSendAsync classes; better processing of SIGTERM by the watchdog process, which is now reliably closes the manager and the worker processes (e.g. if kill <pid> is called on the watchdog process, it does not work with kill -9 <pid>)
Queue Server is using JSON RPC protocol for communication between the manager process and other processes. In this PR the communication code is modified so that the messages are not encoded as JSON string by the process sending the message and the decoded by the receiving process. Conversion data (e.g. a large array) to and from a JSON string is slow and completely unnecessary for interprocess communication within one application. Instead, the JSON RPC message is sent as a dictionary via
multiprocessing.Pipe, which automatically pickles the outgoing message and then unpickles it at the receiving end. The change does not affect user-facing API, but is expected to make Queue Server more stable due to shorter interprocess communication delays.Other changes:
json-rpcpackage is replaced with custom implementation of JSON RPC handler, which supports the option to enable/disable JSON encoding (json-rpcis no longer a dependency); new parameteruse_jsonin the constructors ofPipeJsonRpcReceiveandPipeJsonRpcSendAsyncclasses; better processing ofSIGTERMby the watchdog process, which is now reliably closes the manager and the worker processes (e.g. ifkill <pid>is called on the watchdog process, it does not work withkill -9 <pid>)