IPC rewrite

[DomClark]

Our current IPC system for communicating with the remote plugins (RemoteVstPlugin and RemoteZynAddSubFx) has a number of issues:

• Cannot reliably send messages from the remote plugins to LMMS: Messages can be sent from LMMS to the remote plugin at any time, but not in the other direction. LMMS only checks for incoming messages when waiting for a response to a message it sent itself. This means there are no guarantees as to when a message from the remote plugin will be received. This is blocking a number of bugs and enhancements related to VST plugins.
• Messages are limited in size: Messages cannot be larger than the IPC buffer, which has a couple of drawbacks. First, we have to allocate a relatively large buffer (currently 512KB) in order to be able to handle large messages. Secondly, some very large data cannot be passed through the IPC system and instead requires some alternative channel. For example, VST save data is saved to a temporary file, whose name is passed via the IPC system.
• Inefficient and not real-time safe: Currently all data is passed as strings. This adds the unnecessary overhead of converting non-string data (ints, floats, etc.) to strings, compared to passing a binary representation. Moreover, the use of strings means that serialising and deserialising messages can potentially allocate memory, which is undesirable in a real-time system. There is only a single channel in each direction, which means that real-time audio processing messages have to share a channel with all other messages. This makes real-time safety impossible even if a binary format is used, since memory allocation is required to handle dynamically-sized non-real-time messages, and a mutex needs to be acquired to send messages.
• Varied and undesirable implementations: The current IPC implementation differs between Windows and non-Windows systems. On Windows, shared memory is used along with semaphores, whereas elsewhere sockets are used. Having multiple implementations increases complexity and reduces maintainability of the code. Additionally, Qt is used for the semaphores on Windows, which we are trying to remove from core code.
• Responses are synchronously handled: If a response is required from a message, the thread that sent the message has to block until the response is received. This has caused complex deadlocks in the past, especially in UI code where the remote plugin can block on LMMS through the OS's native message system. We work around this now by running a message pump while waiting for a reply, but that causes high CPU usage, as well as unintuitive behaviour where one UI operation can happen in the middle of another. Asynchronous response processing would be more efficient and less bug prone.

Proposed approach to a replacement:

I propose the following three-layered system as a replacement. It should be possible to implement each layer in sequence, with the existing code modified at each step to work on top of the lower-level replacement.

Layer I: channels for sending arbitrarily large blobs of data. The channels will be agnostic as to the format of the data sent, so will work with the current message implementation, as well as with layer II of this proposal. One implementation will be used for all platforms, and any platform specific code will be first-party (no Qt dependency) and encapsulated in objects with a portable interface. I plan to use a ring buffer in shared memory, with the writer thread signalling a semaphore to inform the reader thread that data has been written. The reader can be awoken either because a complete blob has been written, or because the buffer is full and needs to be drained before more data can be written. The buffer does not transmit the size of the blob; this is the responsibility of the user of the buffer. This is to allow a blob to be written and read in multiple parts while minimising how many times the semaphore is signalled. I chose to use shared memory instead of sockets or pipes as it is generally considered to be more efficient, which is important here.

Layer II: typed binary message format. The message format will be agnostic as to how messages are sent and received, so will work with the current IPC interface, as well as with layer III of this proposal. I plan to use MessagePack here as it seems to be simple and popular, but feel free to shill for CBOR/Protobufs/your format of choice. Rather than manually assembling message objects piece-by-piece, it should be possible to represent the data for each message as a normal C++ tuple or struct, which can be automatically serialised and deserialised using template magic. Ideally, serialisation should involve a single copy from the native in-memory representation to a blob in the layer I channel, and deserialisation a single copy in the other direction.

Layer III: asynchronous bidirectional IPC interface. There will be two layer I channels making up the primary IPC link between each remote plugin and LMMS - one for each direction. Both LMMS and the remote plugin will constantly be listening for messages, so it will be possible to send a message in either direction at any time. Message handlers should be implementable as a callback function, which can be registered with the listener for the appropriate message ID. When sending a message for which a reply is expected, a callback function can be passed along with the message, and will be asynchronously invoked when the reply is received. To avoid holding up real-time code in the normal IPC link, there will be a separate real-time IPC link, also consisting of two channels. Unlike the main link, only the remote plugin actively listens for incoming messages, and replies are handled synchronously, blocking the sending thread until they arrive. This is optimised for the typical use case of this channel: when LMMS needs the plugin to process audio, it will send a message to the plugin, and wait for the plugin to complete processing before continuing. This should, in theory, only require signalling a semaphore once in each direction, minimising the number of synchronisation operations in real-time code.

That's about it; thoughts/comments/ideas/criticism/etc. are welcome.

[JavaGamer]

If you are looking for shilling then Cap'n Proto might be a good fit for this. or this message passing library, X9

[Rossmaxx]

Varied and undesirable implementations: The current IPC implementation differs between Windows and non-Windows systems. On Windows, shared memory is used along with semaphores, whereas elsewhere sockets are used. Having multiple implementations increases complexity and reduces maintainability of the code. Additionally, Qt is used for the semaphores on Windows, which we are trying to remove from core code.

Mark as done via #7212