Inter-process communication (IPC) implementation

[gentlegiantJGC]

I am working through how IPC will be implemented for Amulet.

IPC is required to notify all processes that a plugin has been globally enabled or disabled as well as transferring data between the processes as required.

Multiprocessing.Queue

The current implementation transmits data with a multiprocessing Queue object initialised by the parent process and passed to the child's initialiser. There is no way to initialised this connection after the process has started.

The implementation has a single parent process and all other processes are direct children of that process. Child processes can launch more processes by calling up to the parent process.

This implementation only allows for communication between the parent and child because it is not known which child processes will need to communicate.

If the user ran the Amulet executable again it would not be aware of the first instance and would have no way to interact with it.

Sockets

Sockets use TCP to communicate between processes. They can be opened and closed as required making them easier to work with. Objects would need to be manually serialised to bytes or a subclass wrapper written to handle that. We would need a mechanism to find the address of each process so that a connection can be made. This could be implemented with either a file on disk or a manager process at a fixed address.

Multiprocessing Client and Listener

I have just come across these classes that are a higher level implementation of sockets that handle pickling objects.

QLocalServer and QLocalSocket

These are classes implemented by Qt. They offer some benefits over the first party options. 1) Connections are made by an arbitrary string. The world path could be used for this simplifying connection. 2) It is push based using Qt's signal system to notify when new data comes in.

Extra thoughts

It would be nice to be able to execute Amulet multiple times and have them interact with previous instances like they are all the same. We would need to consider how it would work if different versions of Amulet were launched at the same time for compatibility reasons.

[jevexendo]

In terms of handling multiple versions of Amulet, I think that each version should be treated as a different application since I don't think we want to deal with the bugs that such a feature could introduce.

[jevexendo]

So what kind of data we expect to transfer between instances of Amulet? The way we resolve this will probably vary depending on what all needs to be transferred and how that information will be stored by each instance.

For instance, plugins will be saved as files somewhere on the user's computer so we probably won't need to send plugin files between instances. User selections probably will need to be transferred but I'm curious if we could hook into the clipboard on the user's computer to transfer that data. I'm not familiar with the complexity and limitations of the clipboard.

[gentlegiantJGC]

I came across QLocalServer and QLocalSocket yesterday which seem like a better option. I have added them to the top post.

Transfer of data is one component I am thinking about. Eg moving a copied structure between processes. This problem could also be solved with QSharedMemory or by just saving the data to a file.

The extended version is being able to call functions in a remote process. PySide6 has QDBus which allows calling slots across processes but it is linux only. I think we could implement a cross-platform version of this using the socket mechanism.

A mechanism like this will be required to notify all processes that a plugin has been enabled or disabled globally.

[gentlegiantJGC]

When we are doing world conversion with a source world that is already open we will need to either close the process that holds the source world or use IPC to pull the data for each chunk to paste into the destination world.