Processes

[gentlegiantJGC]

So far the application has been implemented as once process per world and one process for the landing window. This has its advantages and disadvantages and I want to formally discuss them before continuing forward.

One process per world

Advantages

Not limited by the GIL

Each process has its own GIL making it faster to run operations on two or more worlds at the same time.

World based variables

World based variables can be global making it easier to interact with them.

Disadvantages

Unique Memory

Each process has its own memory making it difficult and slow to transfer data between the processes. Eg if a user wanted to copy a selection between open worlds the first process would need to load the data and send it to the second process

Synchronisation

The state of plugins and other data would need to be synchronised between the many processes which could be difficult to manage.

One process for all worlds

Advantages

Shared memory

All memory is shared between worlds making it possible to pull data from one world and apply it to another without the need to pickle and send the data between processes. This means that any inter-level interaction would be faster.

Shared plugins

One plugin instance would manage all worlds making it easier to enable and disable plugins.

Disadvantages

Limited by the GIL

No two python functions could be run at the same time making the program run slower. Running operations on more than one world at the same time would take N times as long. Bottleneck code could be rewritten in Cython to remove this issue.

World based variables

World based variables would need to be stored in a more complex fashion because multiple worlds would be open in the same process.

Conclusion

Both solutions have their advantages and disadvantages. One process for all worlds would be easier to implement because inter-process communication would not be required. It would be slower for multi-world parallel editing unless a significant portion of the world editing library was rewritten in C. There are vague talks of removing the GIL however if it does happen it won't be for a number of years.

[jevexendo]

One thing I would like to note is that there are potential benefits to isolating plugins between world instances. For instance, if you have a project that uses plugins X and Y but is incompatible with Z and another world which uses plugin Z, then the process isolation actually works out quite nicely.

[gentlegiantJGC]

That is a good point. I haven't really thought about how to manage non-global plugins.

[jevexendo]

I would agree in that the lack of shared memory could be troublesome to work with when trying to copy a selection between worlds, but in order to solve that problem, I think it could be reasonable to just export the selection as a schematic in a global clipboard folder.

Perhaps we wouldn't want to apply that logic to all copy operations since I'd imagine it's non-trivially slower than just keeping the selection in RAM, but I feel like this is the only major disadvantage of the one world per instance method and it feels like there should be a reasonable solution.

[Podshot]

I like the one process per world approach, while the separate memory disadvantage is a big factor we can at least kinda workaround that while the GIL is out of our reach and a more complex system for world based variables might be tricky to implement depending on how we approach it. I also agree with Jevex that there are some good benefits from isolating plugins into each world instance, we could probably add in a system down the line to help plugins transfer data between world instances if needed.

[gentlegiantJGC]

I would need to experiment to find the optimal solution. We either: 1) Save all copied regions to a temporary directory on disk where any process can read them. This could be slower for users with slow HDDs. 2) Save all copied regions in RAM and use IPC to send the data between processes as required. 3) Save all copied regions in RAM and push to disk only when another process requires it.

2 is probably the best solution and would allow us to stream data on a chunk basis rather than having to serialise the whole area which may be very large.

I haven't really done IPC before but I can look into it.

[gentlegiantJGC]

I think the scope of this question has been answered. I will open new tickets for future questions.