Using `multiprocessing.managers.SharedMemoryManager`

[jakirkham]

Python has a builtin shared memory manager (added in Python 3.8), which could also be spun up in a separate process and used. It doesn't do its own serialization like some of the others here do, but it could reuse Distributed's own serialization methods. The benefit here being that if this worked reasonable well one could use it without additional dependencies. Even if other options performed better, this could still be a reasonable default.

[martindurant]

I don't believe it does reference counting for you; could try it, though.

[jakirkham]

The docs suggest one needs to call close on each SharedMemory instance.

That said, it looks like this happens in SharedMemory.__del__ already.

So maybe one can just rely on that?

[martindurant]

I'll try to give it a go maybe later this week.