add shared memory between SubInterpreters

[ddovnar]

The pull request before, was not correct, so I try to fix... This feature will give me what I need. You can check test file TestSharedMemoryInSubInterpreter for more detail. It works in the default state as before, as it is only like additional parameter and may be optional (or experimental). Not all developers need to use it, but the main reason why it's needed in my case is that: I want to integrate java with python directly in the spring-cloud application. There are many ways to do such an integration, but I need use python as a micro-service in the spring-cloud ecosystem, and this feature will allow me not to worry about workarround in the load-balancing and use python facilities closer to java application.

[bsteffensmeier]

PEP-684 requires that all shared objects meet strict immutability requirements that this change does not enforce. I am worried that these changes will not be maintainable with future version of python once PEP-684 is integrated

[jsnps]

Hey @bsteffensmeier, a) does your concern also apply for SharedInterpreters? I wish a little bit more control over the MemoryManager lifetime, so that I don't need to keep an interpreter open on a dedicated thread, but can create a SharedInterpreter on the fly on any thread and reuse any PyObject from my MemoryManager, until I close it. This allows to implement a slim wrapper around SharedInterpreter to be accessible from any thread. b) Have you any other concern with the fact that there might be times without any jep instance and PyObjects outliving the instances? In my experiments I didn't see any problem with this.

(sorry for hijacking this pull request, but I thought this is very related)

Thanks :)

[bsteffensmeier]

Hey @bsteffensmeier, a) does your concern also apply for SharedInterpreters? I wish a little bit more control over the MemoryManager lifetime, so that I don't need to keep an interpreter open on a dedicated thread, but can create a SharedInterpreter on the fly on any thread and reuse any PyObject from my MemoryManager, until I close it. This allows to implement a slim wrapper around SharedInterpreter to be accessible from any thread.

I have no similar concerns for SharedInterpreter, the low level c-api fully supports sharing python objects between threads of the same interpreter which is what SharedInterpreter is doing. For SubInterpreters the documentation is not clear on the safety of sharing objects in legacy mode but with the isolated interpreters for PEP-684 it is clearly not safe to share objects between SubInterpreters. Because we plan to support isolated sub-interpreters in future versions of Python I do not think we can accept this PR.

b) Have you any other concern with the fact that there might be times without any jep instance and PyObjects outliving the instances? In my experiments I didn't see any problem with this.

The only problem with PyObjects outliving jep instances is that in some cases it could be considered a memory leak. If your application is going to create a new SharedInterpreters there is no problem with new SharedInterpreters cleaning up the memory for the old SharedInterpreters, it would only be a problem when an application stops making new SharedInterpreters.

I agree the behavior of invalidating all PyObjects when the last SharedInterpreter is closed is annoying in many cases. I had hoped this behavior would be temporary but unfortunately I have not found the time to work on it more. My goal is to eventually let all SharedInterpreters share PyObjects with no requirement for a constantly running SharedInterpreter. My plan is to add the ability for the MainInterpreter to cleanup references for the MemoryManager used by SharedInterpreters. Since the MainInterpreter is always running and using the same cpython interpreter as SharedInterpreters it would always be able to handle the cleanup which would mean lingering PyObjects are not "leaking" while waiting for a new SharedInterpreter. I do not anticipate I will have time to implement this for the 4.2 release of Jep but if anyone else wants to try then i would be happy to review a PR that allows PyObjects to outlive SharedInterpreters.