search

uqfoundation / multiprocess

better multiprocessing and multithreading in Python
http://multiprocess.rtfd.io
Other
639 stars 64 forks source link

Python 3.12 fork multithreaded warning #166

Closed enzbus closed 1 year ago

enzbus commented 1 year ago

Hello, and thank you for providing multiprocess to the community!

I'm getting the following warning with Python 3.12 on MacOS 

.../env/lib/python3.12/site-packages/multiprocess/popen_fork.py:66: DeprecationWarning: This process (pid=XXXXX) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()

My code works fine, and on earlier Python versions it does as well but it doesn't give this warning. Is there anything to worry about?

Thanks!

PS This is the version I'm using:

>>> import multiprocess
>>> multiprocess.__version__
'0.70.15'
>>>
mmckerns commented 1 year ago

python 3.12 is not formally supported in multiprocess 0.70.15, so I don't know what warnings were emitted at that version. I do remember that python has started triggering this warning for open_fork in 3.12. I'm not seeing that warning any further with a quick check in master, where 3.12 is formally supported. (I'm just about to cut a release, btw). Can you check the behavior you experience in master, and if it still occurs, post some simple code that triggers the warning?

Fundamentally, there's nothing to worry about. Python is deprecating the fork context, and it will make pickling a bit harder, so there may be some mild loss of functionality if I don't get a few small issues resolved before they fully deprecate it. I believe that's several major versions away.

enzbus commented 1 year ago

Thank you! It seems I'm unable to reproduce this, I had a virtual environment in which I ran cvxportfolio test suite this morning under 3.12 (which uses multiprocess) and when I recreated it the warning wasn't there any more. I'll let you know if I see it again, in any case everything works under 3.12 even on the current PyPI release.

enzbus commented 1 year ago

Maybe the warning was thrown in a worker process and reported because there was a bug that made it fail (unrelated, due to changed numerical accuracy of sum() in 3.12) which I fixed in the meantime. I'll check.

mmckerns commented 1 year ago

I'm going to close this. Feel free to reopen/comment if you have follow-up.