omnilib / aiomultiprocess

Take a modern Python codebase to the next level of performance.
https://aiomultiprocess.omnilib.dev
MIT License
1.78k stars 101 forks source link

Check for existing asyncio event loops after forking #5

Open amyreese opened 6 years ago

amyreese commented 6 years ago

It seems there are some cases where there is already an existing, active event loop after forking, which causes an exception when the child process tries to create a new event loop. It should check for an existing event loop first, and only create one if there's no active loop available.

Target function: https://github.com/jreese/aiomultiprocess/blob/master/aiomultiprocess/core.py#L93

See #4 for context.

openalmeida commented 6 years ago

Hi,

RuntimeError reproduced, using the code of yours from https://gist.githubusercontent.com/jreese/73e05b7e407f42bb748b406f42e3abd6

i've tried Python 3.6.0, 3.6.1, 3.6.2 all of them pure fresh and clean new installed by pyenv, and with no more operations, just clone aiomultiprocess master (ef791e9) to pwd, and run the code copied from url of above.

Results

python 3.6.1, 3.6.2 works fine, but 3.6.0 will raise RuntimeError the same as what issue #4 said.

Details

I've created a toy script following your example code, but without the database parts, and it works just fine. See https://gist.github.com/jreese/73e05b7e407f42bb748b406f42e3abd6.

It sounds like your code, or one of your dependencies, is automatically creating an asyncio event loop after the process is forked, or otherwise is using as asyncio event loop that isn't friendly to forked processes. In most cases after a fork, there shouldn't be an active asyncio event loop, which is why aiomultiprocess creates one when initializing the child process. That said, it's not difficult to see if there's an existing event loop active first, I've created issue #5 to track that.