Huge number of workers fired up when I tried to run the app using PyInstaller generated byte code

saiprasadjnv commented 3 years ago

Checklist

[Yes ] The bug is reproducible against the latest release and/or master. [Tested using uvicorn version 0.12.1]
[Yes] There are no similar issues or pull requests to fix it yet.

Describe the bug

We bundled our app using pyinstaller byte code. When we try to use 4 workers to run the application, a huge number of workers are fired up resulting in our server going down. When we remove the "workers = 4" parameter in the Uvicorn.run(), everything is working fine.

To reproduce

Server: RHEL Server on AWS EC2 instance

When I generated the pyinstaller byte code using the below files, everything works fine.
"test.py" and "test.spec" files used to generate the byte code executable. Steps:

Install PyInstaller and run "pyinstaller --onefile -w test.spec". A byte code executable 'test' is generated in dist/ directory.
Run the executable as './test'. testspec.txt testpy.txt

But when we set the number of workers to 4, huge number of workers are fired up resulting in server going down. test1spec.txt test1py.txt Steps: Repeat the same procedure as above with test1.spec.

Expected behavior

Expected 4 workers to start.

Actual behavior

Huge number of workers are fired up resulting in server going down.

Debugging material

Environment

OS: RHEL Server on AWS EC2 instance Python: Python 3.6.8 Uvicorn version: uvicorn 0.12.1 with CPython 3.8.3 on Linux

uvicorn.run('test:app', host='0.0.0.0', workers=4, port=7500)

Additional context

euri10 commented 3 years ago

sorry i'm not familiar with pyinstaller and failed at pyinstaller --onefile -w test.spec, you'll need to provide more precise info

❯ pyinstaller --onefile -w test1.spec
56 INFO: PyInstaller: 4.2
57 INFO: Python: 3.8.6
59 INFO: Platform: Linux-4.19.0-13-amd64-x86_64-with-glibc2.2.5
62 INFO: UPX is not available.
Traceback (most recent call last):
  File "/home/lotso/PycharmProjects/uvicorn/venv/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/home/lotso/PycharmProjects/uvicorn/venv/lib/python3.8/site-packages/PyInstaller/__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/home/lotso/PycharmProjects/uvicorn/venv/lib/python3.8/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/home/lotso/PycharmProjects/uvicorn/venv/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 725, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/home/lotso/PycharmProjects/uvicorn/venv/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 672, in build
    exec(code, spec_namespace)
  File "test1.spec", line 17, in <module>
    'uvicorn.logging', test1
NameError: name 'test1' is not defined

more importantly, this uvicorn version is old, please try see if that happens on latest one.

saiprasadjnv commented 3 years ago

Hi @euri10:

Please use the attached files instead of the previous files All the steps remain the same.

testspec.txt test1spec.txt test1py.txt testpy.txt

Let me know if you face any issue.

euri10 commented 3 years ago

ok i can trigger the infinite recursion bug, unfortunately I'm not familiar at all with pyinstaller and this is the kind of stuff where having to reboot the computer for every try is kind of.....annoying to say the least :)

That said, nothing indicates it's on uvicorn's side that the issue lies, the logs in the build folder seem to indicates lots of missing modules, I'm not sure what this means nor if this is relevant.

This file lists modules PyInstaller was not able to find. This does not
necessarily mean this module is required for running you program. Python and
Python 3rd-party packages include a lot of conditional or optional modules. For
example the module 'ntpath' only exists on Windows, whereas the module
'posixpath' only exists on Posix systems.

Types if import:
* top-level: imported at the top-level - look at these first
* conditional: imported within an if-statement
* delayed: imported from within a function
* optional: imported within a try-except-statement

IMPORTANT: Do NOT post this list to the issue-tracker. Use it as a basis for
           yourself tracking down the missing module. Thanks!

missing module named 'org.python' - imported by copy (optional), xml.sax (delayed, conditional)
missing module named _winapi - imported by encodings (delayed, conditional, optional), subprocess (optional), test.support (delayed, conditional), multiprocessing.connection (optional), multiprocessing.spawn (delayed, conditional), multiprocessing.reduction (conditional), multiprocessing.shared_memory (conditional), multiprocessing.heap (conditional), multiprocessing.popen_spawn_win32 (top-level), asyncio.windows_events (top-level), asyncio.windows_utils (top-level)
missing module named msvcrt - imported by subprocess (optional), getpass (optional), test.support (delayed, conditional, optional), multiprocessing.spawn (delayed, conditional), multiprocessing.popen_spawn_win32 (top-level), asyncio.windows_events (top-level), asyncio.windows_utils (top-level), click._compat (conditional, optional), click._winconsole (top-level), click._termui_impl (conditional), prompt_toolkit.input.win32 (top-level), IPython.core.page (conditional)
missing module named _frozen_importlib_external - imported by importlib._bootstrap (delayed), importlib (optional), importlib.abc (optional), zipimport (top-level)
excluded module named _frozen_importlib - imported by importlib (optional), importlib.abc (optional), zipimport (top-level)
missing module named urllib.urlretrieve - imported by urllib (conditional, optional), pygments.lexers._php_builtins (conditional, optional)
missing module named urllib.FancyURLopener - imported by urllib (conditional, optional), pygments.lexers._sourcemod_builtins (conditional, optional)
missing module named urllib.urlopen - imported by urllib (conditional, optional), pygments.lexers._lua_builtins (conditional, optional), pygments.lexers._postgres_builtins (conditional, optional)
missing module named urllib.urlencode - imported by urllib (delayed, conditional, optional), IPython.lib.display (delayed, conditional, optional)
missing module named _overlapped - imported by asyncio.windows_events (top-level)
missing module named multiprocessing.set_start_method - imported by multiprocessing (top-level), multiprocessing.spawn (top-level)
missing module named multiprocessing.get_start_method - imported by multiprocessing (top-level), multiprocessing.spawn (top-level)
missing module named multiprocessing.get_context - imported by multiprocessing (top-level), multiprocessing.pool (top-level), multiprocessing.managers (top-level), multiprocessing.sharedctypes (top-level)
missing module named multiprocessing.TimeoutError - imported by multiprocessing (top-level), multiprocessing.pool (top-level)
missing module named winreg - imported by platform (delayed, optional), mimetypes (optional), urllib.request (delayed, conditional, optional), IPython.utils.path (delayed, conditional, optional), jedi.api.environment (delayed), appdirs (delayed, conditional), pygments.formatters.img (optional), pkg_resources._vendor.appdirs (delayed, conditional)
missing module named _scproxy - imported by urllib.request (conditional)
missing module named 'java.lang' - imported by platform (delayed, optional), xml.sax._exceptions (conditional)
missing module named multiprocessing.BufferTooShort - imported by multiprocessing (top-level), multiprocessing.connection (top-level)
missing module named multiprocessing.AuthenticationError - imported by multiprocessing (top-level), multiprocessing.connection (top-level)
missing module named multiprocessing.Process - imported by multiprocessing (top-level), watchgod.main (top-level)
missing module named multiprocessing.freeze_support - imported by multiprocessing (top-level), black (top-level)
missing module named multiprocessing.Manager - imported by multiprocessing (top-level), black (top-level)
missing module named asyncio.DefaultEventLoopPolicy - imported by asyncio (delayed, conditional), asyncio.events (delayed, conditional)
missing module named vms_lib - imported by platform (delayed, conditional, optional)
missing module named java - imported by platform (delayed)
missing module named _winreg - imported by platform (delayed, optional), appdirs (delayed, conditional), pygments.formatters.img (optional), pkg_resources._vendor.appdirs (delayed, conditional)
missing module named org - imported by pickle (optional)
missing module named nt - imported by os (delayed, conditional, optional), ntpath (optional), shutil (conditional), pathlib (conditional), ctypes (delayed, conditional)
missing module named pyimod03_importers - imported by /home/lotso/PycharmProjects/uvicorn/venv/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py (top-level)
missing module named urlparse - imported by packaging.requirements (conditional)
missing module named netbios - imported by uuid (delayed)
missing module named win32wnet - imported by uuid (delayed)
missing module named __builtin__ - imported by ipython_genutils.py3compat (conditional), ptyprocess.ptyprocess (optional), pkg_resources._vendor.pyparsing (conditional), pyparsing (conditional)
missing module named ordereddict - imported by pkg_resources._vendor.pyparsing (optional), pyparsing (optional)
missing module named _manylinux - imported by pkg_resources._vendor.packaging.tags (delayed, optional), packaging.tags (delayed, optional)
missing module named 'pkg_resources.extern.pyparsing' - imported by pkg_resources._vendor.packaging.markers (top-level), pkg_resources._vendor.packaging.requirements (top-level)
missing module named StringIO - imported by dotenv.compat (conditional), six (conditional), pkg_resources._vendor.six (conditional)
missing module named 'com.sun' - imported by appdirs (delayed, conditional, optional), pkg_resources._vendor.appdirs (delayed, conditional, optional)
missing module named win32api - imported by IPython.utils._process_win32 (delayed, optional), appdirs (delayed, conditional, optional), pkg_resources._vendor.appdirs (delayed, conditional, optional)
missing module named 'win32com.shell' - imported by appdirs (conditional, optional), pkg_resources._vendor.appdirs (delayed, conditional, optional)
missing module named pkg_resources.extern.packaging - imported by pkg_resources.extern (top-level), pkg_resources (top-level)
missing module named pkg_resources.extern.appdirs - imported by pkg_resources.extern (top-level), pkg_resources (top-level)
missing module named 'pkg_resources.extern.six.moves' - imported by pkg_resources (top-level), pkg_resources._vendor.packaging.requirements (top-level)
missing module named pkg_resources.extern.six - imported by pkg_resources.extern (top-level), pkg_resources (top-level)
missing module named orjson - imported by fastapi.responses (optional)
missing module named ujson - imported by starlette.responses (optional)
missing module named 'multipart.multipart' - imported by starlette.formparsers (optional), starlette.requests (optional), fastapi.dependencies.utils (delayed, conditional, optional)
missing module named multipart - imported by starlette.formparsers (optional), fastapi.dependencies.utils (delayed, conditional, optional)
missing module named pydantic.create_model - imported by pydantic (top-level), fastapi.exceptions (top-level), fastapi.utils (top-level), fastapi.dependencies.utils (top-level)
missing module named pydantic.ValidationError - imported by pydantic (top-level), fastapi.exceptions (top-level)
missing module named pydantic.BaseConfig - imported by pydantic (top-level), fastapi.utils (top-level)
missing module named pydantic.EmailStr - imported by pydantic (optional), fastapi.openapi.models (optional)
missing module named pydantic.Field - imported by pydantic (top-level), fastapi.openapi.models (top-level)
missing module named pydantic.BaseModel - imported by pydantic (top-level), fastapi.openapi.models (top-level), fastapi.security.http (top-level), fastapi.utils (top-level), fastapi.dependencies.utils (top-level), fastapi.encoders (top-level), fastapi.routing (top-level), fastapi.openapi.utils (top-level)
missing module named pydantic.AnyUrl - imported by pydantic (top-level), fastapi.openapi.models (top-level)
missing module named email_validator - imported by fastapi.openapi.models (optional)
missing module named async_exit_stack - imported by fastapi.concurrency (optional)
missing module named async_generator - imported by fastapi.concurrency (optional)
missing module named 'ipykernel.pylab' - imported by IPython.core.pylabtools (delayed, conditional), IPython.core.display (delayed)
missing module named ipykernel - imported by IPython.core.pylabtools (delayed, optional), IPython.kernel (top-level)
missing module named 'matplotlib.figure' - imported by IPython.core.pylabtools (delayed)
missing module named 'matplotlib.pyplot' - imported by IPython.core.pylabtools (delayed)
missing module named 'matplotlib.backend_bases' - imported by IPython.core.pylabtools (delayed, conditional)
missing module named nbformat - imported by IPython.core.magics.basic (delayed), IPython.core.interactiveshell (delayed, conditional)
missing module named numpydoc - imported by jedi.inference.docstrings (delayed)
missing module named trio - imported by IPython.core.async_helpers (delayed)
missing module named curio - imported by IPython.core.async_helpers (delayed)
missing module named docrepr - imported by IPython.core.interactiveshell (optional)
missing module named win32clipboard - imported by IPython.lib.clipboard (delayed, optional)
missing module named nbconvert - imported by IPython.utils.io (delayed)
missing module named notebook - imported by IPython.utils.io (delayed)
missing module named traitlets.config.Application - imported by traitlets.config (delayed, conditional), traitlets.log (delayed, conditional)
missing module named pygments.lexers.PythonLexer - imported by pygments.lexers (top-level), IPython.core.oinspect (top-level)
missing module named pygments.formatters.LatexFormatter - imported by pygments.formatters (delayed), IPython.lib.display (delayed)
missing module named pygments.formatters.HtmlFormatter - imported by pygments.formatters (delayed), IPython.lib.display (delayed), IPython.core.oinspect (top-level)
missing module named ctags - imported by pygments.formatters.html (optional)
missing module named PIL - imported by pygments.formatters.img (optional)
missing module named IPython.utils.warn - imported by IPython.utils (delayed), IPython.testing.tools (delayed)
missing module named pudb - imported by coverage.debug (delayed)
missing module named thread - imported by coverage.backward (optional), coverage.debug (optional)
missing module named contracts - imported by coverage.misc (conditional), coverage.inorout (delayed, conditional)
missing module named repr - imported by IPython.core.debugger (delayed, optional), coverage.backward (optional)
missing module named ConfigParser - imported by coverage.backward (optional)
missing module named cStringIO - imported by cPickle (top-level), coverage.backward (optional)
missing module named copy_reg - imported by cPickle (top-level), cStringIO (top-level)
missing module named _pypy_irc_topic - imported by coverage.inorout (conditional, optional)
missing module named _structseq - imported by coverage.inorout (conditional, optional)
missing module named numpy - imported by IPython.external.decorators (optional), IPython.lib.display (delayed), toml.encoder (delayed), IPython.core.formatters (delayed, conditional), IPython.core.magics.namespace (delayed, conditional, optional)
missing module named __pypy__ - imported by coverage.collector (delayed, conditional)
missing module named gevent - imported by coverage.collector (delayed, conditional, optional)
missing module named eventlet - imported by coverage.collector (delayed, conditional, optional)
missing module named greenlet - imported by coverage.collector (delayed, conditional, optional)
missing module named testpath - imported by IPython.testing.plugin.ipdoctest (top-level)
missing module named 'nose.util' - imported by IPython.testing.iptest (top-level), IPython.testing.plugin.ipdoctest (top-level)
missing module named 'nose.core' - imported by IPython.testing.iptest (top-level)
missing module named 'nose.plugins' - imported by IPython.testing.iptest (top-level), IPython.testing.plugin.ipdoctest (top-level), IPython.external.decorators._numpy_testing_noseclasses (top-level)
missing module named nose - imported by IPython.testing.iptest (top-level), IPython.external.decorators._decorators (delayed), IPython.testing.decorators (delayed)
missing module named 'nose.tools' - imported by IPython.testing.tools (delayed, optional), IPython.testing.decorators (delayed)
missing module named 'prompt_toolkit.key_binding.key_bindings.vi' - imported by prompt_toolkit.key_binding.vi_state (conditional)
missing module named prompt_toolkit.filters.is_searching - imported by prompt_toolkit.filters (top-level), prompt_toolkit.search (top-level), prompt_toolkit.key_binding.bindings.search (top-level), prompt_toolkit.key_binding.bindings.vi (top-level)
missing module named prompt_toolkit.filters.vi_mode - imported by prompt_toolkit.filters (top-level), prompt_toolkit.document (top-level), prompt_toolkit.key_binding.bindings.page_navigation (top-level), prompt_toolkit.widgets.toolbars (top-level), IPython.terminal.shortcuts (top-level)
missing module named prompt_toolkit.filters.is_done - imported by prompt_toolkit.filters (top-level), prompt_toolkit.layout.menus (top-level), prompt_toolkit.widgets.base (top-level), prompt_toolkit.shortcuts.progress_bar.base (top-level), prompt_toolkit.shortcuts.prompt (top-level)
missing module named prompt_toolkit.filters.has_completions - imported by prompt_toolkit.filters (top-level), prompt_toolkit.layout.menus (top-level), prompt_toolkit.widgets.toolbars (top-level), prompt_toolkit.widgets.dialogs (top-level), IPython.terminal.shortcuts (top-level)
missing module named prompt_toolkit.filters.vi_insert_mode - imported by prompt_toolkit.filters (top-level), prompt_toolkit.layout.containers (top-level), prompt_toolkit.key_binding.bindings.basic (top-level), IPython.terminal.shortcuts (top-level), IPython.terminal.debugger (top-level)
missing module named prompt_toolkit.filters.emacs_insert_mode - imported by prompt_toolkit.filters (top-level), prompt_toolkit.layout.containers (top-level), prompt_toolkit.key_binding.bindings.basic (top-level), prompt_toolkit.key_binding.bindings.emacs (top-level), IPython.terminal.shortcuts (top-level), IPython.terminal.debugger (top-level)
missing module named backports - imported by wcwidth.wcwidth (optional)
missing module named com - imported by appdirs (delayed)
missing module named regex.DEFAULT_VERSION - imported by regex (delayed, optional), regex.regex (delayed, optional)
missing module named prompt_toolkit.filters.vi_insert_multiple_mode - imported by prompt_toolkit.filters (top-level), prompt_toolkit.layout.processors (top-level)
missing module named pysqlite2 - imported by IPython.core.history (optional)
missing module named cPickle - imported by pickleshare (optional)
missing module named pathlib2 - imported by pickleshare (optional)
missing module named 'IPython.kernel.zmq' - imported by IPython (delayed)
missing module named 'ipykernel.embed' - imported by IPython (delayed)
missing module named System - imported by IPython.utils._process_cli (top-level)
missing module named clr - imported by IPython.utils._process_cli (top-level)
missing module named _yaml - imported by yaml.cyaml (top-level)
missing module named win32evtlog - imported by logging.handlers (delayed, optional)
missing module named win32evtlogutil - imported by logging.handlers (delayed, optional)

All the above to say that I'll be happy to reopen if there's indication that we could do something about it, I suppose you'll get more help on pyinstaller's side, the use-case is not supported here nor will be until someone steps up, and that someone wont be me >_)

selimb commented 3 years ago

@saiprasadjnv Can you try adding multiprocessing.freeze_support() to the if __name__ == "__main__": block?

selimb commented 3 years ago

Even though the docs suggest this is only required on Windows, and that's true if you look at the code, pyinstaller monkeypatches that function.

Relevant issues: https://github.com/pyinstaller/pyinstaller/pull/4868 and https://github.com/pyinstaller/pyinstaller/issues/4865

SMAntony commented 1 week ago

Hi @euri10:

Please use the attached files instead of the previous files All the steps remain the same.

testspec.txt test1spec.txt test1py.txt testpy.txt

Let me know if you face any issue.

Hi, Were you able to find any workarounds?

encode / uvicorn