PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.41k stars 1.64k forks source link

Uvicorn must be installed globally or available in path #9290

Closed flapili closed 1 year ago

flapili commented 1 year ago

First check

Bug summary

if we don't have uvicorn available (either installed globally or via venv activation) the command prefect server start crash

Reproduction

- create a venv via `python3 -m venv .venv`
- run `./venv/bin/prefect server start` (replace `bin` by `Script` on windows)

Error

./venv/bin/prefect server start

 ___ ___ ___ ___ ___ ___ _____ 
| _ \ _ \ __| __| __/ __|_   _| 
|  _/   / _|| _|| _| (__  | |  
|_| |_|_\___|_| |___\___| |_|  

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api

View the API reference documentation at http://127.0.0.1:4200/docs

Check out the dashboard at http://127.0.0.1:4200

Traceback (most recent call last):
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/cli/server.py", line 144, in start
    server_process_id = await tg.start(
                        ^^^^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in start
    return await future
           ^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 258, in run_process
    async with open_process(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/prefect/utilities/processutils.py", line 202, in open_process
    process = await anyio.open_process(command, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/anyio/_core/_subprocesses.py", line 127, in open_process
    return await get_asynclib().open_process(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/benoitdeveaux/Documents/perso/prefect-auth/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1105, in open_process
    process = await asyncio.create_subprocess_exec(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 1694, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/unix_events.py", line 207, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/unix_events.py", line 810, in _start
    self._proc = subprocess.Popen(
                 ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 1901, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'uvicorn'
An exception occurred.

Versions

Version:             2.10.5
API version:         0.8.4
Python version:      3.11.2
Git commit:          d48dfbbc
Built:               Thu, Apr 20, 2023 4:06 PM
OS/Arch:             darwin/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.39.4

Additional context

according to the doc :

You don't specifically need to activate a virtual environment, as you can just specify the full path to that environment's Python interpreter when invoking Python. Furthermore, all scripts installed in the environment should be runnable without activating it. (https://docs.python.org/fr/3/library/venv.html#how-venvs-work)

sometime we can't use activated venv (in systemd service as example) or it's not practical and using the full path is more simple.

flapili commented 1 year ago

I did this fix :

@orion_app.command()
@server_app.command()
async def start(
    host: str = SettingsOption(PREFECT_SERVER_API_HOST),
    port: int = SettingsOption(PREFECT_SERVER_API_PORT),
    keep_alive_timeout: int = SettingsOption(PREFECT_SERVER_API_KEEPALIVE_TIMEOUT),
    log_level: str = SettingsOption(PREFECT_LOGGING_SERVER_LEVEL),
    scheduler: bool = SettingsOption(PREFECT_API_SERVICES_SCHEDULER_ENABLED),
    analytics: bool = SettingsOption(
        PREFECT_SERVER_ANALYTICS_ENABLED, "--analytics-on/--analytics-off"
    ),
    late_runs: bool = SettingsOption(PREFECT_API_SERVICES_LATE_RUNS_ENABLED),
    ui: bool = SettingsOption(PREFECT_UI_ENABLED),
):
    """Start a Prefect server"""

    server_env = os.environ.copy()
    server_env["PREFECT_API_SERVICES_SCHEDULER_ENABLED"] = str(scheduler)
    server_env["PREFECT_SERVER_ANALYTICS_ENABLED"] = str(analytics)
    server_env["PREFECT_API_SERVICES_LATE_RUNS_ENABLED"] = str(late_runs)
    server_env["PREFECT_API_SERVICES_UI"] = str(ui)
    server_env["PREFECT_LOGGING_SERVER_LEVEL"] = log_level

    base_url = f"http://{host}:{port}"

    # Check if uvicorn is installed in a venv
    if os.name == "nt":
        uvicorn_path = Path(sys.executable).parent / "Scripts" / "uvicorn.exe"
    else:
        uvicorn_path = Path(sys.executable).parent / "uvicorn"

    if not uvicorn_path.exists(): # should try to run uvicorn --version and catch the error instead ?
        uvicorn_path = "uvicorn"  # fallback to global uvicorn installation

    async with anyio.create_task_group() as tg:
        app.console.print(generate_welcome_blurb(base_url, ui_enabled=ui))
        app.console.print("\n")

        server_process_id = await tg.start(
            partial(
                run_process,
                command=[
                    str(uvicorn_path),
                    "--app-dir",
                    # quote wrapping needed for windows paths with spaces
                    f'"{prefect.__module_path__.parent}"',
                    "--factory",
                    "prefect.server.api.server:create_app",
                    "--host",
                    str(host),
                    "--port",
                    str(port),
                    "--timeout-keep-alive",
                    str(keep_alive_timeout),
                ],
                env=server_env,
                stream_output=True,
            )
        )

        # Explicitly handle the interrupt signal here, as it will allow us to
        # cleanly stop the uvicorn server. Failing to do that may cause a
        # large amount of anyio error traces on the terminal, because the
        # SIGINT is handled by Typer/Click in this process (the parent process)
        # and will start shutting down subprocesses:
        # https://github.com/PrefectHQ/server/issues/2475

        setup_signal_handlers_server(
            server_process_id, "the Prefect server", app.console.print
        )

    app.console.print("Server stopped!")

any advise about the implementation / idea before I submit an PR ?

rpeden commented 1 year ago

@flapili you can probably achieve something similar with

import sys

# ...existing code 

        server_process_id = await tg.start(
            partial(
                run_process,
                command=[
                    sys.executable,
                    "-m",
                    "uvicorn",
                    "--app-dir",
                    # quote wrapping needed for windows paths with spaces
                    f'"{prefect.__module_path__.parent}"',
                    "--factory",
                    "prefect.server.api.server:create_app",
                    "--host",
                    str(host),
                    "--port",
                    str(port),
                    "--timeout-keep-alive",
                    str(keep_alive_timeout),
                ],
                env=server_env,
                stream_output=True,
            )
        )

This should run Uvicorn using the same Python interpreter you ran the Prefect CLI with, and doesn't need an OS-dependent lookup for the uvicorn executable. I just tested this by editing the server CLI code and it worked as expected on both Windows and Linux.

I recommend waiting for input for someone from the Prefect team before making a PR; there might be a better way to do this that I'm overlooking.

flapili commented 1 year ago

I totally forgot about-m, you rock ! 👍

rpeden commented 1 year ago

@flapili FWIW I believe that both your solution and my suggestion have the same issue with spaces in Windows paths that you see later in the command array. And I don't think the string-inside-a-string approach used for the module path arg will work universally for the command itself. It will work on Windows, but not elsewhere.

So you'd probably still need something like:

if os.name == "nt":
    python_path = f'"{sys.executable}"'
else:
    python_path = sys.executable

server_process_id = await tg.start(
    partial(
        run_process,
        command=[
            python_path,
# ...other code

or even just

server_process_id = await tg.start(
    partial(
        run_process,
        command=[
            f'"{sys.executable}"' if os.name == "nt" else sys.executable,

And that should work no matter where you run it.

zanieb commented 1 year ago
if os.name == "nt":
    python_path = f'"{sys.executable}"'
else:
    python_path = sys.executable

eek. We use sys.executable for spawning flow run processes too. We may want to have a special run_python_process utility that handles this?

rpeden commented 1 year ago

@madkinsz Yeah, seems like a maybe good opportunity to extract this into a reusable function in prefect.utilities.processutils?

adnanmig commented 1 year ago

This issue affects the

if os.name == "nt":
    python_path = f'"{sys.executable}"'
else:
    python_path = sys.executable

eek. We use sys.executable for spawning flow run processes too. We may want to have a special run_python_process utility that handles this?

Yes, there is an issue there as well image