Docker image missing missing dependency 'psycopg2'

dylanmcreynolds commented 2 years ago

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Prefect documentation for this issue.
[X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Using a 2.20 docker image, I am unable to startup using postgres instead of sqlite. The documentation implies that all I need to do is switch the environment variable for the connection url.

Perhaps this is just a documentation isssue, and I need to add the

Reproduction

Using the docker image `prefecthq/prefect:2.2.0-python3.10`

I set an environment variable `PREFECT_ORION_DATABASE_CONNECTION_URL=postgresql://<user>:<pass>@db:5432/prefect`

The orion server fails to start, complaining of the missing package `psycopg2`

Error

Attaching to prefect_server
prefect_server  | 
prefect_server  |  ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
prefect_server  | | _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
prefect_server  | |  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
prefect_server  | |_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|
prefect_server  | 
prefect_server  | Configure Prefect to communicate with the server with:
prefect_server  | 
prefect_server  |     prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
prefect_server  | 
prefect_server  | View the API reference documentation at http://0.0.0.0:4200/docs
prefect_server  | 
prefect_server  | Check out the dashboard at http://0.0.0.0:4200
prefect_server  | 
prefect_server  | 
prefect_server  | 
prefect_server  | INFO:     Started server process [13]
prefect_server  | INFO:     Waiting for application startup.
prefect_server  | ERROR:    Traceback (most recent call last):
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 635, in lifespan
prefect_server  |     async with self.lifespan_context(app):
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 530, in __aenter__
prefect_server  |     await self._router.startup()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 612, in startup
prefect_server  |     await handler()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/api/server.py", line 271, in run_migrations
prefect_server  |     await db.create_db()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 53, in create_db
prefect_server  |     await self.run_migrations_upgrade()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 61, in run_migrations_upgrade
prefect_server  |     await run_sync_in_worker_thread(alembic_upgrade)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
prefect_server  |     return await anyio.to_thread.run_sync(call, cancellable=True)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
prefect_server  |     return await get_asynclib().run_sync_in_worker_thread(
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
prefect_server  |     return await future
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
prefect_server  |     result = context.run(func, *args)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
prefect_server  |     alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 322, in upgrade
prefect_server  |     script.run_env()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
prefect_server  |     util.load_python_file(self.dir, "env.py")
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
prefect_server  |     module = load_module_py(module_id, path)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
prefect_server  |     spec.loader.exec_module(module)  # type: ignore
prefect_server  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
prefect_server  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 147, in <module>
prefect_server  |     apply_migrations()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 189, in wrapper
prefect_server  |     return run_async_from_worker_thread(async_fn, *args, **kwargs)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 136, in run_async_from_worker_thread
prefect_server  |     return anyio.from_thread.run(call)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
prefect_server  |     return asynclib.run_async_from_thread(func, *args)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
prefect_server  |     return f.result()
prefect_server  |   File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
prefect_server  |     return self.__get_result()
prefect_server  |   File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
prefect_server  |     raise self._exception
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/migrations/env.py", line 137, in apply_migrations
prefect_server  |     engine = await db_interface.engine()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/interface.py", line 71, in engine
prefect_server  |     engine = await self.database_config.engine()
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/prefect/orion/database/configurations.py", line 112, in engine
prefect_server  |     engine = create_async_engine(self.connection_url, echo=self.echo, **kwargs)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/ext/asyncio/engine.py", line 43, in create_async_engine
prefect_server  |     sync_engine = _create_engine(*arg, **kw)
prefect_server  |   File "<string>", line 2, in create_engine
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned
prefect_server  |     return fn(*args, **kwargs)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 548, in create_engine
prefect_server  |     dbapi = dialect_cls.dbapi(**dbapi_args)
prefect_server  |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 811, in dbapi
prefect_server  |     import psycopg2
prefect_server  | ModuleNotFoundError: No module named 'psycopg2'
prefect_server  | 
prefect_server  | ERROR:    Application startup failed. Exiting.
prefect_server  | Orion stopped!
prefect_server exited with code 0

Versions


Version:             2.2.0
API version:         0.8.0
Python version:      3.10.6
Git commit:          e3651362
Built:               Tue, Aug 23, 2022 2:18 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.34.1

Additional context

I tried adding the EXTRA_PIP_PACKAGES=psycopg2` environment variable, but pip was unable to install psycopg2:

prefect_server  | WARNING: Discarding https://files.pythonhosted.org/packages/c3/f3/5519551f02ae70fc51f4e608e7b44d59a408fe3264fec4afeea37b8ea317/psycopg2-2.5.2.tar.gz#sha256=50382d8703a16c3c615f328c2676f002787095925c99f0afb125a59962a5bd2e (from https://pypi.org/simple/psycopg2/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
prefect_server  |   Downloading psycopg2-2.5.1.tar.gz (684 kB)
prefect_server  |   Preparing metadata (setup.py): started
prefect_server  |   Preparing metadata (setup.py): finished with status 'error'
prefect_server  |   ERROR: Command errored out with exit status 1:
prefect_server  |    command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6whfib0e/psycopg2_584b51d55c0b416fa2334eecdcc28530/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6whfib0e/psycopg2_584b51d55c0b416fa2334eecdcc28530/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-2qryf1pj
prefect_server  |        cwd: /tmp/pip-install-6whfib0e/psycopg2_584b51d55c0b416fa2334eecdcc28530/
prefect_server  |   Complete output (17 lines):
prefect_server  |   running egg_info
prefect_server  |   creating /tmp/pip-pip-egg-info-2qryf1pj/psycopg2.egg-info
prefect_server  |   writing /tmp/pip-pip-egg-info-2qryf1pj/psycopg2.egg-info/PKG-INFO
prefect_server  |   writing dependency_links to /tmp/pip-pip-egg-info-2qryf1pj/psycopg2.egg-info/dependency_links.txt
prefect_server  |   writing top-level names to /tmp/pip-pip-egg-info-2qryf1pj/psycopg2.egg-info/top_level.txt
prefect_server  |   writing manifest file '/tmp/pip-pip-egg-info-2qryf1pj/psycopg2.egg-info/SOURCES.txt'
prefect_server  |   /usr/local/lib/python3.10/site-packages/setuptools/command/egg_info.py:632: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
prefect_server  |   Please extend command classes from setuptools instead of distutils.
prefect_server  |     warnings.warn(
prefect_server  |   Error: pg_config executable not found.
prefect_server  |   
prefect_server  |   Please add the directory containing pg_config to the PATH
prefect_server  |   or specify the full executable path with the option:
prefect_server  |   
prefect_server  |       python setup.py build_ext --pg-config /path/to/pg_config build ...
prefect_server  |   
prefect_server  |   or with the pg_config option in 'setup.cfg'.

zanieb commented 2 years ago

Hi! It looks like your connection string is wrong. We do not support psycopg2 and instead use asyncpg which requires postgresql+asyncpg:// as your connection URL prefix instead.

@tpdorsey we'll want to make sure to highlight this in a future tutorial on using postgres.

zanieb commented 2 years ago

Note this is covered in the documentation for the connection setting at https://github.com/PrefectHQ/prefect/blob/main/src/prefect/settings.py#L413-L416 but it's missing from the website for some reason. Tracking issue to resolve that at https://github.com/PrefectHQ/prefect/issues/6549.

dylanmcreynolds commented 2 years ago

Yep, that solved it. May thanks!

PrefectHQ / prefect