search

funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
298 stars 46 forks source link

BUG: postgres ModuleNotFoundError: No module named 'psycopg2' #384

Closed spirillen closed 1 month ago

spirillen commented 1 month ago

Description

Tried to setup to the postgresql database, but it fails with the following error in sqlachedemy

pyfunceble --ci --database-type postgresql -d mypdns.org
Traceback (most recent call last):
  File "/home/$USER/Projects/github/funilrys/pyfunceble/venv/bin/pyfunceble", line 33, in <module>
    sys.exit(load_entry_point('PyFunceble-dev', 'console_scripts', 'pyfunceble')())
  File "/home/$USER/Projects/github/funilrys/pyfunceble/PyFunceble/cli/entry_points/pyfunceble/cli.py", line 1423, in tool
    SystemIntegrator(args).start()
  File "/home/$USER/Projects/github/funilrys/pyfunceble/PyFunceble/cli/system/base.py", line 90, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/Projects/github/funilrys/pyfunceble/PyFunceble/cli/system/integrator.py", line 270, in start
    PyFunceble.cli.factory.DBSession.init_db_sessions()
  File "/home/$USER/Projects/github/funilrys/pyfunceble/PyFunceble/database/session.py", line 121, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/Projects/github/funilrys/pyfunceble/PyFunceble/database/session.py", line 107, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/Projects/github/funilrys/pyfunceble/PyFunceble/database/session.py", line 144, in init_db_sessions
    PyFunceble.sessions.DB_ENGINE = sqlalchemy.create_engine(
  File "<string>", line 2, in create_engine
  File "/home/$USER/Projects/github/funilrys/pyfunceble/venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  File "/home/$USER/Projects/github/funilrys/pyfunceble/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 599, in create_engine
    dbapi = dbapi_meth(**dbapi_args)
  File "/home/$USER/Projects/github/funilrys/pyfunceble/venv/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

Configuration

No overwrite

PYFUNCEBLE_DB_CHARSET=utf8mb4
PYFUNCEBLE_DB_HOST=/var/run/postgresql
PYFUNCEBLE_DB_NAME=$USERNAME
PYFUNCEBLE_DB_PASSWORD='PASSWORD'
PYFUNCEBLE_DB_PORT=$PORT
PYFUNCEBLE_DB_USERNAME=$USERNAME
PYFUNCEBLE_OUTPUT_LOCATION=/home/$USER/Projects/pyfunceble
PYFUNCEBLE_PLATFORM_API_URL=https://api.beta.dead-hosts.com
PYFUNCEBLE_PLATFORM_API_TOKEN="a_TOKEN"
PYFUNCEBLE_AUTO_CONFIGURATION=yes
PYFUNCEBLE_PLATFORM_CHECKER_EXCLUDE=reputation

Reproduction

  1. clone git clone git@github.com:funilrys/PyFunceble.git pyfunceble
  2. Activate venv virtualenv -p python3 venv
  3. Install deps pip3 install -e .
  4. Setup the .pyfunceble-env
  5. Run the test line pyfunceble --ci --database-type postgresql -d mypdns.org

Expected behavior

Screenshots

Versions

OS: 22.04 + 24.04

Python Version: 3.11 + 3.12

PyFunceble Version: 4.2.28a1.dev (Blue Duckling: Tulip)

Additional context

Connection is tested via remote app + local access

spirillen commented 1 month ago

The problem is persistent in 4.2.29.dev

line 690, in import_dbapi
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

pyfunceble 4.2.29.dev (Blue Duckling: Tulip)

funilrys commented 1 month ago

That's weird. I couldn't reproduce that anymore with the patch:

rm -fr venv/ ; virtualenv venv; . venv/bin/activate; pip install -e .; python -c 'import psycopg2'

Can you please try:

 rm -fr venv/ ; virtualenv venv; . venv/bin/activate; pip install --upgrade --force -e .; python -c 'import psycopg2'
spirillen commented 1 month ago

I can, but then we would be cheating, from the docs 😉 https://pyfunceble.readthedocs.io/en/dev/installation/index.html#for-development

spirillen commented 1 month ago

image

image

pyfunceble --database-type postgresql -d example.org
Traceback (most recent call last):
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
             ^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/cli/entry_points/pyfunceble/cli.py", line 1423, in tool
    SystemIntegrator(args).start()
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/cli/system/base.py", line 90, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/cli/system/integrator.py", line 270, in start
    PyFunceble.cli.factory.DBSession.init_db_sessions()
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/database/session.py", line 121, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/database/session.py", line 107, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/database/session.py", line 144, in init_db_sessions
    PyFunceble.sessions.DB_ENGINE = sqlalchemy.create_engine(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 599, in create_engine
    dbapi = dbapi_meth(**dbapi_args)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'
spirillen commented 1 month ago 
rm -fr venv/ ; virtualenv venv; . venv/bin/activate; pip install -e .; python -c 'import psycopg2'
Building wheels for collected packages: PyFunceble-dev
  Building editable for PyFunceble-dev (pyproject.toml) ... done
  Created wheel for PyFunceble-dev: filename=PyFunceble_dev-4.2.29-0.editable-py3-none-any.whl size=15339 sha256=9c61cb4b1ff13587ec427fb4bbd282c0e05a8a255c38d438aeda2800f522c374
  Stored in directory: /tmp/pip-ephem-wheel-cache-g_crgd6z/wheels/92/ab/61/f6fe21aebce131d26f6ea6a8350587aa81f4e4375ae04265ca
Successfully built PyFunceble-dev
Installing collected packages: urllib3, typing-extensions, toml, sniffio, shtab, setuptools, ruamel.yaml.clib, PyYAML, python-dotenv, python-box, PySocks, PyMySQL, pycparser, packaging, msgpack, MarkupSafe, inflection, idna, hyperframe, hpack, h11, greenlet, dnspython, colorama, charset-normalizer, certifi, SQLAlchemy, ruamel.yaml, requests, Mako, httpcore, h2, domain2idna, cffi, anyio, httpx, cryptography, alembic, PyFunceble-dev
Successfully installed Mako-1.3.5 MarkupSafe-2.1.5 PyFunceble-dev-4.2.29 PyMySQL-1.1.1 PySocks-1.7.1 PyYAML-6.0.2 SQLAlchemy-2.0.35 alembic-1.13.3 anyio-4.6.0 certifi-2024.8.30 cffi-1.17.1 charset-normalizer-3.3.2 colorama-0.4.6 cryptography-42.0.8 dnspython-2.6.1 domain2idna-1.12.0 greenlet-3.1.1 h11-0.14.0 h2-4.1.0 hpack-4.0.0 httpcore-1.0.5 httpx-0.27.2 hyperframe-6.0.1 idna-3.10 inflection-0.5.1 msgpack-1.1.0 packaging-24.1 pycparser-2.22 python-box-6.0.2 python-dotenv-1.0.1 requests-2.32.3 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 setuptools-75.1.0 shtab-1.7.1 sniffio-1.3.1 toml-0.10.2 typing-extensions-4.12.2 urllib3-2.2.3
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'psycopg2'
pyfunceble --database-type postgresql -d example.org
Traceback (most recent call last):
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
             ^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/cli/entry_points/pyfunceble/cli.py", line 1423, in tool
    SystemIntegrator(args).start()
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/cli/system/base.py", line 90, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/cli/system/integrator.py", line 270, in start
    PyFunceble.cli.factory.DBSession.init_db_sessions()
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/database/session.py", line 121, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/database/session.py", line 107, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/PyFunceble/database/session.py", line 144, in init_db_sessions
    PyFunceble.sessions.DB_ENGINE = sqlalchemy.create_engine(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 599, in create_engine
    dbapi = dbapi_meth(**dbapi_args)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spirillen/Projects/github/funilrys/PyFunceble/venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'
spirillen commented 1 month ago

Thinks I spotted the problem, from one of my virtual machines

WARNING: sqlalchemy 2.0.35 does not provide the extra 'postgresql-psycopg2binary'

ython3.11 -m pip install -U -I pyfunceble-dev
Collecting pyfunceble-dev
  Downloading PyFunceble_dev-4.2.29-py3-none-any.whl (517 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 517.4/517.4 kB 4.4 MB/s eta 0:00:00
Collecting domain2idna~=1.12.0
  Using cached domain2idna-1.12.0-py3-none-any.whl (10 kB)
Collecting dnspython[doh]~=2.6.0
  Using cached dnspython-2.6.1-py3-none-any.whl (307 kB)
Collecting requests[socks]<3
  Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Collecting PyYAML
  Using cached PyYAML-6.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (762 kB)
Collecting inflection
  Using cached inflection-0.5.1-py2.py3-none-any.whl (9.5 kB)
Collecting python-dotenv
  Using cached python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Collecting shtab
  Using cached shtab-1.7.1-py3-none-any.whl (14 kB)
Collecting SQLAlchemy[postgresql-psycopg2binary]~=2.0
  Using cached SQLAlchemy-2.0.35-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting python-box[all]~=6.0.0
  Using cached python_box-6.0.2-py3-none-any.whl (26 kB)
Collecting setuptools>=65.5.1
  Using cached setuptools-75.1.0-py3-none-any.whl (1.2 MB)
Collecting cryptography~=42.0
  Using cached cryptography-42.0.8-cp39-abi3-manylinux_2_28_x86_64.whl (3.9 MB)
Collecting PyMySQL
  Using cached PyMySQL-1.1.1-py3-none-any.whl (44 kB)
Collecting packaging
  Using cached packaging-24.1-py3-none-any.whl (53 kB)
Collecting alembic
  Using cached alembic-1.13.3-py3-none-any.whl (233 kB)
Collecting colorama
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting cffi>=1.12
  Using cached cffi-1.17.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (467 kB)
Collecting h2>=4.1.0
  Using cached h2-4.1.0-py3-none-any.whl (57 kB)
Collecting httpcore>=1.0.0
  Using cached httpcore-1.0.5-py3-none-any.whl (77 kB)
Collecting httpx>=0.26.0
  Using cached httpx-0.27.2-py3-none-any.whl (76 kB)
Collecting ruamel.yaml>=0.17
  Using cached ruamel.yaml-0.18.6-py3-none-any.whl (117 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting msgpack
  Using cached msgpack-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (403 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.10-py3-none-any.whl (70 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.2.3-py3-none-any.whl (126 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
Collecting PySocks!=1.5.7,>=1.5.6
  Using cached PySocks-1.7.1-py3-none-any.whl (16 kB)
WARNING: sqlalchemy 2.0.35 does not provide the extra 'postgresql-psycopg2binary'
Collecting typing-extensions>=4.6.0
  Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Collecting greenlet!=0.4.17
  Using cached greenlet-3.1.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (602 kB)
Collecting Mako
  Using cached Mako-1.3.5-py3-none-any.whl (78 kB)
Collecting pycparser
  Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Collecting hyperframe<7,>=6.0
  Using cached hyperframe-6.0.1-py3-none-any.whl (12 kB)
Collecting hpack<5,>=4.0
  Using cached hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting h11<0.15,>=0.13
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting anyio
  Using cached anyio-4.6.0-py3-none-any.whl (89 kB)
Collecting sniffio
  Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)
Collecting ruamel.yaml.clib>=0.2.7
  Using cached ruamel.yaml.clib-0.2.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (544 kB)
Collecting MarkupSafe>=0.9.2
  Using cached MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Installing collected packages: urllib3, typing-extensions, toml, sniffio, shtab, setuptools, ruamel.yaml.clib, PyYAML, python-dotenv, python-box, PySocks, PyMySQL, pycparser, packaging, msgpack, MarkupSafe, inflection, idna, hyperframe, hpack, h11, greenlet, dnspython, colorama, charset-normalizer, certifi, SQLAlchemy, ruamel.yaml, requests, Mako, httpcore, h2, domain2idna, cffi, anyio, httpx, cryptography, alembic, pyfunceble-dev
Successfully installed Mako-1.3.5 MarkupSafe-2.1.5 PyMySQL-1.1.1 PySocks-1.7.1 PyYAML-6.0.2 SQLAlchemy-2.0.35 alembic-1.13.3 anyio-4.6.0 certifi-2024.8.30 cffi-1.17.1 charset-normalizer-3.3.2 colorama-0.4.6 cryptography-42.0.8 dnspython-2.6.1 domain2idna-1.12.0 greenlet-3.1.1 h11-0.14.0 h2-4.1.0 hpack-4.0.0 httpcore-1.0.5 httpx-0.27.2 hyperframe-6.0.1 idna-3.10 inflection-0.5.1 msgpack-1.1.0 packaging-24.1 pycparser-2.22 pyfunceble-dev-4.2.29 python-box-6.0.2 python-dotenv-1.0.1 requests-2.32.3 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 setuptools-75.1.0 shtab-1.7.1 sniffio-1.3.1 toml-0.10.2 typing-extensions-4.12.2 urllib3-2.2.3
funilrys commented 1 month ago

I guess this is a typo: https://github.com/sqlalchemy/sqlalchemy/blob/ca501d9d38d50dc4eda52c82ebc9c51766bc141b/setup.cfg#L67

Fix on the way.