Internal error after fresh install on bookworm

[truenicoco]

Impacted versions

• Distribution: Debian 12
• Codename: Bookworm
• Arch: arm64
• Database: PostgreSQL

Steps to reproduce

Launch ./run.py example.com, wait, it looks good… But then opening mail.example.com in a browser shows a 500 error. Inspecting /var/log/uwsgi/app/modoboa_instance.log shows:

x.x.x.x:443 [pid: 290485|app: -1|req: -1/2] x.x.x.x () {38 vars in 523 bytes} [Wed Sep 20 15:57:16 2023] GET / => generated 21 bytes in 9 msecs (HTTP/1.1 500) 2 headers in 83 bytes (1 switches on core 0)
Traceback (most recent call last):
  File "/srv/modoboa/instance/instance/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/apps/registry.py", line 83, in populate
    raise RuntimeError("populate() isn't reentrant")
RuntimeError: populate() isn't reentrant
Wed Sep 20 15:58:29 2023 - unable to load app 0 (mountpoint='mail.example.com|') (callable not found or import error)
Wed Sep 20 15:58:29 2023 - --- no python application found, check your startup logs for errors ---
mail.example.com [pid: 290486|app: -1|req: -1/3] 82.66.83.101 () {56 vars in 850 bytes} [Wed Sep 20 15:58:29 2023] GET / => generated 21 bytes in 6 msecs (HTTP/2.0 500) 2 headers in 83 bytes (0 switches on core 0)

Following the advice here, I commented the line in 83 in /srv/modoboa/env/lib/python3.11/site-packages/django/apps/registry.py which revealed:

Traceback (most recent call last):
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/db/backends/postgresql/psycopg_any.py", line 5, in <module>
    from psycopg import ClientCursor, IsolationLevel, adapt, adapters, errors, sql
ModuleNotFoundError: No module named 'psycopg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/modoboa/instance/instance/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/apps/registry.py", line 92, in populate
    app_config = AppConfig.create(entry)
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/apps/config.py", line 123, in create
    mod = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/modoboa/dmarc/apps.py", line 5, in <module>
    from modoboa.dmarc.forms import load_settings
  File "/srv/modoboa/env/lib/python3.11/site-packages/modoboa/dmarc/forms.py", line 9, in <module>
    from modoboa.dmarc.api.v2 import serializers
  File "/srv/modoboa/env/lib/python3.11/site-packages/modoboa/dmarc/api/v2/serializers.py", line 3, in <module>
    from rest_framework import serializers
  File "/srv/modoboa/env/lib/python3.11/site-packages/rest_framework/serializers.py", line 27, in <module>
    from rest_framework.compat import postgres_fields
  File "/srv/modoboa/env/lib/python3.11/site-packages/rest_framework/compat.py", line 26, in <module>
    from django.contrib.postgres import fields as postgres_fields
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/contrib/postgres/fields/__init__.py", line 1, in <module>
    from .array import *  # NOQA
    ^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/contrib/postgres/fields/array.py", line 4, in <module>
    from django.contrib.postgres.forms import SimpleArrayField
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/contrib/postgres/forms/__init__.py", line 3, in <module>
    from .ranges import *  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/contrib/postgres/forms/ranges.py", line 3, in <module>
    from django.db.backends.postgresql.psycopg_any import (
  File "/srv/modoboa/env/lib/python3.11/site-packages/django/db/backends/postgresql/psycopg_any.py", line 77, in <module>
    from psycopg2 import errors, extensions, sql  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/modoboa/env/lib/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
SystemError: initialization of _psycopg raised unreported exception

I then attempted /srv/modoboa/env/bin/pip install psycopg, then systemctl restart uwsgi and now the uwsgi logs look OK, but in the browser I just see "Sorry, an internal error occured" (in French).

In the end, I uninstalled postgresql, installed sqlite3 and relaunched the process, which seemed to work.

[tonioo]

Can you paste a full trace of the installation using the --debug opton?

[truenicoco]

Apologies, I didn't see your reply. I don't really want to mess everything up now that I got it working. However, I can tell that this was on a vanilla bookworm-preinstalled Hetzner VPS, so it might be easy to reproduce…

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ultr41337h4xor]

I have an idea as to why this might be happening @truenicoco @tonioo

I've updated my server to Bookworm from Bullseye only yesterday and the only problem I had is all my python-dependent hosted applications started giving me this same error. I've managed to fix 2 out of 3 (the final one being Modoboa), after doing some research and finding out Python as a project has some issues when system administration is concerned. For the 2 apps I fixed, the problem was their virtual environments being broken/not being recognized (when I tried to install pip or other python plugins, the error I was getting is "externally-managed-environment"). So what I did was run the commands to create the virtual environment again and, in one case, changed the command for installing plugins from "python" to "python3.11". Pretty damn annoying change for Python if you ask me.

Still didn't get around to fixing Modoboa, as it seems a bit more complicated when not doing it step-by-step, but by installer. My best guess is some of the install scripts have to be edited to change lines concerning python and then sudo ./run.py --upgrade example.com would fix it (recreate the virtual environment). I'm going to try to figure it out soon and I'll keep you posted if I do manage to fix it.

[ultr41337h4xor]

Finally got around to it today. There was no need for any editing of the installation scripts even. All I did was this:

1. sudo -u modoboa -i bash (log in as the modoboa user)

2. python3.11 -m venv ./env (create a new virtual environment with the updated version of Python)

3. exit (back to your standard user)

4. cd .../modoboa-installer (replace "..." with the path to where you cloned the modoboa-installer repository)

5. git pull (update the installer)

6. sudo ./run.py --upgrade example.com (replace "example.com" with your domain)

After the update completes, your Modoboa installation should now be up and running again!