ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py)

The playbook finishes successfully, with no failures. However when I try to sign up as a new user, I get the error "Server error - An error occurred while processing your request." (https://ibb.co/PhHt5hK), and the mailman-web.log file shows the following:

[uwsgi-daemons] respawning "python3 manage.py qcluster" (uid: 33 gid: 33) Traceback (most recent call last): File "manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 364, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 206, in fetch_command klass = load_command_class(app_name, subcommand) File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 40, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/django_q/management/commands/qcluster.py", line 4, in from django_q.cluster import Cluster File "/usr/lib/python3/dist-packages/django_q/cluster.py", line 24, in from django_q import tasks File "/usr/lib/python3/dist-packages/django_q/tasks.py", line 12, in from django_q.cluster import worker, monitor ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py)

For reference, this was our installation process from a Digital Ocean instance:

adduser ansible usermod -aG sudo ansible sudo apt update sudo apt -y upgrade sudo dpkg-reconfigure tzdata sudo apt -y install git sudo apt -y install curl sudo apt -y install python3-pip python3 --version [3.7.3] curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3 get-pip.py --user .bashrc export PATH="/home/ansible/.local/bin:$PATH"~/.bashrc python3 -V pip3 -V install python-apt pip3 install django

pip install ansible --user ansiblea nsible --version [2.10.4]

python3 -m pip install argcomplete activate-global-python-argcomplete --user su ansible

mkdir ~/.ansible

ansible-galaxy install ucr.mailman3 cd /home/ansible

sudo nano playbook.yaml

hosts: localhost become: true become_method: sudo vars: mailman3_domain: example.com # It will install and create a certificate using certbot, its nignx plugin and http validation $ mailman3_install_certbot: false # install and create certificate # secrets, please change them mailman3_secret_key: "change_secret key" mailman3_rest_api_pass: "change_api_pass" mailman3_archiver_key: "change_archiver_key" mailman3_db_pass: "change_db_pass" mailman3_web_db_pass: "change_web_db_pass" mailman3_web_admin_pass: "change_web_admin_pass" roles: - ucr.mailman3 cd /home ansible-playbook -K -c local -i localhost, playbook.yaml

From mail.err, looks like these files weren't created.

Jan 3 21:16:40 gpcmembers postfix/smtpd[10693]: error: open database /var/lib/mailman3/data/postfix_domains.db: No such file or directory Jan 3 21:16:40 gpcmembers postfix/smtpd[10693]: error: open database /var/lib/mailman3/data/postfix_lmtp.db: No such file or directory Jan 3 23:23:35 gpcmembers postfix/smtpd[13709]: error: open database /var/lib/mailman3/data/postfix_domains.db: No such file or directory Jan 3 23:23:35 gpcmembers postfix/smtpd[13709]: error: open database /var/lib/mailman3/data/postfix_lmtp.db: No such file or directory Jan 3 23:23:35 gpcmembers postfix/trivial-rewrite[13713]: error: open database /var/lib/mailman3/data/postfix_domains.db: No such file or directory Jan 3 23:23:35 gpcmembers postfix/trivial-rewrite[13713]: error: open database /var/lib/mailman3/data/postfix_lmtp.db: No such file or directory Jan 3 23:23:36 gpcmembers postfix/smtpd[13714]: error: open database /var/lib/mailman3/data/postfix_domains.db: No such file or directory Jan 3 23:23:36 gpcmembers postfix/smtpd[13714]: error: open database /var/lib/mailman3/data/postfix_lmtp.db: No such file or directory

Okay, solved it by creating the files manually. Not sure why the normal playbook didn't create them, but I followed these instructions: https://www.howtoforge.com/community/threads/solved-error-open-database-etc-mailman-virtual-mailman-db-no-such-file-or-directory.70757/

touch /var/lib/mailman3/data/postfix_domains postmap /var/lib/mailman3/data/postfix_domains

My mailman3 is now working, but I'll leave the problem open until you determine if a change is required in the collection.

Hi,

Apparently mailman aliases cmd is needed in order to create them. It will be added. Thanks

I spoke a bit too soon. After creating the database, it allows you to verify your email address and log in as the administrator. However, when you try to create the first list or first domain, you get "Server error. An error occurred while processing your request." https://ibb.co/y5nwP3q

from mailman-web.log ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py) [uwsgi-daemons] respawning "python3 manage.py qcluster" (uid: 33 gid: 33) Traceback (most recent call last): File "manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 364, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 206, in fetch_command klass = load_command_class(app_name, subcommand) File "/usr/lib/python3/dist-packages/django/core/management/init.py", line 40, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/django_q/management/commands/qcluster.py", line 4, in from django_q.cluster import Cluster File "/usr/lib/python3/dist-packages/django_q/cluster.py", line 24, in from django_q import tasks File "/usr/lib/python3/dist-packages/django_q/tasks.py", line 12, in from django_q.cluster import worker, monitor

in postgresql-11-main.log 2021-01-11 18:03:31.218 EST [5220] LOG: listening on IPv4 address "127.0.0.1", port 5432 2021-01-11 18:03:31.220 EST [5220] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-01-11 18:03:31.250 EST [5221] LOG: database system was shut down at 2021-01-11 18:03:29 EST 2021-01-11 18:03:31.258 EST [5220] LOG: database system is ready to accept connections 2021-01-11 18:03:31.798 EST [5228] [unknown]@[unknown] LOG: incomplete startup packet 2021-01-11 18:05:23.322 EST [8152] mailman3@template1 FATAL: password authentication failed for user "mailman3" 2021-01-11 18:05:23.322 EST [8152] mailman3@template1 DETAIL: Role "mailman3" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-11 18:05:23.327 EST [8153] mailman3@template1 FATAL: password authentication failed for user "mailman3" 2021-01-11 18:05:23.327 EST [8153] mailman3@template1 DETAIL: Role "mailman3" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-11 18:05:23.523 EST [8206] postgres@template1 ERROR: role "mailman3" does not exist 2021-01-11 18:05:23.523 EST [8206] postgres@template1 STATEMENT: ALTER USER "mailman3" 2021-01-11 18:05:23.884 EST [8316] postgres@mailman3 FATAL: database "mailman3" does not exist 2021-01-11 18:05:34.591 EST [8556] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:05:34.593 EST [8554] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:07:38.374 EST [9115] mailman3web@template1 FATAL: password authentication failed for user "mailman3web" 2021-01-11 18:07:38.374 EST [9115] mailman3web@template1 DETAIL: Role "mailman3web" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-11 18:07:38.382 EST [9116] mailman3web@template1 FATAL: password authentication failed for user "mailman3web" 2021-01-11 18:07:38.382 EST [9116] mailman3web@template1 DETAIL: Role "mailman3web" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-11 18:07:38.696 EST [9169] postgres@template1 ERROR: role "mailman3web" does not exist 2021-01-11 18:07:38.696 EST [9169] postgres@template1 STATEMENT: ALTER USER "mailman3web" 2021-01-11 18:07:39.144 EST [9279] postgres@mailman3web FATAL: database "mailman3web" does not exist 2021-01-11 18:07:45.922 EST [9437] mailman3web@mailman3web ERROR: relation "social_auth_usersocialauth" does not exist at character 15 2021-01-11 18:07:45.922 EST [9437] mailman3web@mailman3web STATEMENT: SELECT 1 from social_auth_usersocialauth 2021-01-11 18:08:21.141 EST [8658] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.249 EST [8662] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.325 EST [8660] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.334 EST [8709] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.341 EST [8689] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.355 EST [8656] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.360 EST [8657] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.380 EST [8659] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.383 EST [8642] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.397 EST [8654] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.448 EST [8663] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.479 EST [8653] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:21.506 EST [10322] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:25.997 EST [10326] mailman3web@mailman3web LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:08:26.004 EST [9617] mailman3web@mailman3web LOG: could not receive data from client: Connection reset by peer 2021-01-11 18:10:45.804 EST [5220] LOG: received fast shutdown request 2021-01-11 18:10:45.806 EST [5220] LOG: aborting any active transactions

In case it's helpful, this is how I currently prepare a digitalocean new Deb 10 build for ansible (in case I'm doing anything wrong):

adduser mailman usermod -aG sudo mailman

su mailman

sudo apt update

sudo apt -y install git sudo apt -y install curl sudo apt -y install python3-pip

add export PATH="$HOME/bin:$PATH" to .bashrc. ~/.bashrc

pip3 install django [possibly not needed here]

pip3 install ansible --user mailman

python3 -m pip install argcomplete activate-global-python-argcomplete --user

ansible-galaxy install ucr.mailman3

nano playbook.yaml

hosts: localhost become: true become_method: sudo vars: mailman3_domain: example.com # It will install and create a certificate using certbot, its nignx plugin and http validation $ mailman3_install_certbot: false # install and create certificate # secrets, please change them mailman3_secret_key: "change_secretkey" mailman3_rest_api_pass: "change_api_pass" mailman3_archiver_key: "change_archiver_key" mailman3_db_pass: "change_db_pass" mailman3_web_db_pass: "change_web_db_pass" mailman3_web_admin_pass: "change_web_admin_pass" roles: - ucr.mailman3 ansible-playbook -K -c local -i localhost playbook.yaml [enter sudo password]

sudo touch /var/lib/mailman3/data/postfix_domains sudo postmap /var/lib/mailman3/data/postfix_domains sudo touch /var/lib/mailman3/data/postfix_lmtp sudo postmap /var/lib/mailman3/data/postfix_lmtp

Hi,

What is the output of (as root)?:

su - postgres -c "psql -c '\list'"
su - postgres -c "psql -c '\du'"

Databases and users are there. Why is it say there aren't?!

We'll try to replicate this. Thanks

No worries, let me know if you need anything else.

A bit more information, I received an email warning. One thing to look at, which I saw in other contexts, was that it's sometimes sensitive to the difference between when you use mailman vs. mailman@localhost.

Internal Server Error: /postorius/domains/new/

NoReverseMatch at /postorius/domains/new/ Reverse for 'sites_site_changelist' not found. 'sites_site_changelist' is not a valid view function or pattern name.

Request Method: GET Request URL: https://www.gpcmembers.ca/postorius/domains/new/ Django Version: 1.11.29 Python Executable: /usr/bin/uwsgi-core Python Version: 3.7.3 Python Path: ['.', '', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages'] Server time: Tue, 12 Jan 2021 21:58:54 +0000 Installed Applications: ('hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount', 'django_mailman3.lib.auth.fedora') Installed Middleware: ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')

Traceback:

File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py" in inner

response = get_response(request)

File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response

response = self.process_exception_by_middleware(e, request)

File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response

response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/lib/python3/dist-packages/django/contrib/auth/decorators.py" in _wrapped_view

return view_func(request, *args, **kwargs)

File "/usr/lib/python3/dist-packages/postorius/auth/decorators.py" in wrapper

return fn(*args, **kwargs)

File "/usr/lib/python3/dist-packages/postorius/views/domain.py" in domain_new

return render(request, 'postorius/domain/new.html', {'form': form})

File "/usr/lib/python3/dist-packages/django/shortcuts.py" in render

content = loader.render_to_string(template_name, context, request, using=using)

File "/usr/lib/python3/dist-packages/django/template/loader.py" in render_to_string

return template.render(context, request)

File "/usr/lib/python3/dist-packages/django/template/backends/django.py" in render

return self.template.render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render

return self._render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in _render

return self.nodelist.render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render

bit = node.render_annotated(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render_annotated

return self.render(context)

File "/usr/lib/python3/dist-packages/django/template/loader_tags.py" in render

return compiled_parent._render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in _render

return self.nodelist.render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render

bit = node.render_annotated(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render_annotated

return self.render(context)

File "/usr/lib/python3/dist-packages/django/template/loader_tags.py" in render

result = block.nodelist.render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render

bit = node.render_annotated(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render_annotated

return self.render(context)

File "/usr/lib/python3/dist-packages/django/template/library.py" in render

return t.render(new_context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render

return self._render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in _render

return self.nodelist.render(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render

bit = node.render_annotated(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in render_annotated

return self.render(context)

File "/usr/lib/python3/dist-packages/django/template/defaulttags.py" in render

nodelist.append(node.render_annotated(context))

File "/usr/lib/python3/dist-packages/django/template/base.py" in render_annotated

return self.render(context)

File "/usr/lib/python3/dist-packages/django/template/defaulttags.py" in render

match = condition.eval(context)

File "/usr/lib/python3/dist-packages/django/template/defaulttags.py" in eval

return self.value.resolve(context, ignore_failures=True)

File "/usr/lib/python3/dist-packages/django/template/base.py" in resolve

obj = self.var.resolve(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in resolve

value = self._resolve_lookup(context)

File "/usr/lib/python3/dist-packages/django/template/base.py" in _resolve_lookup

current = current()

File "/usr/lib/python3/dist-packages/postorius/forms/domain_forms.py" in _get_web_host_help

) % reverse("admin:sites_site_changelist"))

File "/usr/lib/python3/dist-packages/django/urls/base.py" in reverse

return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))

File "/usr/lib/python3/dist-packages/django/urls/resolvers.py" in _reverse_with_prefix

raise NoReverseMatch(msg)

Exception Type: NoReverseMatch at /postorius/domains/new/ Exception Value: Reverse for 'sites_site_changelist' not found. 'sites_site_changelist' is not a valid view function or pattern name. Request information: USER: admin

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: csrftoken = 'N1Sd9LpZQW8QSCuSaPL81M5AcJbr2PAgUALOffrATJn8yCG9IxV6yLLnvwL74OJs' sessionid = 'yli5r1f6ubxibhal9fmg6rypi3in0lgb'

META: CONTENT_LENGTH = '' CONTENT_TYPE = '' CSRF_COOKIE = 'N1Sd9LpZQW8QSCuSaPL81M5AcJbr2PAgUALOffrATJn8yCG9IxV6yLLnvwL74OJs' CSRF_COOKIE_USED = True DOCUMENT_ROOT = '/usr/share/nginx/html' HTTPS = 'on' HTTP_ACCEPT = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' HTTP_ACCEPT_ENCODING = 'gzip, deflate, br' HTTP_ACCEPT_LANGUAGE = 'en-CA,en-US;q=0.7,en;q=0.3' HTTP_CONNECTION = 'keep-alive' HTTP_COOKIE = 'csrftoken=N1Sd9LpZQW8QSCuSaPL81M5AcJbr2PAgUALOffrATJn8yCG9IxV6yLLnvwL74OJs; sessionid=yli5r1f6ubxibhal9fmg6rypi3in0lgb' HTTP_DNT = '1' HTTP_HOST = 'www.gpcmembers.ca' HTTP_REFERER = 'https://www.gpcmembers.ca/postorius/lists/?all-lists' HTTP_SEC_GPC = '1' HTTP_UPGRADE_INSECURE_REQUESTS = '1' HTTP_USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0' PATH_INFO = '/postorius/domains/new/' QUERY_STRING = '' REMOTE_ADDR = '99.233.222.142' REMOTE_PORT = '44300' REQUEST_METHOD = 'GET' REQUEST_SCHEME = 'https' REQUEST_URI = '/postorius/domains/new/' SCRIPT_NAME = '' SERVER_NAME = 'www.gpcmembers.ca' SERVER_PORT = '443' SERVER_PROTOCOL = 'HTTP/1.1' uwsgi.core = 1 uwsgi.node = b'lists' uwsgi.version = b'2.0.18-debian' wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'> wsgi.file_wrapper = '' wsgi.input = <uwsgi._Input object at 0x7f461b825900> wsgi.multiprocess = False wsgi.multithread = True wsgi.run_once = False wsgi.url_scheme = 'https' wsgi.version = '(1, 0)'

Settings: Using settings module settings ABSOLUTE_URL_OVERRIDES = {} ACCOUNT_AUTHENTICATION_METHOD = 'username_email' ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'https' ACCOUNT_EMAIL_REQUIRED = True ACCOUNT_EMAIL_VERIFICATION = 'mandatory' ACCOUNT_UNIQUE_EMAIL = True ADMINS = "(('Mailman Suite Admin', 'myemailaddress'),)" ALLOWED_HOSTS = ['*'] APPEND_SLASH = True AUTHENTICATION_BACKENDS = "('django.contrib.auth.backends.ModelBackend', 'allauth.account.auth_backends.AuthenticationBackend')" AUTH_PASSWORD_VALIDATORS = '****' AUTH_USER_MODEL = 'auth.User' BASE_DIR = '/usr/share/mailman3-web' CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} CACHE_MIDDLEWARE_ALIAS = 'default' CACHE_MIDDLEWARE_KEY_PREFIX = '****' CACHE_MIDDLEWARE_SECONDS = 600 COMPRESS_CACHEABLE_PRECOMPILERS = '()' COMPRESS_CACHE_BACKEND = 'default' COMPRESS_CACHE_KEY_FUNCTION = '****' COMPRESS_CLEAN_CSS_ARGUMENTS = '' COMPRESS_CLEAN_CSS_BINARY = 'cleancss' COMPRESS_CLOSURE_COMPILER_ARGUMENTS = '' COMPRESS_CLOSURE_COMPILER_BINARY = 'java -jar compiler.jar' COMPRESS_CSS_COMPRESSOR = 'compressor.css.CssCompressor' COMPRESS_CSS_FILTERS = ['compressor.filters.css_default.CssAbsoluteFilter'] COMPRESS_CSS_HASHING_METHOD = 'mtime' COMPRESS_DATA_URI_MAX_SIZE = 1024 COMPRESS_DEBUG_TOGGLE = None COMPRESS_ENABLED = True COMPRESS_JINJA2_GET_ENVIRONMENT = <function CompressorConf.JINJA2_GET_ENVIRONMENT at 0x7f4620ed52f0> COMPRESS_JS_COMPRESSOR = 'compressor.js.JsCompressor' COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter'] COMPRESS_MINT_DELAY = 30 COMPRESS_MTIME_DELAY = 10 COMPRESS_OFFLINE = True COMPRESS_OFFLINE_CONTEXT = {'STATIC_URL': '/mailman3/static/'} COMPRESS_OFFLINE_MANIFEST = 'manifest.json' COMPRESS_OFFLINE_TIMEOUT = 31536000 COMPRESS_OUTPUT_DIR = 'CACHE' COMPRESS_PARSER = 'compressor.parser.AutoSelectParser' COMPRESS_PRECOMPILERS = "(('text/less', 'lessc {infile} {outfile}'), ('text/x-scss', 'sassc -t compressed {infile} {outfile}'), ('text/x-sass', 'sassc -t compressed {infile} {outfile}'))" COMPRESS_REBUILD_TIMEOUT = 2592000 COMPRESS_ROOT = '/var/lib/mailman3/web/static' COMPRESS_STORAGE = 'compressor.storage.CompressorFileStorage' COMPRESS_TEMPLATE_FILTER_CONTEXT = {'STATIC_URL': '/mailman3/static/'} COMPRESS_URL = '/mailman3/static/' COMPRESS_URL_PLACEHOLDER = '/compressor_url_placeholder/' COMPRESS_VERBOSE = False COMPRESS_YUGLIFY_BINARY = 'yuglify' COMPRESS_YUGLIFY_CSS_ARGUMENTS = '--terminal' COMPRESS_YUGLIFY_JS_ARGUMENTS = '--terminal' COMPRESS_YUI_BINARY = 'java -jar yuicompressor.jar' COMPRESS_YUI_CSS_ARGUMENTS = '' COMPRESS_YUI_JS_ARGUMENTS = '' CSRF_COOKIE_AGE = 31449600 CSRF_COOKIE_DOMAIN = None CSRF_COOKIE_HTTPONLY = False CSRF_COOKIE_NAME = 'csrftoken' CSRF_COOKIE_PATH = '/' CSRF_COOKIE_SECURE = False CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure' CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN' CSRF_TRUSTED_ORIGINS = [] CSRF_USE_SESSIONS = False DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'mailman3web', 'USER': 'mailman3web', 'PASSWORD': '****', 'HOST': 'localhost', 'PORT': '', 'OPTIONS': {}, 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'NAME': None, 'MIRROR': None}}} DATABASE_ROUTERS = [] DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440 DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000 DATETIME_FORMAT = 'N j, Y, P' DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y'] DATE_FORMAT = 'N j, Y' DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y'] DEBUG = False DEBUG_PROPAGATE_EXCEPTIONS = False DECIMAL_SEPARATOR = '.' DEFAULT_CHARSET = 'utf-8' DEFAULT_CONTENT_TYPE = 'text/html' DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter' DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' DEFAULT_FROM_EMAIL = 'postorius@localhost.local' DEFAULT_INDEX_TABLESPACE = '' DEFAULT_TABLESPACE = '' DISALLOWED_USER_AGENTS = [] EMAILNAME = 'localhost.local' EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_HOST_PASSWORD = '****' EMAIL_HOST_USER = '' EMAIL_PORT = 25 EMAIL_SSL_CERTFILE = None EMAIL_SSL_KEYFILE = '****' EMAIL_SUBJECT_PREFIX = '[Django] ' EMAIL_TIMEOUT = None EMAIL_USE_LOCALTIME = False EMAIL_USE_SSL = False EMAIL_USE_TLS = False FILE_CHARSET = 'utf-8' FILE_UPLOAD_DIRECTORY_PERMISSIONS = None FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler'] FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440 FILE_UPLOAD_PERMISSIONS = None FILE_UPLOAD_TEMP_DIR = None FILTER_VHOST = False FIRST_DAY_OF_WEEK = 0 FIXTURE_DIRS = [] FORCE_SCRIPT_NAME = None FORMAT_MODULE_PATH = None FORM_RENDERER = 'django.forms.renderers.DjangoTemplates' HAYSTACK_CONNECTIONS = {'default': {'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine', 'PATH': '/var/lib/mailman3/web/fulltext_index'}} HOSTNAME = 'localhost.local' IGNORABLE_404_URLS = [] INSTALLED_APPS = "('hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount', 'django_mailman3.lib.auth.fedora')" INTERNAL_IPS = [] LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')] LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur'] LANGUAGE_CODE = 'en-us' LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_NAME = 'django_language' LANGUAGE_COOKIE_PATH = '/' LOCALE_PATHS = [] LOGGING = {'version': 1, 'disable_existing_loggers': False, 'filters': {'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'}}, 'handlers': {'mail_admins': {'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler'}, 'file': {'level': 'INFO', 'class': 'logging.handlers.RotatingFileHandler', 'filename': '/var/log/mailman3/web/mailman-web.log', 'formatter': 'verbose'}, 'console': {'class': 'logging.StreamHandler', 'formatter': 'simple'}}, 'loggers': {'django.request': {'handlers': ['mail_admins', 'file'], 'level': 'INFO', 'propagate': True}, 'django': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}, 'hyperkitty': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}, 'postorius': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}}, 'formatters': {'verbose': {'format': '%(levelname)s %(asctime)s %(process)d %(name)s %(message)s'}, 'simple': {'format': '%(levelname)s %(message)s'}}} LOGGING_CONFIG = 'logging.config.dictConfig' LOGIN_REDIRECT_URL = 'list_index' LOGIN_URL = 'account_login' LOGOUT_REDIRECT_URL = None LOGOUT_URL = 'account_logout' MAILMAN_ARCHIVER_FROM = "('127.0.0.1', '::1')" MAILMAN_ARCHIVER_KEY = '****' MAILMAN_REST_API_PASS = '****' MAILMAN_REST_API_URL = '****' MAILMAN_REST_API_USER = '****' MANAGERS = [] MEDIA_ROOT = '' MEDIA_URL = '' MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage' MESSAGE_TAGS = {40: 'danger'} MIDDLEWARE = "('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')" MIDDLEWARE_CLASSES = ['django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware'] MIGRATION_MODULES = {} MONTH_DAY_FORMAT = 'F j' NUMBER_GROUPING = 0 PASSWORD_HASHERS = '****' PASSWORD_RESET_TIMEOUT_DAYS = '****' POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost/' PREPEND_WWW = False Q_CLUSTER = {'timeout': 300, 'save_limit': 100, 'orm': 'default', 'poll': 5} ROOT_URLCONF = 'urls' SECRET_KEY = '****' SECURE_BROWSER_XSS_FILTER = False SECURE_CONTENT_TYPE_NOSNIFF = False SECURE_HSTS_INCLUDE_SUBDOMAINS = False SECURE_HSTS_PRELOAD = False SECURE_HSTS_SECONDS = 0 SECURE_PROXY_SSL_HEADER = None SECURE_REDIRECT_EXEMPT = [] SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False SERVER_EMAIL = 'root@localhost.local' SESSION_CACHE_ALIAS = 'default' SESSION_COOKIE_AGE = 1209600 SESSION_COOKIE_DOMAIN = None SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_NAME = 'sessionid' SESSION_COOKIE_PATH = '/' SESSION_COOKIE_SECURE = False SESSION_ENGINE = 'django.contrib.sessions.backends.db' SESSION_EXPIRE_AT_BROWSER_CLOSE = False SESSION_FILE_PATH = None SESSION_SAVE_EVERY_REQUEST = False SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer' SETTINGS_MODULE = 'settings' SHORT_DATETIME_FORMAT = 'm/d/Y P' SHORT_DATE_FORMAT = 'm/d/Y' SIGNING_BACKEND = 'django.core.signing.TimestampSigner' SILENCED_SYSTEM_CHECKS = [] SITE_ID = 1 SOCIALACCOUNT_PROVIDERS = {} STATICFILES_DIRS = '()' STATICFILES_FINDERS = "('django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'compressor.finders.CompressorFinder')" STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' STATIC_ROOT = '/var/lib/mailman3/web/static' STATIC_URL = '/mailman3/static/' TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.media', 'django.template.context_processors.static', 'django.template.context_processors.tz', 'django.template.context_processors.csrf', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'django_mailman3.context_processors.common', 'hyperkitty.context_processors.common', 'postorius.context_processors.postorius']}}] TEST_NON_SERIALIZED_APPS = [] TEST_RUNNER = 'django.test.runner.DiscoverRunner' THOUSAND_SEPARATOR = ',' TIME_FORMAT = 'P' TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] TIME_ZONE = 'UTC' USE_ETAGS = False USE_I18N = True USE_L10N = True USE_THOUSAND_SEPARATOR = False USE_TZ = True USE_X_FORWARDED_HOST = True USE_X_FORWARDED_PORT = False WSGI_APPLICATION = 'wsgi.application' X_FRAME_OPTIONS = 'SAMEORIGIN' YEAR_MONTH_FORMAT = 'F Y'

I tried three tests with different parameters, with the new commits, on Digital Ocean. I get the same error as before. Although I notice that the postgresql user names have changed to no longer include @localhost.

I'll keep experimenting, with playbook changes and other platforms, and see if I can make any more progress on my side.

Outputs of the the two commands requested above produce the same output (e.g. su - postgres -c "psql -c '\list'")

mailman@lists:/var/log/postgresql$ sudo cat postgresql-11-main.log 2021-01-14 13:34:31.904 EST [5753] LOG: listening on IPv4 address "127.0.0.1", port 5432 2021-01-14 13:34:31.905 EST [5753] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-01-14 13:34:31.925 EST [5754] LOG: database system was shut down at 2021-01-14 13:34:29 EST 2021-01-14 13:34:31.931 EST [5753] LOG: database system is ready to accept connections 2021-01-14 13:34:32.487 EST [5761] [unknown]@[unknown] LOG: incomplete startup packet 2021-01-14 13:36:01.240 EST [8678] mailman3@template1 FATAL: password authentication failed for user "mailman3" 2021-01-14 13:36:01.240 EST [8678] mailman3@template1 DETAIL: Role "mailman3" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-14 13:36:01.244 EST [8679] mailman3@template1 FATAL: password authentication failed for user "mailman3" 2021-01-14 13:36:01.244 EST [8679] mailman3@template1 DETAIL: Role "mailman3" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-14 13:36:01.426 EST [8732] postgres@template1 ERROR: role "mailman3" does not exist 2021-01-14 13:36:01.426 EST [8732] postgres@template1 STATEMENT: ALTER USER "mailman3" 2021-01-14 13:36:01.910 EST [8842] postgres@mailman3 FATAL: database "mailman3" does not exist 2021-01-14 13:36:07.328 EST [9075] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:36:07.331 EST [9073] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:08.497 EST [9611] mailman3web@template1 FATAL: password authentication failed for user "mailman3web" 2021-01-14 13:38:08.497 EST [9611] mailman3web@template1 DETAIL: Role "mailman3web" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-14 13:38:08.502 EST [9612] mailman3web@template1 FATAL: password authentication failed for user "mailman3web" 2021-01-14 13:38:08.502 EST [9612] mailman3web@template1 DETAIL: Role "mailman3web" does not exist. Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5" 2021-01-14 13:38:08.744 EST [9665] postgres@template1 ERROR: role "mailman3web" does not exist 2021-01-14 13:38:08.744 EST [9665] postgres@template1 STATEMENT: ALTER USER "mailman3web" 2021-01-14 13:38:09.362 EST [9775] postgres@mailman3web FATAL: database "mailman3web" does not exist 2021-01-14 13:38:19.291 EST [9933] mailman3web@mailman3web ERROR: relation "social_auth_usersocialauth" does not exist at character 15 2021-01-14 13:38:19.291 EST [9933] mailman3web@mailman3web STATEMENT: SELECT 1 from social_auth_usersocialauth 2021-01-14 13:38:51.173 EST [9217] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.187 EST [9218] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.190 EST [9220] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.358 EST [9214] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.369 EST [9197] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.422 EST [9184] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.520 EST [10811] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.605 EST [9221] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.666 EST [9213] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.707 EST [9211] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.835 EST [9194] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:51.846 EST [9163] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:55.816 EST [10855] mailman3web@mailman3web LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:38:55.820 EST [10136] mailman3web@mailman3web LOG: could not receive data from client: Connection reset by peer 2021-01-14 13:43:17.672 EST [5753] LOG: received fast shutdown request 2021-01-14 13:43:17.673 EST [5753] LOG: aborting any active transactions 2021-01-14 13:43:17.676 EST [10895] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.678 EST [10893] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.680 EST [10879] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.682 EST [10878] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.684 EST [10877] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.686 EST [10875] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.688 EST [10874] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.690 EST [10873] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.692 EST [10872] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.694 EST [10870] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.696 EST [10869] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.698 EST [9078] mailman3@mailman3 FATAL: terminating connection due to administrator command 2021-01-14 13:43:17.702 EST [5753] LOG: background worker "logical replication launcher" (PID 5760) exited with exit code 1 2021-01-14 13:43:17.757 EST [5755] LOG: shutting down 2021-01-14 13:43:17.787 EST [5753] LOG: database system is shut down 2021-01-14 13:43:39.018 EST [570] LOG: listening on IPv4 address "127.0.0.1", port 5432 2021-01-14 13:43:39.024 EST [570] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2021-01-14 13:43:39.094 EST [589] LOG: database system was shut down at 2021-01-14 13:43:17 EST 2021-01-14 13:43:39.111 EST [570] LOG: database system is ready to accept connections 2021-01-14 13:43:39.627 EST [642] [unknown]@[unknown] LOG: incomplete startup packet 2021-01-14 13:43:43.783 EST [783] mailman3@mailman3 LOG: could not receive data from client: Connection reset by peer

Thanks for the info and tests, very much appreciated

No worries, I'd love to see this running smoothly. I notice that the databases still aren't getting created, in case that helps.

I tried to "postmap" them in advance, but then ansible seems to fail when it tries to create /var/lib/mailman3 and it's already there.

sudo touch /var/lib/mailman3/data/postfix_domains sudo postmap /var/lib/mailman3/data/postfix_domains sudo touch /var/lib/mailman3/data/postfix_lmtp sudo postmap /var/lib/mailman3/data/postfix_lmtp

I tried to "postmap" them in advance, but then ansible seems to fail when it tries to create /var/lib/mailman3 and it's already there.

Hi all,

I have been testing this issue and it seems Mailman creates the files in /var/lib/mailman3/data/ just after creating the first Mailing List (you need to create a Domain first). Then, postfix catches the lookup databases right away. After that, everything seems to work well.

There are some other issues, tho (I'm working on them). But so far, I didn't found problems with postgres, django or other steps in the role.

Also, I used a VM created from a pristine Debian 10 Cloud Image in my tests and ran the Ansible right after creating the VM. No extra software installed, just the Playbook.

Vars:

mailman3_secret_key: ****
mailman3_rest_api_pass: ****
mailman3_archiver_key: ****
mailman3_db_pass: ****
mailman3_web_db_pass: ****
mailman3_web_admin_pass: ****
mailman3_domain: "mailman.test.domain"
mailman3_service_name: "{{ mailman3_domain }}"
mailman3_service_desc: Listas Prueba UCR
mailman3_locale: es
mailman3_mail: "admin@{{ mailman3_domain }}"
mailman3_tz: America/Costa_Rica
mailman3_update_translations: true
mailman3_extra_pkgs:
  - rsync

# nginx
mailman3_nginx_remove_default: true
mailman3_nginx_main_template: true

# certbot
#mailman3_install_certbot: false

Playbook:

---
- name: Setup Mailman 3
  hosts: mailman
  roles:
    - role: mailman3
      become: true
      tags:
        - mailman3

Let us know if we can provide any additional information to help solve this issue.

I checked the commands you were running whiile prepering the droptlet. One in particular might be causing issues: pip3 install ansible --user mailman. It is installing mailman in addition to ansible using pip. Mailman is not required from pip since it will be installed later on with the ansible role.

You may want to try:

the following commands
and then just create a mailman list trough the web interface

## a new user besides root
adduser mailman
usermod -aG sudo mailman
su - mailman # use this instead of `su mailman` to source ~/.profile and load its $PATH

## upgrade system
sudo apt update
sudo apt upgrade -y

## requirements
sudo apt -y install python3-pip
pip3 install --user ansible    # you were previously installing mailman in this step. You shouldn't since the role will do it
# you may need to log again with user mailman. Ex: `exit` and then `su - mailman`
ansible-galaxy install ucr.mailman3

## run  playbook
cat << EOF > playbook.yaml
- hosts: localhost
  become: true
  become_method: sudo
  vars:
    mailman3_domain: example.com
    # It will install and create a certificate using certbot, its nignx plugin and http validation $
    mailman3_install_certbot: false # install and create certificate
    # secrets, please change them
    mailman3_secret_key: "change_secretkey"
    mailman3_rest_api_pass: "change_api_pass"
    mailman3_archiver_key: "change_archiver_key"
    mailman3_db_pass: "change_db_pass"
    mailman3_web_db_pass: "change_web_db_pass"
    mailman3_web_admin_pass: "change_web_admin_pass"
  roles:
    - ucr.mailman3
EOF

ansible-playbook -i localhost playbook.yaml playbook.yaml -K

So, strangely, this approach seems to introduce a failure that I wasn't getting before.

PLAY RECAP ***** localhost : ok=15 changed=10 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

It finishes with the nginx and mailman services running, but attempts to reach potorius fail (something that I was able to do with the previous method).

I'm going to keep investigating to make sure it's not related to the hostname or to the SSL cert, but let me know if the error below suggests anything to you. One thought... is there any chance that mailman could be using a postgresql extension not supported by the digitalocean virtualized servers?

TASK [ucr.mailman3 : install postgres] ***** fatal: [localhost]: FAILED! => {"cache_update_time": 1613837291, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'postgresql'' failed: E: Sub-process /usr/bin/dpkg returned an error code (1)\n", "rc": 100, "stderr": "E: Sub-process /usr/bin/dpkg returned an error code (1)\n", "stderr_lines": ["E: Sub-process /usr/bin/dpkg returned an error code (1)"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n libllvm7 libpq5 libsensors-config libsensors5 postgresql-11\n postgresql-client-11 postgresql-client-common postgresql-common sysstat\nSuggested packages:\n lm-sensors postgresql-doc postgresql-doc-11 libjson-perl isag\nThe following NEW packages will be installed:\n libllvm7 libpq5 libsensors-config libsensors5 postgresql postgresql-11\n postgresql-client-11 postgresql-client-common postgresql-common sysstat\n0 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 29.8 MB of archives.\nAfter this operation, 116 MB of additional disk space will be used.\nGet:1 http://deb.debian.org/debian buster/main amd64 libllvm7 amd64 1:7.0.1-8+deb10u2 [13.1 MB]\nGet:2 http://deb.debian.org/debian buster/main amd64 libpq5 amd64 11.10-0+deb10u1 [168 kB]\nGet:3 http://deb.debian.org/debian buster/main amd64 libsensors-config all 1:3.5.0-3 [31.6 kB]\nGet:4 http://deb.debian.org/debian buster/main amd64 libsensors5 amd64 1:3.5.0-3 [52.6 kB]\nGet:5 http://deb.debian.org/debian buster/main amd64 postgresql-client-common all 200+deb10u4 [85.1 kB]\nGet:6 http://deb.debian.org/debian buster/main amd64 postgresql-client-11 amd64 11.10-0+deb10u1 [1405 kB]\nGet:7 http://deb.debian.org/debian buster/main amd64 postgresql-common all 200+deb10u4 [225 kB]\nGet:8 http://deb.debian.org/debian buster/main amd64 postgresql-11 amd64 11.10-0+deb10u1 [14.1 MB]\nGet:9 http://deb.debian.org/debian buster/main amd64 postgresql all 11+200+deb10u4 [61.1 kB]\nGet:10 http://deb.debian.org/debian buster/main amd64 sysstat amd64 12.0.3-2 [562 kB]\nPreconfiguring packages ...\nFetched 29.8 MB in 1s (48.5 MB/s)\nSelecting previously unselected package libllvm7:amd64.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 38292 files and directories currently installed.)\r\nPreparing to unpack .../0-libllvm7_1%3a7.0.1-8+deb10u2_amd64.deb ...\r\nUnpacking libllvm7:amd64 (1:7.0.1-8+deb10u2) ...\r\nSelecting previously unselected package libpq5:amd64.\r\nPreparing to unpack .../1-libpq5_11.10-0+deb10u1_amd64.deb ...\r\nUnpacking libpq5:amd64 (11.10-0+deb10u1) ...\r\nSelecting previously unselected package libsensors-config.\r\nPreparing to unpack .../2-libsensors-config_1%3a3.5.0-3_all.deb ...\r\nUnpacking libsensors-config (1:3.5.0-3) ...\r\nSelecting previously unselected package libsensors5:amd64.\r\nPreparing to unpack .../3-libsensors5_1%3a3.5.0-3_amd64.deb ...\r\nUnpacking libsensors5:amd64 (1:3.5.0-3) ...\r\nSelecting previously unselected package postgresql-client-common.\r\nPreparing to unpack .../4-postgresql-client-common_200+deb10u4_all.deb ...\r\nUnpacking postgresql-client-common (200+deb10u4) ...\r\nSelecting previously unselected package postgresql-client-11.\r\nPreparing to unpack .../5-postgresql-client-11_11.10-0+deb10u1_amd64.deb ...\r\nUnpacking postgresql-client-11 (11.10-0+deb10u1) ...\r\nSelecting previously unselected package postgresql-common.\r\nPreparing to unpack .../6-postgresql-common_200+deb10u4_all.deb ...\r\nAdding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'\r\nUnpacking postgresql-common (200+deb10u4) ...\r\nSelecting previously unselected package postgresql-11.\r\nPreparing to unpack .../7-postgresql-11_11.10-0+deb10u1_amd64.deb ...\r\nUnpacking postgresql-11 (11.10-0+deb10u1) ...\r\nSelecting previously unselected package postgresql.\r\nPreparing to unpack .../8-postgresql_11+200+deb10u4_all.deb ...\r\nUnpacking postgresql (11+200+deb10u4) ...\r\nSelecting previously unselected package sysstat.\r\nPreparing to unpack .../9-sysstat_12.0.3-2_amd64.deb ...\r\nUnpacking sysstat (12.0.3-2) ...\r\nSetting up postgresql-client-common (200+deb10u4) ...\r\nSetting up libsensors-config (1:3.5.0-3) ...\r\nSetting up libpq5:amd64 (11.10-0+deb10u1) ...\r\nSetting up postgresql-client-11 (11.10-0+deb10u1) ...\r\nupdate-alternatives: using /usr/share/postgresql/11/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode\r\nSetting up postgresql-common (200+deb10u4) ...\r\nYou are required to change your password immediately (administrator enforced)\r\nchfn: PAM: Authentication token is no longer valid; new one required\r\nadduser: /bin/chfn -f PostgreSQL administrator postgres' returned error code 1. Exiting.\r\ndpkg: error processing package postgresql-common (--configure):\r\n installed postgresql-common package post-installation script subprocess returned error exit status 1\r\nSetting up libsensors5:amd64 (1:3.5.0-3) ...\r\ndpkg: dependency problems prevent configuration of postgresql-11:\r\n postgresql-11 depends on postgresql-common (>= 194~); however:\r\n Package postgresql-common is not configured yet.\r\n\r\ndpkg: error processing package postgresql-11 (--configure):\r\n dependency problems - leaving unconfigured\r\nSetting up libllvm7:amd64 (1:7.0.1-8+deb10u2) ...\r\ndpkg: dependency problems prevent configuration of postgresql:\r\n postgresql depends on postgresql-11; however:\r\n Package postgresql-11 is not configured yet.\r\n\r\ndpkg: error processing package postgresql (--configure):\r\n dependency problems - leaving unconfigured\r\nSetting up sysstat (12.0.3-2) ...\r\n\r\nCreating config file /etc/default/sysstat with new version\r\nupdate-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode\r\nCreated symlink /etc/systemd/system/multi-user.target.wants/sysstat.service -> /lib/systemd/system/sysstat.service.\r\nProcessing triggers for systemd (241-7~deb10u6) ...\r\nProcessing triggers for libc-bin (2.28-10) ...\r\nErrors were encountered while processing:\r\n postgresql-common\r\n postgresql-11\r\n postgresql\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " libllvm7 libpq5 libsensors-config libsensors5 postgresql-11", " postgresql-client-11 postgresql-client-common postgresql-common sysstat", "Suggested packages:", " lm-sensors postgresql-doc postgresql-doc-11 libjson-perl isag", "The following NEW packages will be installed:", " libllvm7 libpq5 libsensors-config libsensors5 postgresql postgresql-11", " postgresql-client-11 postgresql-client-common postgresql-common sysstat", "0 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.", "Need to get 29.8 MB of archives.", "After this operation, 116 MB of additional disk space will be used.", "Get:1 http://deb.debian.org/debian buster/main amd64 libllvm7 amd64 1:7.0.1-8+deb10u2 [13.1 MB]", "Get:2 http://deb.debian.org/debian buster/main amd64 libpq5 amd64 11.10-0+deb10u1 [168 kB]", "Get:3 http://deb.debian.org/debian buster/main amd64 libsensors-config all 1:3.5.0-3 [31.6 kB]", "Get:4 http://deb.debian.org/debian buster/main amd64 libsensors5 amd64 1:3.5.0-3 [52.6 kB]", "Get:5 http://deb.debian.org/debian buster/main amd64 postgresql-client-common all 200+deb10u4 [85.1 kB]", "Get:6 http://deb.debian.org/debian buster/main amd64 postgresql-client-11 amd64 11.10-0+deb10u1 [1405 kB]", "Get:7 http://deb.debian.org/debian buster/main amd64 postgresql-common all 200+deb10u4 [225 kB]", "Get:8 http://deb.debian.org/debian buster/main amd64 postgresql-11 amd64 11.10-0+deb10u1 [14.1 MB]", "Get:9 http://deb.debian.org/debian buster/main amd64 postgresql all 11+200+deb10u4 [61.1 kB]", "Get:10 http://deb.debian.org/debian buster/main amd64 sysstat amd64 12.0.3-2 [562 kB]", "Preconfiguring packages ...", "Fetched 29.8 MB in 1s (48.5 MB/s)", "Selecting previously unselected package libllvm7:amd64.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 38292 files and directories currently installed.)", "Preparing to unpack .../0-libllvm7_1%3a7.0.1-8+deb10u2_amd64.deb ...", "Unpacking libllvm7:amd64 (1:7.0.1-8+deb10u2) ...", "Selecting previously unselected package libpq5:amd64.", "Preparing to unpack .../1-libpq5_11.10-0+deb10u1_amd64.deb ...", "Unpacking libpq5:amd64 (11.10-0+deb10u1) ...", "Selecting previously unselected package libsensors-config.", "Preparing to unpack .../2-libsensors-config_1%3a3.5.0-3_all.deb ...", "Unpacking libsensors-config (1:3.5.0-3) ...", "Selecting previously unselected package libsensors5:amd64.", "Preparing to unpack .../3-libsensors5_1%3a3.5.0-3_amd64.deb ...", "Unpacking libsensors5:amd64 (1:3.5.0-3) ...", "Selecting previously unselected package postgresql-client-common.", "Preparing to unpack .../4-postgresql-client-common_200+deb10u4_all.deb ...", "Unpacking postgresql-client-common (200+deb10u4) ...", "Selecting previously unselected package postgresql-client-11.", "Preparing to unpack .../5-postgresql-client-11_11.10-0+deb10u1_amd64.deb ...", "Unpacking postgresql-client-11 (11.10-0+deb10u1) ...", "Selecting previously unselected package postgresql-common.", "Preparing to unpack .../6-postgresql-common_200+deb10u4_all.deb ...", "Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'", "Unpacking postgresql-common (200+deb10u4) ...", "Selecting previously unselected package postgresql-11.", "Preparing to unpack .../7-postgresql-11_11.10-0+deb10u1_amd64.deb ...", "Unpacking postgresql-11 (11.10-0+deb10u1) ...", "Selecting previously unselected package postgresql.", "Preparing to unpack .../8-postgresql_11+200+deb10u4_all.deb ...", "Unpacking postgresql (11+200+deb10u4) ...", "Selecting previously unselected package sysstat.", "Preparing to unpack .../9-sysstat_12.0.3-2_amd64.deb ...", "Unpacking sysstat (12.0.3-2) ...", "Setting up postgresql-client-common (200+deb10u4) ...", "Setting up libsensors-config (1:3.5.0-3) ...", "Setting up libpq5:amd64 (11.10-0+deb10u1) ...", "Setting up postgresql-client-11 (11.10-0+deb10u1) ...", "update-alternatives: using /usr/share/postgresql/11/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode", "Setting up postgresql-common (200+deb10u4) ...", "You are required to change your password immediately (administrator enforced)", "chfn: PAM: Authentication token is no longer valid; new one required", "adduser:/bin/chfn -f PostgreSQL administrator postgres' returned error code 1. Exiting.", "dpkg: error processing package postgresql-common (--configure):", " installed postgresql-common package post-installation script subprocess returned error exit status 1", "Setting up libsensors5:amd64 (1:3.5.0-3) ...", "dpkg: dependency problems prevent configuration of postgresql-11:", " postgresql-11 depends on postgresql-common (>= 194~); however:", " Package postgresql-common is not configured yet.", "", "dpkg: error processing package postgresql-11 (--configure):", " dependency problems - leaving unconfigured", "Setting up libllvm7:amd64 (1:7.0.1-8+deb10u2) ...", "dpkg: dependency problems prevent configuration of postgresql:", " postgresql depends on postgresql-11; however:", " Package postgresql-11 is not configured yet.", "", "dpkg: error processing package postgresql (--configure):", " dependency problems - leaving unconfigured", "Setting up sysstat (12.0.3-2) ...", "", "Creating config file /etc/default/sysstat with new version", "update-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode", "Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service -> /lib/systemd/system/sysstat.service.", "Processing triggers for systemd (241-7~deb10u6) ...", "Processing triggers for libc-bin (2.28-10) ...", "Errors were encountered while processing:", " postgresql-common", " postgresql-11", " postgresql"]}

is there any chance that mailman could be using a postgresql extension not supported by the digitalocean virtualized servers?

I don't think it is an extension but not fully sure. Are these extension restrictions applied when you install the postgresql by your self? or are only for managed PG Cluster provided by DO?

The following additional packages will be installed: libllvm7 libpq5 libsensors-config libsensors5 postgresql-11 postgresql-client-11 postgresql-client-common postgresql-common sysstat

Error arises with postgresql-common:

Setting up postgresql-common (200+deb10u4) ... You are required to change your password immediately (administrator enforced) chfn: PAM: Authentication token is no longer valid; new one required adduser: /bin/chfn -f PostgreSQL administrator postgres' returned error code 1. Exiting. dpkg: error processing package postgresql-common (--configure): installed postgresql-common package post-installation script subprocess returned error exit status 1

This prevents postgresql-11 from installing:

dpkg: dependency problems prevent configuration of postgresql-11: postgresql-11 depends on postgresql-common (>= 194~); however: Package postgresql-common is not configured yet. dpkg: error processing package postgresql-11 (--configure): dependency problems - leaving unconfigured

I'm not sure what this means: "chfn: PAM: Authentication token is no longer valid; new one required"

Good catch valarauco! If I preinstall these packages...

libllvm7 libpq5 libsensors-config libsensors5 postgresql-11 postgresql-client-11 postgresql-client-common postgresql-common sysstat

Then I get a much better outcome:

localhost : ok=52 changed=20 unreachable=0 failed=0 skipped=31 rescued=0 ignored=0

I'll keep testing and report back.

I spoke to soon. Pre installing the postgresql packages solved the ansible script failure, but we end up right back to the original 'cannot import name worker from django_q.cluster error.

Universidad-de-Costa-Rica / role-mailman3

ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py) #2