Synapse fails to start due to potential postgre DB issues

[keda82]

Matrix Server:

• OS: Ubuntu 22.04
• Architecture amd64

Problem description: Unfortunately I managed to brick my Matrix server when trying to get OIDC to work. The services appears to be starting and there are no errors in Ansible after "just setup-all"

I reverted back to previous vars.yml config that has been working for quite som time but to no success

Matrix-synapse keeps getting spammed with errors, pointing to DB issues - journalctl -fu matrix-synapse:

Jan 10 10:10:37 matrix matrix-synapse[35671]: return self.currentContext().callWithContext(ctx, func, *args, kw) Jan 10 10:10:37 matrix matrix-synapse[35671]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext Jan 10 10:10:37 matrix matrix-synapse[35671]: return func(*args, *kw) Jan 10 10:10:37 matrix matrix-synapse[35671]: ^^^^^^^^^^^^^^^^^ Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection Jan 10 10:10:37 matrix matrix-synapse[35671]: result = func(conn, args, kw) Jan 10 10:10:37 matrix matrix-synapse[35671]: ^^^^^^^^^^^^^^^^^^^^^^^ Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1033, in inner_func Jan 10 10:10:37 matrix matrix-synapse[35671]: return func(db_conn, *args, kwargs) Jan 10 10:10:37 matrix matrix-synapse[35671]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 773, in new_transaction Jan 10 10:10:37 matrix matrix-synapse[35671]: r = func(cursor, *args, *kwargs) Jan 10 10:10:37 matrix matrix-synapse[35671]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/lock.py", line 166, in _try_acquire_lock_txn Jan 10 10:10:37 matrix matrix-synapse[35671]: txn.execute( Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 419, in execute Jan 10 10:10:37 matrix matrix-synapse[35671]: self._do_execute(self.txn.execute, sql, parameters) Jan 10 10:10:37 matrix matrix-synapse[35671]: File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 481, in _do_execute Jan 10 10:10:37 matrix matrix-synapse[35671]: return func(sql, args, kwargs) Jan 10 10:10:37 matrix matrix-synapse[35671]: ^^^^^^^^^^^^^^^^^^^^^^^^^^ Jan 10 10:10:37 matrix matrix-synapse[35671]: psycopg2.errors.InvalidColumnReference: there is no unique or exclusion constraint matching the ON CONFLICT specification

Playbook Configuration:

My vars.yml file looks like this:

matrix_domain: mydomain.com
matrix_ssl_lets_encrypt_support_email: 'myemail@email.com'
matrix_playbook_reverse_proxy_type: playbook-managed-traefik
devture_traefik_config_certificatesResolvers_acme_email: 'myemail@email.com'
matrix_sliding_sync_enabled: true
matrix_coturn_enabled: true
matrix_coturn_turn_static_auth_secret: SECRET
matrix_synapse_turn_uris:
- turns:turn.mydomain.com:4443?transport=tcp
matrix_coturn_turn_external_ip_address: 'IP ADRESS'
matrix_synapse_macaroon_secret_key: 'SECRET'
matrix_homeserver_generic_secret_key: "{{ matrix_synapse_macaroon_secret_key }}"
devture_postgres_connection_password: 'PASSWORD'
matrix_nginx_proxy_base_domain_serving_enabled: true
matrix_client_hydrogen_enabled: true
matrix_client_cinny_enabled: true

[spantaleev]

You're not telling us how you "tried to get OIDC working" - what did you do to end up in this situation? Perhaps you upgraded Synapse and then trying to switch to an older version?

Also, this issue is probably more suitable for the Synapse repository. I suppose that whatever you did is not really related to the Ansible way of setting up things, but rather to a problem with Synapse.. Or to you trying to do something that's not meant to be supported here or upstream (downgrading Synapse).

[keda82]

@spantaleev I added the authentik oidc: settings that was recommended in the Matrix chatroom for this project. But I could not get it to work, so i played around a bit with the settings (reran setup-all multiple times to try different oidc config)

I did not downgrade synapse in any way

So you recommendation is to turn to the synapse project instead?

[keda82]

so I am recommended to nuke the installation and start over again, is this the recommended steps for uninstalling and starting over from scratch?

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/uninstalling.md

[spantaleev]

Yes. If your setup had federated, you may also consider saving your signing key (see /matrix/synapse/config).

Preserving /matrix/traefik/ssl/acme.json (I may be misremembering the exact path to the acme.json file) may also prevent unnecessarily retrieving new SSL certificates from Let's Encrypt. You haven't obtained too many in too short a time though, so you're unlikely hit their rate limits, so you may simply nuke this too and get new certificates.