search

spantaleev / matrix-docker-ansible-deploy

🐳 Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker
GNU Affero General Public License v3.0
4.81k stars 1.04k forks source link

pg_dump version mismatch with borg backup enabled #3579

Open Taubin opened 1 week ago

Taubin commented 1 week ago

Describe the bug

When enabling borg backup and running the backup fails due to a mismatch within pg_dump

To Reproduce Enable borg backup in the config. Run the playbook. Run the backup on the server using systemctl start matrix-backup-borg

My vars.yml file looks like this:

---
# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
#
# Example value: example.com
matrix_domain: [redacted.tld]

# Use base domain
matrix_static_files_container_labels_base_domain_enabled: true

# The Matrix homeserver software to install.
# See:
#  - `roles/custom/matrix-base/defaults/main.yml` for valid options
# - the `docs/configuring-playbook-IMPLEMENTATION_NAME.md` documentation page, if one is available for your implementation choice
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: '[redacted]'

# By default, the playbook manages its own Traefik (https://doc.traefik.io/traefik/) reverse-proxy server.
# It will retrieve SSL certificates for you on-demand and forward requests to all other components.
# For alternatives, see `docs/configuring-playbook-own-webserver.md`.
matrix_playbook_reverse_proxy_type: playbook-managed-traefik

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certificates (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: someone@example.com
traefik_config_certificatesResolvers_acme_email: 'me@here.com'

# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service)
# using this superuser account.
postgres_connection_password: '[redacted]'

# By default, we configure Coturn's external IP address using the value specified for `ansible_host` in your `inventory/hosts` file.
# If this value is an external IP address, you can skip this section.
#
# If `ansible_host` is not the server's external IP address, you have 2 choices:
# 1. Uncomment the line below, to allow IP address auto-detection to happen (more on this below)
# 2. Uncomment and adjust the line below to specify an IP address manually
#
# By default, auto-detection will be attempted using the `https://ifconfig.co/json` API.
# Default values for this are specified in `matrix_coturn_turn_external_ip_address_auto_detection_*` variables in the Coturn role
# (see `roles/custom/matrix-coturn/defaults/main.yml`).
#
# If your server has multiple IP addresses, you may define them in another variable which allows a list of addresses.
# Example: `matrix_coturn_turn_external_ip_addresses: ['1.2.3.4', '4.5.6.7']`
#
# matrix_coturn_turn_external_ip_address: ''

prometheus_enabled: true

# You can remove this, if unnecessary.
prometheus_node_exporter_enabled: true

# You can remove this, if unnecessary.
prometheus_postgres_exporter_enabled: true

# You can remove this, if unnecessary.
matrix_prometheus_nginxlog_exporter_enabled: true

grafana_enabled: true

grafana_anonymous_access: false

# This has no relation to your Matrix user id. It can be any username you'd like.
# Changing the username subsequently won't work.
grafana_default_admin_user: "[redacted]"

# Changing the password subsequently won't work.
grafana_default_admin_password: "[redacted]"

# Enable Synapse Admin
matrix_synapse_admin_enabled: true

# Enable auto compression
matrix_synapse_auto_compressor_enabled: true

# Use staging for testing
traefik_config_certificatesResolvers_acme_use_staging: false

# NTFY install
ntfy_enabled: true

# Disable Encryption
matrix_synapse_ext_encryption_disabler_enabled: true

# Email settings
exim_relay_sender_address: "[redacted]"
exim_relay_relay_use: true
exim_relay_relay_host_name: "[redacted]"
exim_relay_relay_host_port: 587
exim_relay_relay_auth: true
exim_relay_relay_auth_username: "[redacted]"
exim_relay_relay_auth_password: "[redacted]"

# Registration bot
matrix_bot_matrix_registration_bot_enabled: true

# By default, the playbook will set use the bot with a username like this: `@bot.matrix-registration-bot:DOMAIN`.
# To use a different username, uncomment & adjust the variable below:
# matrix_bot_matrix_registration_bot_matrix_user_id_localpart: bot.matrix-registration-bot

# Generate a strong password here. Consider generating it with `pwgen -s 64 1`
matrix_bot_matrix_registration_bot_bot_password: [redacted]

# Enables registration
matrix_synapse_enable_registration: true

# Restrict registration to users with a token
matrix_synapse_registration_requires_token: true

# Bog Backup to Hetzner
backup_borg_enabled: true
backup_borg_location_repositories:
 - ssh://[redacted].repo.borgbase.com/./repo
backup_borg_storage_encryption_passphrase: "[redacted]"
backup_borg_ssh_key_private: |
  -----BEGIN OPENSSH PRIVATE KEY-----
  [redacted]
  -----END OPENSSH PRIVATE KEY-----

Expected behavior I expect the server to perform the backup properly using the current version of pg_dump

Matrix Server:

Additional context Full error:

Oct 08 22:14:09 [redacted] matrix-backup-borg[266697]: Remote: Warning: Permanently added '[redacted]' (ED25519) to the list of known hosts.
Oct 08 22:14:16 [redacted] matrix-backup-borg[266850]: [redacted]/./repo: Ignoring configured "read_special" value of false, as true is needed for database hooks.
Oct 08 22:14:20 [redacted] matrix-backup-borg[266850]: Remote: Warning: Permanently added '[redacted]' (ED25519) to the list of known hosts.
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: ------------------------------------------------------------------------------
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Repository: [redacted]/./repo
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Archive name: matrix-2024-10-08-221419
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Archive fingerprint: 8038a8f1387fd548dc338e129e1737a8ba901e47618885d1b861e6784cd43486
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Time (start): Tue, 2024-10-08 22:14:21
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Time (end):   Tue, 2024-10-08 22:14:23
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Duration: 1.59 seconds
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Number of files: 139
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Utilization of max. archive size: 0%
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: ------------------------------------------------------------------------------
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:                        Original size      Compressed size    Deduplicated size
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: This archive:              163.31 MB             66.23 MB            174.18 kB
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: All archives:              326.45 MB            132.40 MB             66.17 MB
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: pg_dump: error: aborting because of server version mismatch
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: pg_dump: detail: server version: 17.0; pg_dump version: 16.3
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: [redacted]/./repo: Error running actions for repository
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: --- Logging error ---
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Traceback (most recent call last):
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 621, in log_error_records
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     raise error
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 145, in run_configuration
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     yield from run_actions(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 330, in run_actions
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     yield from borgmatic.actions.create.run_create(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/actions/create.py", line 100, in run_create
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     json_output = borgmatic.borg.create.create_archive(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/borg/create.py", line 554, in create_archive
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     return execute_command_with_processes(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/execute.py", line 424, in execute_command_with_processes
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     captured_outputs = log_outputs(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:                        ^^^^^^^^^^^^
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/execute.py", line 210, in log_outputs
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     raise subprocess.CalledProcessError(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: subprocess.CalledProcessError: Command 'pg_dump --no-password --clean --if-exists --host matrix-postgres --port 5432 --username matrix --format custom synapse > /root/.borgmatic/postgresql_databases/matrix-postgres/synapse' returned non-zero exit status 1.
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: During handling of the above exception, another exception occurred:
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Traceback (most recent call last):
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1163, in emit
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     stream.write(msg + self.terminator)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:                  ~~~~^~~~~~~~~~~~~~~~~
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: TypeError: unsupported operand type(s) for +: 'CalledProcessError' and 'str'
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Call stack:
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/bin/borgmatic", line 8, in <module>
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     sys.exit(main())
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 922, in main
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     or list(collect_configuration_run_summary_logs(configs, config_paths, arguments))
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 804, in collect_configuration_run_summary_logs
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     results = list(run_configuration(config_filename, config, config_paths, arguments))
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 176, in run_configuration
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     yield from log_error_records(
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 642, in log_error_records
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     yield log_record(levelno=levelno, levelname=level_name, msg=error)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 599, in log_record
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     logger.handle(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1700, in handle
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     self.callHandlers(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     hdlr.handle(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1028, in handle
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     self.emit(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/logger.py", line 76, in emit
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     self.log_level_to_handler[record.levelno].emit(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Message: CalledProcessError(1, 'pg_dump --no-password --clean --if-exists --host matrix-postgres --port 5432 --username matrix --format custom synapse > /root/.borgmatic/postgresql_databases/matrix-postgres/synapse', 'pg_dump: error: aborting because of server version mismatch\npg_dump: detail: server version: 17.0; pg_dump version: 16.3')
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Arguments: ()
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Error while creating a backup.
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: /etc/borgmatic.d/config.yaml: An error occurred
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: summary:
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: /etc/borgmatic.d/config.yaml: An error occurred
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: [redacted]/./repo: Error running actions for repository
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: pg_dump: error: aborting because of server version mismatch
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: pg_dump: detail: server version: 17.0; pg_dump version: 16.3
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: --- Logging error ---
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Traceback (most recent call last):
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1163, in emit
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     stream.write(msg + self.terminator)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:                  ~~~~^~~~~~~~~~~~~~~~~
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: TypeError: unsupported operand type(s) for +: 'CalledProcessError' and 'str'
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Call stack:
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/bin/borgmatic", line 8, in <module>
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     sys.exit(main())
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/commands/borgmatic.py", line 935, in main
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     logger.handle(log)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1700, in handle
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     self.callHandlers(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     hdlr.handle(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/logging/__init__.py", line 1028, in handle
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     self.emit(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:   File "/usr/lib/python3.12/site-packages/borgmatic/logger.py", line 76, in emit
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]:     self.log_level_to_handler[record.levelno].emit(record)
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Message: CalledProcessError(1, 'pg_dump --no-password --clean --if-exists --host matrix-postgres --port 5432 --username matrix --format custom synapse > /root/.borgmatic/postgresql_databases/matrix-postgres/synapse', 'pg_dump: error: aborting because of server version mismatch\npg_dump: detail: server version: 17.0; pg_dump version: 16.3')
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Arguments: ()
Oct 08 22:14:25 [redacted] matrix-backup-borg[266850]: Need some help? https://torsion.org/borgmatic/#issues
Oct 08 22:14:25 [redacted] systemd[1]: matrix-backup-borg.service: Main process exited, code=exited, status=1/FAILURE
dropthehate commented 1 day ago

Same here

dropthehate commented 1 day ago

It seems that borgmatic docker image uses an older version of postgresql client (pg_dump is v16.3) that is incompatible with the latest 17 version of postgres server that comes with this ansible playbook.

We'll have to wait for the update from borgmatic it seems. Other option would be to install and configure borgmatic manually on the server machine and install compatible postgresql client.

As for this ansible playbook, it's worth the thought to make and use a custom docker image for postgres and include borgmatic in it, so it always uses the same client that comes with postgres. There are lots of possible issues with that kind of setup too. I hate containerized builds sometimes for exactly this.