[Backup/Restore] Unable to restore backup in another cluster

javierdelapuente commented 10 months ago

Steps to reproduce

Backup database to a S3 bucket.
Create a new different cluster. Same controller and model names as in the previous one.
juju deploy postgresql-k8s --trust
juju deploy s3-integrator
juju run s3-integrator/leader sync-s3-credentials access-key= secret-key=
juju config s3-integrator .... (same as in the original cluster)
juju relate s3-integrator postgresql-k8s At this point, with s3 correctly configured, the workload gets into blocked state with message failed to initialize stanza, check your S3 settings
juju run postgresql-k8s/leader list-backups Backups are correctly shown.
juju run postgresql-k8s/leader restore backup-id=2023-11-27T09:37:51Z Fails to restore backups, as the charm does not expect the current block state.

Expected behavior

The backup should be restored in the new cluster.

Actual behavior

Versions

Operating system: Ubuntu 22.04.3 LTS

Juju CLI: 3.1.6-genericlinux-amd64

Juju agent:

Model             Controller  Cloud/Region        Version  SLA          Timestamp
tutorial-synapse  microk8s    microk8s/localhost  3.1.6    unsupported  15:26:31+01:00

App             Version  Status   Scale  Charm           Channel    Rev  Address         Exposed  Message
postgresql-k8s  14.9     waiting      1  postgresql-k8s  14/stable  158  10.152.183.134  no       installing agent
s3-integrator            active       1  s3-integrator   stable      13  10.152.183.70   no       

Unit               Workload  Agent  Address     Ports  Message
postgresql-k8s/0*  blocked   idle   10.1.42.34         failed to initialize stanza, check your S3 settings
s3-integrator/0*   active    idle   10.1.42.35

Charm revision:

microk8s: MicroK8s v1.27.7 revision 6101

Log output

Juju debug log:

When creating the integration:

unit-postgresql-k8s-0: 15:22:15 INFO unit.postgresql-k8s/0.juju-log s3-parameters:9: Bucket backups-bucket exists.
unit-postgresql-k8s-0: 15:22:16 ERROR unit.postgresql-k8s/0.juju-log s3-parameters:9: non-zero exit code 28 executing ['pgbackrest', '--stanza=tutorial-synapse.patroni-postgresql-k8s', 'stanza-create'], stdout='', stderr='ERROR: [028]: backup and archive info files exist but do not match the database\n       HINT: is this the correct stanza?\n       HINT: did an error occur during stanza-upgrade?\n'
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/src/backups.py", line 299, in _initialise_stanza
    self._execute_command(["pgbackrest", f"--stanza={self.stanza_name}", "stanza-create"])
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/src/backups.py", line 212, in _execute_command
    ).wait_output()
  File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 1359, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 28 executing ['pgbackrest', '--stanza=tutorial-synapse.patroni-postgresql-k8s', 'stanza-create'], stdout='', stderr='ERROR: [028]: backup and archive info files exist but do not match the database\n       HINT: is this the correct stanza?\n       HINT: did an error occur during stanza-upgrade?\n'
unit-postgresql-k8s-0: 15:22:16 INFO juju.worker.uniter.operation ran "s3-parameters-relation-changed" hook (via hook dispatching script: dispatch)

When trying to restore:

unit-postgresql-k8s-0: 15:25:12 INFO unit.postgresql-k8s/0.juju-log Checking if cluster is in blocked state
unit-postgresql-k8s-0: 15:25:12 ERROR unit.postgresql-k8s/0.juju-log Restore failed: Cluster or unit is in a blocking state