K8SPG-509: backup/restore improvements

[pooknull]

https://perconadev.atlassian.net/browse/K8SPG-509

DESCRIPTION

Problem: In some rare cases, pg-cluster may have the .spec.restore field set to true even when there are no active restores.

Cause: After the restore job is complete, the operator attempts to update the status of pg-restore to Succeeded and then updates the pg-cluster. If the operator encounters an error during the pg-cluster update, no further attempts are made to update pg-cluster since pg-restore already has the Succeeded state.

Solution: Update pg-cluster first and update pg-restore as the last step.

Also, this PR fixes a problem when the user decides to delete the pg-backup while it's running. In that case, pg-cluster will have an annotation pgv2.percona.com/backup-in-progress which blocks the start of other backups. Now, the operator will check if the pg-backup provided in this annotation exists and delete the annotation if needed.

CHECKLIST

Jira

• [x] Is the Jira ticket created and referenced properly?
• [x] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• [x] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• [x] Is an E2E test/test case added for the new feature/change?
• [x] Are unit tests added where appropriate?

Config/Logging/Testability

• [x] Are all needed new/changed options added to default YAML files?
• [x] Are the manifests (crd/bundle) regenerated if needed?
• [x] Did we add proper logging messages for operator actions?
• [x] Did we ensure compatibility with the previous version or cluster upgrade process?
• [x] Does the change support oldest and newest supported PG version?
• [x] Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test name	Status
custom-extensions	passed
demand-backup	passed
init-deploy	passed
monitoring	passed
one-pod	passed
operator-self-healing	passed
pitr	passed
scaling	passed
scheduled-backup	passed
self-healing	passed
start-from-backup	passed
tablespaces	passed
telemetry-transfer	passed
upgrade-consistency	passed
upgrade-minor	passed
users	passed
We run 16 out of 16

commit: https://github.com/percona/percona-postgresql-operator/pull/797/commits/07922ddb061684e48f2057bd4d3e2ccaeab39a6b image: perconalab/percona-postgresql-operator:PR-797-07922ddb0