Backup_Restore not working using clone config

[pawanku2]

Please, answer some short questions which should help us to understand your problem / question better?

BackUp_Restore_PG.txt

Some general remarks when posting a bug report:

• Please, check the operator, pod (Patroni) and postgresql logs first. When copy-pasting many log lines please do it in a separate GitHub gist together with your Postgres CRD and configuration manifest.
• If you feel this issue might be more related to the Spilo docker image or Patroni, consider opening issues in the respective repos.

[pawanku2]

• Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.6.3

• **Where do you run it - cloud or metal? Kubernetes or OpenShift?

K8s

• Are you running Postgres Operator in production? [yes | no] No

• Type of issue? [Bug report, question, feature request, etc.] Question

I am trying to do backup and restore on openstack s3 . I have tried evrything which is mention in the Docs.

Backup is done on S3 but restore is fail.

Steps)

Created Operator

kubectl create -f manifests/configmap.yaml kubectl create -f manifests/operator-service-account-rbac.yaml kubectl create -f manifests/postgres-operator.yaml kubectl create -f manifests/api-service.yaml kubectl create -f manifests/my-custom-config.yaml ## Configmap for BACKUP and RESTORE enviorment

my-custom-config.yaml kind: ConfigMap metadata: name: my-custom-config # The name must be same as configured in the values file earlier data: BACKUP_SCHEDULE: "/5 *" # Schedule a base backup every 12 hours; you can customise as you wish USE_WALG_BACKUP: "true" # Use the golang backup tool (faster) USE_WALG_RESTORE: "true" BACKUP_NUM_TO_RETAIN: "7" # For 2 backups per day, keep 7 days of base backups AWS_ACCESS_KEY_ID: "xxxxxxxxxxxxxxxxxx" AWS_SECRET_ACCESS_KEY: "xxxxxxxxxxxxxxxxxxxxxxxxxx" AWS_ENDPOINT: "xxxxxxxxxxxxxxxxxxxxxx" AWS_REGION: "us-east-1" AWS_S3_FORCE_PATH_STYLE: "true" PGHOST: "loacalhost" PGPORT: "5432" WALG_S3_PREFIX: "s3://bucket-postgres-backup" WALG_DISABLE_S3_SSE: "true" # I disable this because it's not supported by Scaleway WALE_ENV_DIR : /run/etc/wal.d/env

WAL_S3_BUCKET: "backup-staging-postgres"

SCOPE: acid-minimal-cluster

CLONE_AWS_ENDPOINT: "xxxxxxxxxxxxxxxxxxxxxxx" # SAME as AWS_ENDPOINT CLONE_AWS_DEFAULT_REGION: "us-east-1" CLONE_AWS_S3_FORCE_PATH_STYLE: "true" CLONE_USE_WALG_BACKUP: "true" CLONE_USE_WALG_RESTORE: "false"

CLONE_WAL_S3_BUCKET: "backup-staging-postgres"

CLONE_SCOPE: acid-minimal-cluster CLONE_WALG_S3_PREFIX: s3://bucket-postgres-backup

Create Cluster

apiVersion: "acid.zalan.do/v1" kind: postgresql metadata: name: acid-minimal-cluster namespace: default spec: teamId: "acid" volume: size: 1Gi numberOfInstances: 2 users: zalando: # database owner

• superuser
• createdb foo_user: [] # role for application foo databases: foo: zalando # dbname: owner preparedDatabases: bar: {} postgresql: version: "13"

Once cluster created , i have created one DB and add some data and saw buckup happen

s3cmd ls s3://bucket-postgres-backup/basebackups_005/ DIR s3://bucket-postgres-backup/basebackups_005/base_000000010000000000000004/ DIR s3://bucket-postgres-backup/basebackups_005/base_000000010000000000000006/ DIR s3://bucket-postgres-backup/basebackups_005/base_00000003000000000000000A/ DIR s3://bucket-postgres-backup/basebackups_005/base_00000003000000000000000C/ DIR s3://bucket-postgres-backup/basebackups_005/base_00000003000000000000000E/ DIR s3://bucket-postgres-backup/basebackups_005/base_000000030000000000000010/ DIR s3://bucket-postgres-backup/basebackups_005/base_000000030000000000000012/ DIR s3://bucket-postgres-backup/basebackups_005/base_000000030000000000000014/ 2021-06-08 21:00 217122 s3://bucket-postgres-backup/basebackups_005/base_000000010000000000000004_backup_stop_sentinel.json 2021-06-08 21:05 217274 s3://bucket-postgres-backup/basebackups_005/base_000000010000000000000006_backup_stop_sentinel.json 2021-06-08 21:15 217344 s3://bucket-postgres-backup/basebackups_005/base_00000003000000000000000A_backup_stop_sentinel.json 2021-06-08 21:20 217499 s3://bucket-postgres-backup/basebackups_005/base_00000003000000000000000C_backup_stop_sentinel.json 2021-06-08 21:25 217499 s3://bucket-postgres-backup/basebackups_005/base_00000003000000000000000E_backup_stop_sentinel.json 2021-06-08 21:30 217499 s3://bucket-postgres-backup/basebackups_005/base_000000030000000000000010_backup_stop_sentinel.json 2021-06-08 21:35 217499 s3://bucket-postgres-backup/basebackups_005/base_000000030000000000000012_backup_stop_sentinel.json 2021-06-08 21:40 217498 s3://bucket-postgres-backup/basebackups_005/base_000000030000000000000014_backup_stop_sentinel.json

After that i delete the cluster and recreate with new manifest adding clone param

apiVersion: "acid.zalan.do/v1" kind: postgresql metadata: name: acid-minimal-cluster namespace: default spec: teamId: "acid" volume: size: 1Gi numberOfInstances: 2 users: zalando: # database owner

• superuser
• createdb foo_user: [] # role for application foo databases: foo: zalando # dbname: owner preparedDatabases: bar: {} postgresql: version: "13" clone: uid: "53e1acd7-6b14-4e85-849f-d349ef6ebe62" cluster: "acid-minimal-cluster" timestamp: "2021-06-08T21:00:00+01:00"

Cluster created but DB crashed.

Any input is appreciate, I tried everything but no success

[FxKu]

@pawanku2 so you are exercising an in-place restoring of a cluster (cloning a deleted cluster). Can you provide error messages from operator / Patroni / Postgres logs? Did you create the clone immediately after deleting the old cluster? If there was too little time in between some resources from the old cluster can block the creation.

[pawanku2]

Yes , I have deleted the cluster and create in 1min gap.

Below operator logs:

time="2021-06-09T08:13:26Z" level=info msg="Fully qualified configmap name: default/postgres-operator"
time="2021-06-09T08:13:26Z" level=info msg="Spilo operator v1.6.3\n"
time="2021-06-09T08:13:27Z" level=info msg="Parse role bindings" pkg=controller
time="2021-06-09T08:13:27Z" level=info msg="successfully parsed" pkg=controller
time="2021-06-09T08:13:27Z" level=info msg="Listening to all namespaces" pkg=controller
time="2021-06-09T08:13:27Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="{" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ReadyWaitInterval\": 3000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ReadyWaitTimeout\": 30000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ResyncPeriod\": 1800000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"RepairPeriod\": 300000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableCRDValidation\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ResourceCheckInterval\": 3000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ResourceCheckTimeout\": 600000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodLabelWaitTimeout\": 600000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodDeletionWaitTimeout\": 600000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SpiloRunAsUser\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SpiloRunAsGroup\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SpiloFSGroup\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodPriorityClassName\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ClusterDomain\": \"cluster.local\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SpiloPrivileged\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SpiloAllowPrivilegeEscalation\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"AdditionalPodCapabilities\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ClusterLabels\": {" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="      \"application\": \"spilo\"" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   }," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"InheritedLabels\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"InheritedAnnotations\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DownscalerAnnotations\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ClusterNameLabel\": \"cluster-name\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DeleteAnnotationDateKey\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DeleteAnnotationNameKey\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodRoleLabel\": \"spilo-role\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodToleration\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DefaultCPURequest\": \"100m\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DefaultCPULimit\": \"1\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MinCPULimit\": \"250m\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MinMemoryLimit\": \"250Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodEnvironmentConfigMap\": \"default/my-custom-config\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodEnvironmentSecret\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"NodeReadinessLabel\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MaxInstances\": -1," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MinInstances\": -1," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ShmVolume\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SecretNameTemplate\": \"{username}.{cluster}.credentials\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PamRoleName\": \"zalandos\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"OAuthTokenSecretName\": \"default/postgresql-operator\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"InfrastructureRolesSecretName\": \"/\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"InfrastructureRoles\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"InfrastructureRolesDefs\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SuperUsername\": \"postgres\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ReplicationUsername\": \"standby\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrAPIKey\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrImage\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrServerURL\": \"https://upload.eu.scalyr.com\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrCPURequest\": \"100m\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrMemoryRequest\": \"50Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrCPULimit\": \"1\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ScalyrMemoryLimit\": \"500Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupSchedule\": \"30 00 * * *\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.6.3\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupProvider\": \"s3\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupS3Bucket\": \"my-bucket-url\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupS3Region\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupS3Endpoint\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupS3AccessKeyID\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupS3SecretAccessKey\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupS3SSE\": \"AES256\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogicalBackupJobPrefix\": \"logical-backup-\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"NumberOfInstances\": 2," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-16\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"Mode\": \"transaction\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MaxDBConnections\": 60," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"WatchedNamespace\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"KubernetesUseConfigMaps\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EtcdHost\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-13:2.0-p7\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SidecarImages\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SidecarContainers\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodServiceAccountName\": \"postgres-pod\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodServiceAccountDefinition\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"}}\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodServiceAccountRoleBindingDefinition\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"postgres-pod\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"postgres-pod\\\"}]}\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MasterPodMoveTimeout\": 1200000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DbHostedZone\": \"db.example.com\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"AWSRegion\": \"eu-central-1\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"WALES3Bucket\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"LogS3Bucket\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"KubeIAMRole\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"WALGSBucket\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"GCPCredentials\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"AdditionalSecretMount\": \"\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableEBSGp3Migration\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"DebugLogging\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableDBAccess\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableTeamsAPI\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableTeamSuperuser\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"TeamAdminRole\": \"admin\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"RoleDeletionSuffix\": \"_deleted\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableTeamMemberDeprecation\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableAdminRoleForUsers\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnablePostgresTeamCRD\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableMasterLoadBalancer\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableReplicaLoadBalancer\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"CustomServiceAnnotations\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"CustomPodAnnotations\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnablePodAntiAffinity\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"StorageResizeMode\": \"pvc\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableLoadBalancer\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnablePodDisruptionBudget\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableInitContainers\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableSidecars\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"Workers\": 8," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"APIPort\": 8080," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"RingLogLines\": 100," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ClusterHistoryEntries\": 1000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"TeamAPIRoleConfiguration\": {" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="      \"log_statement\": \"all\"" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   }," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodTerminateGracePeriod\": 300000000000," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"ProtectedRoles\": [" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="      \"admin\"" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   ]," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"PostgresSuperuserTeams\": null," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"SetMemoryRequestToLimit\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableLazySpiloUpgrade\": false," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnablePgVersionEnvVar\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"EnableSpiloWalPathCompat\": true," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MajorVersionUpgradeMode\": \"manual\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"MinimalMajorVersion\": \"9.5\"," pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="   \"TargetMajorVersion\": \"13\"" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="}" pkg=controller
time="2021-06-09T08:13:31Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="no clusters running" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="started working in background" pkg=controller
time="2021-06-09T08:13:31Z" level=info msg="listening on :8080" pkg=apiserver
time="2021-06-09T08:13:31Z" level=debug msg="new node has been added: /shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-65d75-jrrn6 (openstack:///1bc9d88f-2010-4397-8912-4a969f740126)" pkg=controller
time="2021-06-09T08:13:31Z" level=debug msg="new node has been added: /shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-65d75-wkqpk (openstack:///e0ac6b79-8a99-4a36-886e-da2801439178)" pkg=controller
time="2021-06-09T08:14:18Z" level=info msg="ADD event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T08:14:18Z" level=info msg="creating a new Postgres cluster" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T08:14:18Z" level=warning msg="master is not running, generated master endpoint does not contain any addresses" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=info msg="endpoint \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=info msg="master service \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=info msg="replica service \"default/acid-minimal-cluster-repl\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=info msg="users have been initialized" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=debug msg="secret default/postgres.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:18Z" level=debug msg="secret default/standby.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:19Z" level=debug msg="secret default/zalando.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:19Z" level=debug msg="secret default/foo-user.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:19Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:19Z" level=info msg="pod disruption budget \"default/postgres-acid-minimal-cluster-pdb\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:20Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:20Z" level=debug msg="[{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:20Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:20Z" level=debug msg="created new statefulset \"default/acid-minimal-cluster\", uid: \"55d1f330-16ee-463e-904a-0539d86f6547\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:20Z" level=info msg="statefulset \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:20Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:23Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:50Z" level=info msg="pods are ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:50Z" level=info msg="Create roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:14:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:15:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:15:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:15:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:15:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:16:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:16:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:16:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.71.108.47:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:16:36Z" level=error msg="could not create cluster: could not create users: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T08:27:17Z" level=info msg="ADD event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T08:27:17Z" level=info msg="creating a new Postgres cluster" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T08:27:17Z" level=warning msg="master is not running, generated master endpoint does not contain any addresses" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=info msg="endpoint \"default/acid-minimal-cluster2\" has been successfully created" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=info msg="master service \"default/acid-minimal-cluster2\" has been successfully created" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=info msg="replica service \"default/acid-minimal-cluster2-repl\" has been successfully created" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=info msg="users have been initialized" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=debug msg="created new secret default/foo-user.acid-minimal-cluster2.credentials, uid: 4ebfe8e9-1113-42a1-bfc5-9c682a929d88" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:17Z" level=debug msg="created new secret default/zalando.acid-minimal-cluster2.credentials, uid: a438edde-298c-4ac8-adb6-0ab76cb95e2e" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=debug msg="created new secret default/postgres.acid-minimal-cluster2.credentials, uid: 9570ff13-9285-4229-a000-7e9f68f1b6ea" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=debug msg="created new secret default/standby.acid-minimal-cluster2.credentials, uid: 9500478e-52b4-4d6e-a3b6-1f5c24c8eefa" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=info msg="pod disruption budget \"default/postgres-acid-minimal-cluster2-pdb\" has been successfully created" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=info msg="Clone from S3 bucket" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=info msg="Figure out which S3 bucket to use from env" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=error msg="Cannot figure out S3 or GS bucket. Both are empty." cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=debug msg="[{SCOPE acid-minimal-cluster2 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_WAL_BUCKET_SCOPE_SUFFIX /7e0b5c84-d8eb-44e3-904b-c7a0995e062f nil} {CLONE_METHOD CLONE_WITH_WALE nil} {CLONE_TARGET_TIME 2021-06-09T08:17:35+00:01 nil} {CLONE_WAL_BUCKET_SCOPE_PREFIX  nil} {AWS_ACCESS_KEY_ID   XXXXXXXXXXXXXXXXXXXXXXXXX                                                                                                                             nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=warning msg="variable \"CLONE_SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=debug msg="created new statefulset \"default/acid-minimal-cluster2\", uid: \"4c0ecbe7-d4ac-4147-b980-8913d284e37b\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=info msg="statefulset \"default/acid-minimal-cluster2\" has been successfully created" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:18Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:27:21Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:37:18Z" level=error msg="failed to create cluster: pod labels error: still failing after 200 retries" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:37:18Z" level=error msg="could not create cluster: pod labels error: still failing after 200 retries" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T08:43:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T08:43:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T08:43:31Z" level=info msg="there are 2 clusters running" pkg=controller
time="2021-06-09T08:43:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T08:43:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T08:43:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:31Z" level=debug msg="secret default/postgres.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:31Z" level=debug msg="secret default/standby.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:32Z" level=debug msg="secret default/standby.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:32Z" level=debug msg="secret default/foo-user.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:32Z" level=debug msg="secret default/postgres.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:35Z" level=debug msg="[{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:35Z" level=info msg="Clone from S3 bucket" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=info msg="Figure out which S3 bucket to use from env" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=error msg="Cannot figure out S3 or GS bucket. Both are empty." cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=debug msg="[{SCOPE acid-minimal-cluster2 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_WAL_BUCKET_SCOPE_SUFFIX /7e0b5c84-d8eb-44e3-904b-c7a0995e062f nil} {CLONE_METHOD CLONE_WITH_WALE nil} {CLONE_TARGET_TIME 2021-06-09T08:17:35+00:01 nil} {CLONE_WAL_BUCKET_SCOPE_PREFIX  nil} {AWS_ACCESS_KEY_ID eXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=warning msg="variable \"CLONE_SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:36Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:36Z" level=debug msg="syncing databases" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:36Z" level=info msg="creating database \"bar\" owner \"bar_owner\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:43:38Z" level=info msg="creating database \"foo\" owner \"zalando\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=warning msg="closing an existing connection before opening a new one to bar" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="syncing prepared database \"bar\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=info msg="creating database schema \"data\" owner \"bar_data_owner\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=debug msg="could not get connection pooler secret pooler.acid-minimal-cluster.credentials: secrets \"pooler.acid-minimal-cluster.credentials\" not found" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=info msg="healthy cluster ready to upgrade, current: 130003 desired: 130000" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T08:43:40Z" level=info msg="cluster has been synced" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T08:43:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:44:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:44:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:44:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:44:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:45:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:45:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:45:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T08:45:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T09:13:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T09:13:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T09:13:31Z" level=info msg="there are 2 clusters running" pkg=controller
time="2021-06-09T09:13:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T09:13:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T09:13:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:31Z" level=debug msg="secret default/standby.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:31Z" level=debug msg="secret default/postgres.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:32Z" level=debug msg="secret default/foo-user.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:32Z" level=debug msg="secret default/standby.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:33Z" level=debug msg="secret default/postgres.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:33Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=info msg="Clone from S3 bucket" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=info msg="Figure out which S3 bucket to use from env" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=error msg="Cannot figure out S3 or GS bucket. Both are empty." cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=debug msg="[{SCOPE acid-minimal-cluster2 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_WAL_BUCKET_SCOPE_SUFFIX /7e0b5c84-d8eb-44e3-904b-c7a0995e062f nil} {CLONE_METHOD CLONE_WITH_WALE nil} {CLONE_TARGET_TIME 2021-06-09T08:17:35+00:01 nil} {CLONE_WAL_BUCKET_SCOPE_PREFIX  nil} {AWS_ACCESS_KEY_ID eXXXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=warning msg="variable \"CLONE_SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="[{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXXXXXXX0e8831d52f7d825 nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing databases" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing prepared database \"bar\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=debug msg="could not get connection pooler secret pooler.acid-minimal-cluster.credentials: secrets \"pooler.acid-minimal-cluster.credentials\" not found" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=info msg="cluster version up to date. current: 130003, min desired: 130000" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:13:35Z" level=info msg="cluster has been synced" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T09:13:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:13:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:14:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:14:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:14:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:14:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:15:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:15:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:15:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:15:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T09:43:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T09:43:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T09:43:31Z" level=info msg="there are 2 clusters running" pkg=controller
time="2021-06-09T09:43:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T09:43:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T09:43:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:31Z" level=debug msg="secret default/postgres.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:31Z" level=debug msg="secret default/standby.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:31Z" level=debug msg="secret default/standby.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:32Z" level=debug msg="secret default/postgres.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:32Z" level=debug msg="secret default/zalando.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="[{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=info msg="Clone from S3 bucket" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=info msg="Figure out which S3 bucket to use from env" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=error msg="Cannot figure out S3 or GS bucket. Both are empty." cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=debug msg="[{SCOPE acid-minimal-cluster2 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_WAL_BUCKET_SCOPE_SUFFIX /7e0b5c84-d8eb-44e3-904b-c7a0995e062f nil} {CLONE_METHOD CLONE_WITH_WALE nil} {CLONE_TARGET_TIME 2021-06-09T08:17:35+00:01 nil} {CLONE_WAL_BUCKET_SCOPE_PREFIX  nil} {AWS_ACCESS_KEY_ID eXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=warning msg="variable \"CLONE_SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="syncing databases" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="syncing prepared database \"bar\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=debug msg="could not get connection pooler secret pooler.acid-minimal-cluster.credentials: secrets \"pooler.acid-minimal-cluster.credentials\" not found" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=info msg="cluster version up to date. current: 130003, min desired: 130000" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T09:43:35Z" level=info msg="cluster has been synced" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T09:43:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:43:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:44:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:44:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:44:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:44:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:45:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:45:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:45:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T09:45:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:13:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:13:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:13:31Z" level=info msg="there are 2 clusters running" pkg=controller
time="2021-06-09T10:13:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:13:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:13:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:31Z" level=debug msg="secret default/postgres.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:32Z" level=debug msg="secret default/postgres.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:32Z" level=debug msg="secret default/standby.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:32Z" level=debug msg="secret default/standby.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=info msg="Clone from S3 bucket" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=info msg="Figure out which S3 bucket to use from env" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=error msg="Cannot figure out S3 or GS bucket. Both are empty." cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=debug msg="[{SCOPE acid-minimal-cluster2 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_WAL_BUCKET_SCOPE_SUFFIX /7e0b5c84-d8eb-44e3-904b-c7a0995e062f nil} {CLONE_METHOD CLONE_WITH_WALE nil} {CLONE_TARGET_TIME 2021-06-09T08:17:35+00:01 nil} {CLONE_WAL_BUCKET_SCOPE_PREFIX  nil} {AWS_ACCESS_KEY_ID eXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=warning msg="variable \"CLONE_SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="[{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT  nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="syncing databases" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="syncing prepared database \"bar\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=debug msg="could not get connection pooler secret pooler.acid-minimal-cluster.credentials: secrets \"pooler.acid-minimal-cluster.credentials\" not found" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=info msg="cluster version up to date. current: 130003, min desired: 130000" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:13:35Z" level=info msg="cluster has been synced" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:13:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:13:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:14:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:14:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:14:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:14:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:15:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:15:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:15:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:15:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:43:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:43:31Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:43:31Z" level=info msg="there are 2 clusters running" pkg=controller
time="2021-06-09T10:43:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:43:31Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:43:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:31Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:31Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:31Z" level=debug msg="secret default/foo-user.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:31Z" level=debug msg="secret default/postgres.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:31Z" level=debug msg="secret default/zalando.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:31Z" level=debug msg="secret default/standby.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:31Z" level=debug msg="secret default/postgres.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:32Z" level=debug msg="secret default/foo-user.acid-minimal-cluster2.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:32Z" level=debug msg="secret default/standby.acid-minimal-cluster.credentials already exists, fetching its password" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:32Z" level=debug msg="syncing master service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:33Z" level=debug msg="syncing replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:34Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:34Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:34Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:34Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=info msg="Clone from S3 bucket" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=info msg="Figure out which S3 bucket to use from env" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=error msg="Cannot figure out S3 or GS bucket. Both are empty." cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=debug msg="[{SCOPE acid-minimal-cluster2 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster2.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_WAL_BUCKET_SCOPE_SUFFIX /7e0b5c84-d8eb-44e3-904b-c7a0995e062f nil} {CLONE_METHOD CLONE_WITH_WALE nil} {CLONE_TARGET_TIME 2021-06-09T08:17:35+00:01 nil} {CLONE_WAL_BUCKET_SCOPE_PREFIX  nil} {AWS_ACCESS_KEY_ID eXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT XXXXXXXXXXXXXXXXXXXXXXXXX nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=warning msg="variable \"CLONE_SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:35Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="[{SCOPE acid-minimal-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-minimal-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_ENDPOINT XXXXXXXXXXXXXXXXXXXXXXXXX nil} {AWS_REGION us-east-1 nil} {AWS_S3_FORCE_PATH_STYLE true nil} {AWS_SECRET_ACCESS_KEY XXXXXXXXXXXXXXXXXXXXXXXXX nil} {BACKUP_NUM_TO_RETAIN 7 nil} {BACKUP_SCHEDULE */5 * * * * nil} {CLONE_AWS_DEFAULT_REGION us-east-1 nil} {CLONE_AWS_ENDPOINT  nil} {CLONE_AWS_S3_FORCE_PATH_STYLE true nil} {CLONE_SCOPE acid-minimal-cluster nil} {CLONE_USE_WALG_BACKUP true nil} {CLONE_USE_WALG_RESTORE false nil} {CLONE_WALG_S3_PREFIX s3://bucket-postgres-backup nil} {ENABLE_WAL_PATH_COMPAT true nil} {PGHOST loacalhost nil} {PGPORT 5432 nil} {SCOPE acid-minimal-cluster nil} {USE_WALG_BACKUP true nil} {USE_WALG_RESTORE true nil} {WALE_ENV_DIR /run/etc/wal.d/env nil} {WALG_DISABLE_S3_SSE true nil} {WALG_S3_PREFIX s3://bucket-postgres-backup nil}]" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=warning msg="variable \"SCOPE\" is defined in \"postgres\" more than once, the subsequent definitions are ignored" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="syncing roles" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="syncing databases" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:35Z" level=debug msg="syncing prepared database \"bar\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:36Z" level=debug msg="closing database connection" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:36Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:36Z" level=debug msg="could not get connection pooler secret pooler.acid-minimal-cluster.credentials: secrets \"pooler.acid-minimal-cluster.credentials\" not found" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:36Z" level=info msg="cluster version up to date. current: 130003, min desired: 130000" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:43:36Z" level=info msg="cluster has been synced" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:43:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:43:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:44:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:44:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:44:36Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:44:51Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:45:06Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:45:21Z" level=warning msg="could not connect to Postgres database: dial tcp 100.64.106.184:5432: connect: connection refused" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:45:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:45:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:59:16Z" level=info msg="DELETE event has been queued" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:59:16Z" level=info msg="deletion of the cluster started" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T10:59:16Z" level=info msg="removing the logical backup job" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:16Z" level=warning msg="could not remove the logical backup k8s cron job; cronjobs.batch \"logical-backup-acid-minimal-cluster2\" not found" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:16Z" level=debug msg="deleting statefulset" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:16Z" level=info msg="statefulset \"default/acid-minimal-cluster2\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:16Z" level=debug msg="deleting pods" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:16Z" level=debug msg="deleting pod \"default/acid-minimal-cluster2-0\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:16Z" level=debug msg="subscribing to pod \"default/acid-minimal-cluster2-0\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T10:59:52Z" level=info msg="DELETE event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:59:52Z" level=info msg="deletion of the cluster started" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T10:59:52Z" level=info msg="removing the logical backup job" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:59:52Z" level=warning msg="could not remove the logical backup k8s cron job; cronjobs.batch \"logical-backup-acid-minimal-cluster\" not found" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:59:52Z" level=debug msg="deleting statefulset" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:59:52Z" level=info msg="statefulset \"default/acid-minimal-cluster\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:59:52Z" level=debug msg="deleting pods" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:59:52Z" level=debug msg="deleting pod \"default/acid-minimal-cluster-0\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T10:59:52Z" level=debug msg="subscribing to pod \"default/acid-minimal-cluster-0\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="unsubscribing from pod \"default/acid-minimal-cluster-0\" events" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=info msg="pod \"default/acid-minimal-cluster-0\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="pods have been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting PVCs" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting PVC \"default/pgdata-acid-minimal-cluster-0\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="PVCs have been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting secret \"default/standby.acid-minimal-cluster.credentials\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=info msg="secret \"default/standby.acid-minimal-cluster.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting secret \"default/zalando.acid-minimal-cluster.credentials\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=info msg="secret \"default/zalando.acid-minimal-cluster.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting secret \"default/foo-user.acid-minimal-cluster.credentials\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=info msg="secret \"default/foo-user.acid-minimal-cluster.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting secret \"default/postgres.acid-minimal-cluster.credentials\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=info msg="secret \"default/postgres.acid-minimal-cluster.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=debug msg="deleting pod disruption budget" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:19Z" level=info msg="pod disruption budget \"default/postgres-acid-minimal-cluster-pdb\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="deleting endpoint" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=info msg="endpoint \"default/acid-minimal-cluster\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="deleting service master" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=info msg="master service \"default/acid-minimal-cluster\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="deleting endpoint" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=info msg="endpoint \"default/acid-minimal-cluster-repl\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="deleting service replica" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=info msg="replica service \"default/acid-minimal-cluster-repl\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="removing leftover Patroni objects (endpoints / services and configmaps)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="deleting Patroni cluster object \"endpoint\" with name \"default/acid-minimal-cluster-config\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:22Z" level=debug msg="deleting Patroni cluster object \"service\" with name \"default/acid-minimal-cluster-config\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:23Z" level=info msg="deleting connection pooler spilo-role=master" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:23Z" level=debug msg="no connection pooler service object to delete" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:23Z" level=info msg="deleting connection pooler spilo-role=replica" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:23Z" level=debug msg="no connection pooler service object to delete" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-06-09T11:00:23Z" level=info msg="cluster has been deleted" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-06-09T11:04:29Z" level=debug msg="unsubscribing from pod \"default/acid-minimal-cluster2-0\" events" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=info msg="pod \"default/acid-minimal-cluster2-0\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="pods have been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting PVCs" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting PVC \"default/pgdata-acid-minimal-cluster2-0\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="PVCs have been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting secret \"default/foo-user.acid-minimal-cluster2.credentials\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=info msg="secret \"default/foo-user.acid-minimal-cluster2.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting secret \"default/zalando.acid-minimal-cluster2.credentials\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=info msg="secret \"default/zalando.acid-minimal-cluster2.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting secret \"default/postgres.acid-minimal-cluster2.credentials\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=info msg="secret \"default/postgres.acid-minimal-cluster2.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting secret \"default/standby.acid-minimal-cluster2.credentials\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=info msg="secret \"default/standby.acid-minimal-cluster2.credentials\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=debug msg="deleting pod disruption budget" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:29Z" level=info msg="pod disruption budget \"default/postgres-acid-minimal-cluster2-pdb\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="deleting endpoint" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=info msg="endpoint \"default/acid-minimal-cluster2\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="deleting service master" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=info msg="master service \"default/acid-minimal-cluster2\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="deleting endpoint" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=info msg="endpoint \"default/acid-minimal-cluster2-repl\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="deleting service replica" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=info msg="replica service \"default/acid-minimal-cluster2-repl\" has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="removing leftover Patroni objects (endpoints / services and configmaps)" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="deleting Patroni cluster object \"endpoint\" with name \"default/acid-minimal-cluster2-config\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:32Z" level=debug msg="deleting Patroni cluster object \"service\" with name \"default/acid-minimal-cluster2-config\"" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:33Z" level=info msg="deleting connection pooler spilo-role=master" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:33Z" level=debug msg="no connection pooler to delete" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:33Z" level=info msg="deleting connection pooler spilo-role=replica" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:33Z" level=debug msg="no connection pooler to delete" cluster-name=default/acid-minimal-cluster2 pkg=cluster worker=1
time="2021-06-09T11:04:33Z" level=info msg="cluster has been deleted" cluster-name=default/acid-minimal-cluster2 pkg=controller worker=1
time="2021-06-09T11:04:47Z" level=info msg="Shutting down... terminated"
time="2021-06-09T11:04:47Z" level=info msg="Http server shut down" pkg=apiserver
rpc error: code = Unknown desc = Error: No such container: 3cff48a8eef58ce9da52d706a31f6727367f2407016087658296 

[FxKu]

level=error msg="Cannot figure out S3 or GS bucket. Both are empty.

Did you forget to change the operator config options? https://github.com/zalando/postgres-operator/blob/master/docs/reference/operator_parameters.md#aws-or-gcp-interaction