search

zalando / postgres-operator

Postgres operator creates and manages PostgreSQL clusters running in Kubernetes
https://postgres-operator.readthedocs.io/
MIT License
4.22k stars 968 forks source link

Postgres Operator cannot sync clusters #1496

Open vishantsolanki opened 3 years ago

vishantsolanki commented 3 years ago

Hi Folks,

I have deployed zalando postgres operator in my production environment k8s, and my cluster and pod are running fine after 3 days,Im facing some problems with Postgres clusters and my db clusters are presenting the status SyncFailed error getting (could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up)

image

Postgres operator pod logs:

time="2021-05-16T20:33:06Z" level=debug msg="syncing pod disruption budgets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T20:33:06Z" level=debug msg="syncing roles" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T20:33:06Z" level=warning msg="could not connect to Postgres database: dial tcp 10.43.252.153:5432: connect: connection refused" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T20:33:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T20:33:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T21:02:59Z" level=info msg="SYNC event has been queued" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T21:02:59Z" level=info msg="syncing of the cluster started" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T21:02:59Z" level=debug msg="fetching possible additional team members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:02:59Z" level=debug msg="team API is disabled, only returning 0 members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:02:59Z" level=info msg="syncing secrets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:02:59Z" level=debug msg="secret affo-test-validation-staging/bar-owner-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:02:59Z" level=debug msg="secret affo-test-validation-staging/zalando.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:00Z" level=debug msg="secret affo-test-validation-staging/postgres.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:00Z" level=debug msg="secret affo-test-validation-staging/bar-writer-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:01Z" level=debug msg="secret affo-test-validation-staging/bar-reader-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:02Z" level=debug msg="secret affo-test-validation-staging/standby.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:02Z" level=debug msg="syncing master service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:03Z" level=debug msg="syncing replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:04Z" level=debug msg="No load balancer created for the replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:04Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:04Z" level=info msg="volume claims do not require changes" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:04Z" level=debug msg="syncing statefulsets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:04Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="[{SCOPE z-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.z-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.z-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"archive_mode\":\"true\",\"archive_timeout\":\"60\",\"autovacuum_analyze_scale_factor\":\"0.02\",\"autovacuum_max_workers\":\"12\",\"autovacuum_vacuum_cost_delay\":\"5ms\",\"autovacuum_vacuum_scale_factor\":\"0.05\",\"checkpoint_completion_target\":\"0.8\",\"checkpoint_timeout\":\"30min\",\"cron_database_name\":\"postgres\",\"default_statistics_target\":\"100\",\"effective_cache_size\":\"36GB\",\"effective_io_concurrency\":\"200\",\"log_statement\":\"all\",\"maintenance_work_mem\":\"2GB\",\"max_parallel_maintenance_workers\":\"4\",\"max_parallel_workers\":\"48\",\"max_parallel_workers_per_gather\":\"12\",\"max_wal_senders\":\"90\",\"max_wal_size\":\"8GB\",\"min_wal_size\":\"2GB\",\"random_page_cost\":\"1.1\",\"shared_buffers\":\"12GB\",\"shared_preload_libraries\":\"pg_cron\",\"vacuum_cost_limit\":\"2400\",\"wal_buffers\":\"16MB\",\"wal_keep_segments\":\"32\",\"work_mem\":\"7860kB\"}},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"200\",\"max_worker_processes\":\"48\",\"wal_level\":\"logical\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:s3-zalando-secret,},Key:AWS_ACCESS_KEY_ID,Optional:nil,},}} {AWS_ENDPOINT https://bh-dc-s3-dhn-15.eecloud.nsn-net.net nil} {AWS_SECRET_ACCESS_KEY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:s3-zalando-secret,},Key:AWS_SECRET_ACCESS_KEY,Optional:nil,},}} {BACKUP_NUM_TO_RETAIN 2 nil} {BACKUP_SCHEDULE 0 */4 * * * nil} {WAL_S3_BUCKET affo-test-validation/staging/backup/postgres nil}]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=info msg="statefulset affo-test-validation-staging/z-postgres is not in the desired state and needs to be updated" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-          }," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="+          }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      restartPolicy: Always," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      securityContext: {}," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="+      securityContext: {}" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      apiVersion: v1," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      status: {" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-        phase: Pending" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-      }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="+      status: {}" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-  }," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="+  }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="metadata.annotation are different" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="-  field.cattle.io/publicEndpoints: [{\\addresses\\:[\\10.133.131.150\\],\\port\\:31169,\\protocol\\:\\TCP\\,\\serviceName\\:\ affo-test-validation-staging:dev-postgres-custom-svc\\,\\allNodes\\:true}]," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=info msg="reason: new statefulset's annotations does not match the current one" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="updating statefulset" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="patching statefulset annotations" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:05Z" level=debug msg="patching statefulset annotations" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:06Z" level=debug msg="calling Patroni API on a pod affo-test-validation-staging/z-postgres-0 to set the following Postgres options: map[max_connections:200 max_worker_processes:48 wal_level:logical]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:06Z" level=debug msg="making PATCH http request: http://10.42.19.70:8008/config" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:06Z" level=debug msg="syncing pod disruption budgets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:06Z" level=debug msg="syncing roles" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:06Z" level=warning msg="could not connect to Postgres database: dial tcp 10.43.252.153:5432: connect: connection refused" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:03:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T21:32:59Z" level=info msg="SYNC event has been queued" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T21:32:59Z" level=info msg="syncing of the cluster started" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T21:32:59Z" level=debug msg="fetching possible additional team members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:32:59Z" level=debug msg="team API is disabled, only returning 0 members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:32:59Z" level=info msg="syncing secrets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:32:59Z" level=debug msg="secret affo-test-validation-staging/zalando.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:32:59Z" level=debug msg="secret affo-test-validation-staging/postgres.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:32:59Z" level=debug msg="secret affo-test-validation-staging/standby.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:00Z" level=debug msg="secret affo-test-validation-staging/bar-reader-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:01Z" level=debug msg="secret affo-test-validation-staging/bar-owner-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:02Z" level=debug msg="secret affo-test-validation-staging/bar-writer-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:02Z" level=debug msg="syncing master service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:03Z" level=debug msg="syncing replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:03Z" level=debug msg="No load balancer created for the replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:03Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:04Z" level=info msg="volume claims do not require changes" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:04Z" level=debug msg="syncing statefulsets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:04Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="[{SCOPE z-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.z-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.z-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"archive_mode\":\"true\",\"archive_timeout\":\"60\",\"autovacuum_analyze_scale_factor\":\"0.02\",\"autovacuum_max_workers\":\"12\",\"autovacuum_vacuum_cost_delay\":\"5ms\",\"autovacuum_vacuum_scale_factor\":\"0.05\",\"checkpoint_completion_target\":\"0.8\",\"checkpoint_timeout\":\"30min\",\"cron_database_name\":\"postgres\",\"default_statistics_target\":\"100\",\"effective_cache_size\":\"36GB\",\"effective_io_concurrency\":\"200\",\"log_statement\":\"all\",\"maintenance_work_mem\":\"2GB\",\"max_parallel_maintenance_workers\":\"4\",\"max_parallel_workers\":\"48\",\"max_parallel_workers_per_gather\":\"12\",\"max_wal_senders\":\"90\",\"max_wal_size\":\"8GB\",\"min_wal_size\":\"2GB\",\"random_page_cost\":\"1.1\",\"shared_buffers\":\"12GB\",\"shared_preload_libraries\":\"pg_cron\",\"vacuum_cost_limit\":\"2400\",\"wal_buffers\":\"16MB\",\"wal_keep_segments\":\"32\",\"work_mem\":\"7860kB\"}},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"200\",\"max_worker_processes\":\"48\",\"wal_level\":\"logical\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:s3-zalando-secret,},Key:AWS_ACCESS_KEY_ID,Optional:nil,},}} {AWS_ENDPOINT https://bh-dc-s3-dhn-15.eecloud.nsn-net.net nil} {AWS_SECRET_ACCESS_KEY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:s3-zalando-secret,},Key:AWS_SECRET_ACCESS_KEY,Optional:nil,},}} {BACKUP_NUM_TO_RETAIN 2 nil} {BACKUP_SCHEDULE 0 */4 * * * nil} {WAL_S3_BUCKET affo-test-validation/staging/backup/postgres nil}]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=info msg="statefulset affo-test-validation-staging/z-postgres is not in the desired state and needs to be updated" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-          }," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="+          }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      restartPolicy: Always," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      securityContext: {}," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="+      securityContext: {}" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      apiVersion: v1," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      status: {" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-        phase: Pending" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-      }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="+      status: {}" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-  }," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="+  }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="metadata.annotation are different" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="-  field.cattle.io/publicEndpoints: [{\\addresses\\:[\\10.133.131.150\\],\\port\\:31169,\\protocol\\:\\TCP\\,\\serviceName\\:\ affo-test-validation-staging:dev-postgres-custom-svc\\,\\allNodes\\:true}]," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=info msg="reason: new statefulset's annotations does not match the current one" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="updating statefulset" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="patching statefulset annotations" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:05Z" level=debug msg="patching statefulset annotations" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:06Z" level=debug msg="calling Patroni API on a pod affo-test-validation-staging/z-postgres-0 to set the following Postgres options: map[max_connections:200 max_worker_processes:48 wal_level:logical]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:06Z" level=debug msg="making PATCH http request: http://10.42.19.70:8008/config" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:06Z" level=debug msg="syncing pod disruption budgets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:06Z" level=debug msg="syncing roles" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:06Z" level=warning msg="could not connect to Postgres database: dial tcp 10.43.252.153:5432: connect: connection refused" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:21Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T21:33:21Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T22:02:59Z" level=info msg="SYNC event has been queued" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T22:02:59Z" level=info msg="syncing of the cluster started" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T22:02:59Z" level=debug msg="fetching possible additional team members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:02:59Z" level=debug msg="team API is disabled, only returning 0 members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:02:59Z" level=info msg="syncing secrets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:02:59Z" level=debug msg="secret affo-test-validation-staging/bar-owner-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:02:59Z" level=debug msg="secret affo-test-validation-staging/bar-reader-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:02:59Z" level=debug msg="secret affo-test-validation-staging/zalando.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:00Z" level=debug msg="secret affo-test-validation-staging/postgres.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:01Z" level=debug msg="secret affo-test-validation-staging/standby.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:02Z" level=debug msg="secret affo-test-validation-staging/bar-writer-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:02Z" level=debug msg="syncing master service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:03Z" level=debug msg="syncing replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:03Z" level=debug msg="No load balancer created for the replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:03Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:04Z" level=info msg="volume claims do not require changes" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:04Z" level=debug msg="syncing statefulsets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:04Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="[{SCOPE z-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.z-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.z-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"archive_mode\":\"true\",\"archive_timeout\":\"60\",\"autovacuum_analyze_scale_factor\":\"0.02\",\"autovacuum_max_workers\":\"12\",\"autovacuum_vacuum_cost_delay\":\"5ms\",\"autovacuum_vacuum_scale_factor\":\"0.05\",\"checkpoint_completion_target\":\"0.8\",\"checkpoint_timeout\":\"30min\",\"cron_database_name\":\"postgres\",\"default_statistics_target\":\"100\",\"effective_cache_size\":\"36GB\",\"effective_io_concurrency\":\"200\",\"log_statement\":\"all\",\"maintenance_work_mem\":\"2GB\",\"max_parallel_maintenance_workers\":\"4\",\"max_parallel_workers\":\"48\",\"max_parallel_workers_per_gather\":\"12\",\"max_wal_senders\":\"90\",\"max_wal_size\":\"8GB\",\"min_wal_size\":\"2GB\",\"random_page_cost\":\"1.1\",\"shared_buffers\":\"12GB\",\"shared_preload_libraries\":\"pg_cron\",\"vacuum_cost_limit\":\"2400\",\"wal_buffers\":\"16MB\",\"wal_keep_segments\":\"32\",\"work_mem\":\"7860kB\"}},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"200\",\"max_worker_processes\":\"48\",\"wal_level\":\"logical\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {AWS_ACCESS_KEY_ID  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:s3-zalando-secret,},Key:AWS_ACCESS_KEY_ID,Optional:nil,},}} {AWS_ENDPOINT https://bh-dc-s3-dhn-15.eecloud.nsn-net.net nil} {AWS_SECRET_ACCESS_KEY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:s3-zalando-secret,},Key:AWS_SECRET_ACCESS_KEY,Optional:nil,},}} {BACKUP_NUM_TO_RETAIN 2 nil} {BACKUP_SCHEDULE 0 */4 * * * nil} {WAL_S3_BUCKET affo-test-validation/staging/backup/postgres nil}]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=info msg="statefulset affo-test-validation-staging/z-postgres is not in the desired state and needs to be updated" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-          }," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="+          }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      restartPolicy: Always," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      securityContext: {}," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="+      securityContext: {}" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      apiVersion: v1," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      status: {" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-        phase: Pending" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-      }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="+      status: {}" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-  }," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="+  }" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="metadata.annotation are different" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="-  field.cattle.io/publicEndpoints: [{\\addresses\\:[\\10.133.131.150\\],\\port\\:31169,\\protocol\\:\\TCP\\,\\serviceName\\:\ affo-test-validation-staging:dev-postgres-custom-svc\\,\\allNodes\\:true}]," cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=info msg="reason: new statefulset's annotations does not match the current one" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="updating statefulset" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="patching statefulset annotations" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:05Z" level=debug msg="patching statefulset annotations" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:06Z" level=debug msg="calling Patroni API on a pod affo-test-validation-staging/z-postgres-0 to set the following Postgres options: map[max_connections:200 max_worker_processes:48 wal_level:logical]" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:06Z" level=debug msg="making PATCH http request: http://10.42.19.70:8008/config" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:06Z" level=debug msg="syncing pod disruption budgets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:06Z" level=debug msg="syncing roles" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:06Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:03:06Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: the database system is starting up" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T22:32:59Z" level=info msg="SYNC event has been queued" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T22:32:59Z" level=info msg="syncing of the cluster started" cluster-name=affo-test-validation-staging/z-postgres pkg=controller worker=0
time="2021-05-16T22:32:59Z" level=debug msg="fetching possible additional team members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:32:59Z" level=debug msg="team API is disabled, only returning 0 members for team \"z\"" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:32:59Z" level=info msg="syncing secrets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:32:59Z" level=debug msg="secret affo-test-validation-staging/bar-reader-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:32:59Z" level=debug msg="secret affo-test-validation-staging/postgres.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:00Z" level=debug msg="secret affo-test-validation-staging/standby.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:00Z" level=debug msg="secret affo-test-validation-staging/bar-owner-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:01Z" level=debug msg="secret affo-test-validation-staging/zalando.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:02Z" level=debug msg="secret affo-test-validation-staging/bar-writer-user.z-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:02Z" level=debug msg="syncing master service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:03Z" level=debug msg="syncing replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:04Z" level=debug msg="No load balancer created for the replica service" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:04Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:04Z" level=info msg="volume claims do not require changes" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:04Z" level=debug msg="syncing statefulsets" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0
time="2021-05-16T22:33:04Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=affo-test-validation-staging/z-postgres pkg=cluster worker=0

Can you please help me out to resolve this issue i have tried to find out solution but not yet up to mark

FxKu commented 3 years ago

Your Postgres seems to got stuck and is not running. The operator logs are not very helpful here. Exec into the pod and check the Postgres logs (under $PGLOG). Check also patroni logs kubectl logs on the database pod(s). There you should find the reason to the problem.

vishantsolanki commented 3 years ago

Hi, FxKu

Again i have faced same issue in our postgres cluster sysnc failed due to my production impacted can you please help me out where it got stuck i have checked logs but didn't able to found any concrete reason.

Operator Logs: image

Postgres Pod Logs: image

FxKu commented 3 years ago

@vishantsolanki please, check the Postgres logs in the pod for errors on why it's not starting. And if this is for your production system, go with at least two pods so that there's always a replica to switch over to.

hnusttyj530 commented 3 years ago

I also encountered the same problem, and there will be a core dump in the pod data directory.Check the contents of postgresql-5.csv and postgresql-5.log in the pg_log directory, and the following errors appear respectively:

log

2021-07-02 03:01:24 UTC [24347]: [1-1] 60de8182.5f1b 0 LOG: could not rename file "current_logfiles.tmp" to "current_logfiles": No such file or directory

csv

ERROR:replication slot "acid_upgrade_test_1" does not exist
FATAL:could not start WAL streaming: ERROR: replication slot "acid_upgrade_test_1" does not exist