PG cluster status is ConditionsError for about 800 seconds after pod is Running
➜ ~ k describe pod postgres-2g757-postgresql-0
Name: postgres-2g757-postgresql-0
Namespace: default
Priority: 0
Node: gke-cluster-1-default-pool-56d1d4e0-fttg/10.128.0.49
Start Time: Mon, 04 Sep 2023 17:16:39 +0800
Labels: app.kubernetes.io/component=postgresql
app.kubernetes.io/instance=postgres-2g757
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=postgresql
app.kubernetes.io/version=postgresql-14.7.2
apps.kubeblocks.io/component-name=postgresql
apps.kubeblocks.io/workload-type=Replication
apps.kubeblocks.postgres.patroni/role=master
apps.kubeblocks.postgres.patroni/scope=postgres-2g757-postgresql-patroni5f2bfe19
controller-revision-hash=postgres-2g757-postgresql-79fcff654c
kubeblocks.io/role=primary
statefulset.kubernetes.io/pod-name=postgres-2g757-postgresql-0
Annotations: apps.kubeblocks.io/component-replicas: 1
apps.kubeblocks.io/last-role-changed-event-timestamp: 2023-09-04T09:17:28Z
rs.apps.kubeblocks.io/primary: postgres-2g757-postgresql-0
status:
{"conn_url":"postgres://10.4.0.196:5432/postgres","api_url":"http://10.4.0.196:8008/patroni","state":"running","role":"master","version":"...
Status: Running
IP: 10.4.0.196
IPs:
IP: 10.4.0.196
Controlled By: StatefulSet/postgres-2g757-postgresql
Init Containers:
pg-init-container:
Container ID: containerd://d458d65e9698d480f2f8893de6e457dc015076643f7e6e4771e8dcbbff8ec3c5
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo@sha256:796331813dec29254a87faefeca0b3a2c9c4ac3c462f89d17a06c2d4ecde39c7
Port: <none>
Host Port: <none>
Command:
/kb-scripts/init_container.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 04 Sep 2023 17:16:50 +0800
Finished: Mon, 04 Sep 2023 17:16:50 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
postgres-2g757-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgres-2g757-postgresql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgres-2g757
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgres-2g757-postgresql
KB_CLUSTER_UID_POSTFIX_8: 5f2bfe19
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
Mounts:
/home/postgres/conf from postgresql-config (rw)
/home/postgres/pgdata from data (rw)
/kb-podinfo from pod-info (rw)
/kb-scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
Containers:
postgresql:
Container ID: containerd://a31c3bbf43cf6c4921030d17f0d7a98fed908ff9715bd25d577c00add9fa0767
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo@sha256:796331813dec29254a87faefeca0b3a2c9c4ac3c462f89d17a06c2d4ecde39c7
Ports: 5432/TCP, 8008/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/kb-scripts/setup.sh
State: Running
Started: Mon, 04 Sep 2023 17:16:53 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 102Mi
Readiness: exec [/bin/sh -c -ee exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
[ -f /postgresql/tmp/.initialized ] || [ -f /postgresql/.initialized ]
] delay=10s timeout=5s period=30s #success=1 #failure=3
Environment Variables from:
postgres-2g757-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgres-2g757-postgresql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgres-2g757
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgres-2g757-postgresql
KB_CLUSTER_UID_POSTFIX_8: 5f2bfe19
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
DCS_ENABLE_KUBERNETES_API: true
KUBERNETES_USE_CONFIGMAPS: true
SCOPE: $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni$(KB_CLUSTER_UID_POSTFIX_8)
KUBERNETES_SCOPE_LABEL: apps.kubeblocks.postgres.patroni/scope
KUBERNETES_ROLE_LABEL: apps.kubeblocks.postgres.patroni/role
KUBERNETES_LABELS: {"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)"}
RESTORE_DATA_DIR: /home/postgres/pgdata/kb_restore
KB_PG_CONFIG_PATH: /home/postgres/conf/postgresql.conf
SPILO_CONFIGURATION: bootstrap:
initdb:
- auth-host: md5
- auth-local: trust
ALLOW_NOSSL: true
PGROOT: /home/postgres/pgdata/pgroot
POD_IP: (v1:status.podIP)
POD_NAMESPACE: default (v1:metadata.namespace)
PGUSER_SUPERUSER: <set to the key 'username' in secret 'postgres-2g757-conn-credential'> Optional: false
PGPASSWORD_SUPERUSER: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
PGUSER_ADMIN: superadmin
PGPASSWORD_ADMIN: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
PGUSER_STANDBY: standby
PGPASSWORD_STANDBY: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
PGUSER: <set to the key 'username' in secret 'postgres-2g757-conn-credential'> Optional: false
PGPASSWORD: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
Mounts:
/dev/shm from dshm (rw)
/home/postgres/conf from postgresql-config (rw)
/home/postgres/pgdata from data (rw)
/kb-podinfo from pod-info (rw)
/kb-scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
pgbouncer:
Container ID: containerd://4cd7d4b99ee3472f0372811fed403f5cfc69a74e113293660f4e4f8659855ab9
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer@sha256:c788fda7436d178e6ea94cf7b8fa767005f0ef483edd90b328c560e1935b26b5
Port: 6432/TCP
Host Port: 0/TCP
Command:
/kb-scripts/pgbouncer_setup.sh
State: Running
Started: Mon, 04 Sep 2023 17:16:53 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Liveness: tcp-socket :tcp-pgbouncer delay=15s timeout=5s period=30s #success=1 #failure=3
Readiness: tcp-socket :tcp-pgbouncer delay=15s timeout=5s period=30s #success=1 #failure=3
Environment Variables from:
postgres-2g757-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgres-2g757-postgresql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgres-2g757
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgres-2g757-postgresql
KB_CLUSTER_UID_POSTFIX_8: 5f2bfe19
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
PGBOUNCER_AUTH_TYPE: md5
POSTGRESQL_USERNAME: <set to the key 'username' in secret 'postgres-2g757-conn-credential'> Optional: false
POSTGRESQL_PASSWORD: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
POSTGRESQL_PORT: 5432
POSTGRESQL_HOST: (v1:status.podIP)
PGBOUNCER_PORT: 6432
PGBOUNCER_BIND_ADDRESS: 0.0.0.0
Mounts:
/home/pgbouncer/conf from pgbouncer-config (rw)
/kb-scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
metrics:
Container ID: containerd://33e24a2285a81a91a5012a182a2960a6f76c24d38271cccdc37664e34b1906d0
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
Port: 9187/TCP
Host Port: 0/TCP
Command:
/bin/agamotto
--config=/opt/agamotto/agamotto-config.yaml
State: Running
Started: Mon, 04 Sep 2023 17:16:55 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
postgres-2g757-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgres-2g757-postgresql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgres-2g757
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgres-2g757-postgresql
KB_CLUSTER_UID_POSTFIX_8: 5f2bfe19
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
ENDPOINT: 127.0.0.1:5432
DATA_SOURCE_PASS: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
DATA_SOURCE_USER: <set to the key 'username' in secret 'postgres-2g757-conn-credential'> Optional: false
Mounts:
/opt/agamotto from agamotto-configuration (rw)
/opt/conf from postgresql-custom-metrics (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
kb-checkrole:
Container ID: containerd://c17f0a5b8b2424f808b4bc90745f2eb495aacef25b7aca71912685399cac1106
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6760e31e4094714bbe126134d43d19c7d4bd82c1619695908c1c6c696474cb9f
Ports: 3501/TCP, 50001/TCP
Host Ports: 0/TCP, 0/TCP
Command:
probe
--app-id
batch-sdk
--dapr-http-port
3501
--dapr-grpc-port
50001
--log-level
info
--config
/config/probe/config.yaml
--components-path
/config/probe/components
State: Running
Started: Mon, 04 Sep 2023 17:16:56 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Readiness: http-get http://:3501/v1.0/bindings/postgresql%3Foperation=checkRole&workloadType=Replication delay=0s timeout=1s period=1s #success=1 #failure=2
Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
postgres-2g757-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgres-2g757-postgresql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgres-2g757
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgres-2g757-postgresql
KB_CLUSTER_UID_POSTFIX_8: 5f2bfe19
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
KB_SERVICE_USER: <set to the key 'username' in secret 'postgres-2g757-conn-credential'> Optional: false
KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'postgres-2g757-conn-credential'> Optional: false
KB_SERVICE_PORT: 5432
KB_SERVICE_ROLES: {}
KB_SERVICE_CHARACTER_TYPE: postgresql
KB_WORKLOAD_TYPE: Replication
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
config-manager:
Container ID: containerd://ee803697fd6060e4cac6ab3b54d203253fcf567c7ce72cc8d60fff22c0349124
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6760e31e4094714bbe126134d43d19c7d4bd82c1619695908c1c6c696474cb9f
Port: <none>
Host Port: <none>
Command:
env
Args:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
/bin/reloader
--log-level
info
--operator-update-enable
--tcp
9901
--config
/opt/config-manager/config-manager.yaml
State: Running
Started: Mon, 04 Sep 2023 17:16:56 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
postgres-2g757-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgres-2g757-postgresql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgres-2g757
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgres-2g757-postgresql
KB_CLUSTER_UID_POSTFIX_8: 5f2bfe19
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
CONFIG_MANAGER_POD_IP: (v1:status.podIP)
DB_TYPE: postgresql
TOOLS_PATH: /opt/kb-tools/reload/postgresql-configuration:/opt/config-manager
Mounts:
/home/postgres/conf from postgresql-config (rw)
/opt/config-manager from config-manager-config (rw)
/opt/kb-tools/reload/postgresql-configuration from cm-script-postgresql-configuration (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-postgres-2g757-postgresql-0
ReadOnly: false
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
pod-info:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels['kubeblocks.io/role'] -> pod-role
metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod
metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
agamotto-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgres-2g757-postgresql-agamotto-configuration
Optional: false
pgbouncer-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgres-2g757-postgresql-pgbouncer-configuration
Optional: false
postgresql-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgres-2g757-postgresql-postgresql-configuration
Optional: false
postgresql-custom-metrics:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgres-2g757-postgresql-postgresql-custom-metrics
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgres-2g757-postgresql-postgresql-scripts
Optional: false
cm-script-postgresql-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-patroni-reload-script-postgres-2g757
Optional: false
config-manager-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-postgres-2g757-postgresql-config-manager-config
Optional: false
kube-api-access-fcqkm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27m default-scheduler Successfully assigned default/postgres-2g757-postgresql-0 to gke-cluster-1-default-pool-56d1d4e0-fttg
Warning FailedMount 27m kubelet MountVolume.SetUp failed for volume "pgbouncer-config" : failed to sync configmap cache: timed out waiting for the condition
Normal Pulled 27m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2" already present on machine
Normal Created 27m kubelet Created container pg-init-container
Normal Started 27m kubelet Started container pg-init-container
Normal Pulled 27m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2" already present on machine
Normal Created 27m kubelet Created container postgresql
Normal Started 27m kubelet Started container postgresql
Normal Pulled 27m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0" already present on machine
Normal Created 27m kubelet Created container pgbouncer
Normal Started 27m kubelet Started container pgbouncer
Normal Pulled 27m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" already present on machine
Normal Created 27m kubelet Created container metrics
Normal Started 27m kubelet Started container metrics
Normal Pulled 27m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1" already present on machine
Normal Created 27m kubelet Created container kb-checkrole
Normal Started 27m kubelet Started container kb-checkrole
Normal Pulled 27m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1" already present on machine
Normal Created 27m kubelet Created container config-manager
Normal Started 27m kubelet Started container config-manager
Normal checkRole 27m sqlchannel {"event":"Failed","message":"error executing select pg_is_in_recovery();: failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)","operation":"checkRole","originalRole":""}
Warning Unhealthy 26m (x3 over 26m) kubelet Readiness probe failed: 127.0.0.1:5432 - no response
Normal checkRole 26m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"","role":"secondary"}
Normal checkRole 26m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
➜ ~ k describe cluster postgres-2g757
Name: postgres-2g757
Namespace: default
Labels: clusterdefinition.kubeblocks.io/name=postgresql
clusterversion.kubeblocks.io/name=postgresql-14.7.2
Annotations: kubeblocks.io/reconcile: 2023-09-04T09:16:39.574547783Z
API Version: apps.kubeblocks.io/v1alpha1
Kind: Cluster
Metadata:
Creation Timestamp: 2023-09-04T08:12:18Z
Finalizers:
cluster.kubeblocks.io/finalizer
Generate Name: postgres-
Generation: 1
Managed Fields:
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:generateName:
f:labels:
.:
f:clusterdefinition.kubeblocks.io/name:
f:clusterversion.kubeblocks.io/name:
f:spec:
.:
f:affinity:
.:
f:nodeLabels:
f:podAntiAffinity:
f:tenancy:
f:topologyKeys:
f:clusterDefinitionRef:
f:clusterVersionRef:
f:componentSpecs:
.:
k:{"name":"postgresql"}:
.:
f:componentDefRef:
f:monitor:
f:name:
f:replicas:
f:resources:
.:
f:limits:
.:
f:cpu:
f:memory:
f:requests:
.:
f:cpu:
f:memory:
f:serviceAccountName:
f:switchPolicy:
.:
f:type:
f:volumeClaimTemplates:
f:terminationPolicy:
f:tolerations:
Manager: kubectl-create
Operation: Update
Time: 2023-09-04T08:12:18Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:clusterDefGeneration:
f:components:
.:
f:postgresql:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:replicationSetStatus:
.:
f:primary:
.:
f:pod:
f:conditions:
f:observedGeneration:
f:phase:
Manager: manager
Operation: Update
Subresource: status
Time: 2023-09-04T09:07:29Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubeblocks.io/reconcile:
f:finalizers:
.:
v:"cluster.kubeblocks.io/finalizer":
Manager: manager
Operation: Update
Time: 2023-09-04T09:16:39Z
Resource Version: 442678
UID: fba0a375-be2a-42f3-aa94-ebde5f2bfe19
Spec:
Affinity:
Node Labels:
Pod Anti Affinity: Preferred
Tenancy: SharedNode
Topology Keys:
Cluster Definition Ref: postgresql
Cluster Version Ref: postgresql-14.7.2
Component Specs:
Component Def Ref: postgresql
Monitor: true
Name: postgresql
No Create PDB: false
Replicas: 1
Resources:
Limits:
Cpu: 1000m
Memory: 1024Mi
Requests:
Cpu: 100m
Memory: 102Mi
Service Account Name: dbname
Switch Policy:
Type: Noop
Volume Claim Templates:
Name: data
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 3Gi
Storage Class Name: standard-rwo
Termination Policy: WipeOut
Tolerations:
Status:
Cluster Def Generation: 2
Components:
Postgresql:
Phase: Running
Pods Ready: true
Pods Ready Time: 2023-09-04T08:46:41Z
Replication Set Status:
Primary:
Pod: postgres-2g757-postgresql-0
Conditions:
Last Transition Time: 2023-09-04T08:12:49Z
Message: The operator has started the provisioning of Cluster: postgres-2g757
Observed Generation: 1
Reason: PreCheckSucceed
Status: True
Type: ProvisioningStarted
Last Transition Time: 2023-09-04T09:07:27Z
Message: Successfully applied for resources
Observed Generation: 1
Reason: ApplyResourcesSucceed
Status: True
Type: ApplyResources
Last Transition Time: 2023-09-04T08:14:04Z
Message: all pods of components are ready, waiting for the probe detection successful
Reason: AllReplicasReady
Status: True
Type: ReplicasReady
Last Transition Time: 2023-09-04T08:14:04Z
Message: Cluster: postgres-2g757 is ready, current phase is Running
Reason: ClusterReady
Status: True
Type: Ready
Observed Generation: 1
Phase: Running
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ApplyResourcesFailed 51m cluster-controller Operation cannot be fulfilled on statefulsets.apps "postgres-2g757-postgresql": the object has been modified; please apply your changes to the latest version and try again
Normal ApplyResourcesSucceed 37m cluster-controller Successfully applied for resources
Normal WaitingForProbeSuccess 28m (x2 over 51m) cluster-controller Waiting for probe success
Warning RecreatingFailedPod 28m event-controller StatefulSet default/postgres-2g757-postgresql is recreating failed Pod postgres-2g757-postgresql-0
➜ ~ k logs postgres-2g757-postgresql-0
Defaulted container "postgresql" out of: postgresql, pgbouncer, metrics, kb-checkrole, config-manager, pg-init-container (init)
2023-09-04 09:17:23,483 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2023-09-04 09:17:23,494 - bootstrapping - INFO - Looks like you are running google
2023-09-04 09:17:23,518 - bootstrapping - INFO - kubeblocks generate local configuration:
bootstrap:
dcs:
postgresql:
parameters:
archive_command: /bin/true
archive_mode: 'on'
autovacuum_analyze_scale_factor: '0.05'
autovacuum_max_workers: '1'
autovacuum_vacuum_scale_factor: '0.1'
checkpoint_completion_target: '0.95'
hot_standby: 'on'
log_autovacuum_min_duration: 1s
log_checkpoints: 'True'
log_lock_waits: 'True'
log_min_duration_statement: '100'
log_statement: ddl
max_connections: '112'
max_locks_per_transaction: '128'
max_prepared_transactions: '0'
max_replication_slots: '16'
max_wal_senders: '24'
max_worker_processes: '8'
track_commit_timestamp: 'True'
track_functions: all
wal_keep_size: 20GB
wal_level: replica
wal_log_hints: 'on'
initdb:
- auth-host: md5
- auth-local: trust
postgresql:
config_dir: /home/postgres/pgdata/conf
custom_conf: /home/postgres/conf/postgresql.conf
parameters:
pg_stat_statements.track_utility: 'False'
shared_buffers: 256MB
pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- local replication all trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5
2023-09-04 09:17:23,532 - bootstrapping - INFO - Configuring crontab
2023-09-04 09:17:23,532 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2023-09-04 09:17:23,532 - bootstrapping - INFO - Configuring bootstrap
2023-09-04 09:17:23,532 - bootstrapping - INFO - Configuring standby-cluster
2023-09-04 09:17:23,533 - bootstrapping - INFO - Configuring certificate
2023-09-04 09:17:23,533 - bootstrapping - INFO - Generating ssl self-signed certificate
2023-09-04 09:17:23,823 - bootstrapping - INFO - Configuring pgqd
2023-09-04 09:17:23,823 - bootstrapping - INFO - Configuring pam-oauth2
2023-09-04 09:17:23,824 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring pgbouncer
2023-09-04 09:17:23,824 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring wal-e
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring log
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring patroni
2023-09-04 09:17:23,833 - bootstrapping - INFO - Writing to file /run/postgres.yml
2023-09-04 09:17:25,516 INFO: Selected new K8s API server endpoint https://10.128.0.48:443
2023-09-04 09:17:25,783 INFO: No PostgreSQL configuration items changed, nothing to reload.
2023-09-04 09:17:25,790 WARNING: Postgresql is not running.
2023-09-04 09:17:25,790 INFO: Lock owner: None; I am postgres-2g757-postgresql-0
2023-09-04 09:17:25,791 INFO: pg_controldata:
pg_control version number: 1300
Catalog version number: 202107181
Database system identifier: 7274880837373194364
Database cluster state: shut down
pg_control last modified: Mon Sep 4 09:16:25 2023
Latest checkpoint location: 0/4000028
Latest checkpoint's REDO location: 0/4000028
Latest checkpoint's REDO WAL file: 000000020000000000000004
Latest checkpoint's TimeLineID: 2
Latest checkpoint's PrevTimeLineID: 2
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID: 0:1287
Latest checkpoint's NextOID: 16818
Latest checkpoint's NextMultiXactId: 1
Latest checkpoint's NextMultiOffset: 0
Latest checkpoint's oldestXID: 727
Latest checkpoint's oldestXID's DB: 1
Latest checkpoint's oldestActiveXID: 0
Latest checkpoint's oldestMultiXid: 1
Latest checkpoint's oldestMulti's DB: 1
Latest checkpoint's oldestCommitTsXid: 734
Latest checkpoint's newestCommitTsXid: 1286
Time of latest checkpoint: Mon Sep 4 09:16:25 2023
Fake LSN counter for unlogged rels: 0/3E8
Minimum recovery ending location: 0/0
Min recovery ending loc's timeline: 0
Backup start location: 0/0
Backup end location: 0/0
End-of-backup record required: no
wal_level setting: replica
wal_log_hints setting: on
max_connections setting: 112
max_worker_processes setting: 8
max_wal_senders setting: 24
max_prepared_xacts setting: 0
max_locks_per_xact setting: 128
track_commit_timestamp setting: on
Maximum data alignment: 8
Database block size: 8192
Blocks per segment of large relation: 131072
WAL block size: 8192
Bytes per WAL segment: 16777216
Maximum length of identifiers: 64
Maximum columns in an index: 32
Maximum size of a TOAST chunk: 1996
Size of a large-object chunk: 2048
Date/time type storage: 64-bit integers
Float8 argument passing: by value
Data page checksum version: 0
Mock authentication nonce: 7f0a2c2e5bd8aa26aded1b3e7c036431e75d1960203084f0900ef2fe3cb69fe9
2023-09-04 09:17:25,799 INFO: Lock owner: None; I am postgres-2g757-postgresql-0
2023-09-04 09:17:26,054 INFO: starting as a secondary
2023-09-04 09:17:26 GMT [91]: [1-1] 64f5a0a6.5b 0 LOG: Auto detecting pg_stat_kcache.linux_hz parameter...
2023-09-04 09:17:26 GMT [91]: [2-1] 64f5a0a6.5b 0 LOG: pg_stat_kcache.linux_hz is set to 500000
2023-09-04 09:17:26,305 INFO: postmaster pid=91
/var/run/postgresql:5432 - no response
2023-09-04 09:17:26 GMT [91]: [3-1] 64f5a0a6.5b 0 LOG: redirecting log output to logging collector process
2023-09-04 09:17:26 GMT [91]: [4-1] 64f5a0a6.5b 0 HINT: Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2023-09-04 09:17:27,347 INFO: establishing a new patroni connection to the postgres cluster
2023-09-04 09:17:27,433 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2023-09-04 09:17:28,079 INFO: promoted self to leader by acquiring session lock
server promoting
2023-09-04 09:17:28,081 INFO: cleared rewind state after becoming the leader
DO
NOTICE: role "admin" is already a member of role "cron_admin"
GRANT ROLE
DO
DO
NOTICE: extension "pg_auth_mon" already exists, skipping
CREATE EXTENSION
NOTICE: version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
NOTICE: extension "pg_cron" already exists, skipping
CREATE EXTENSION
DO
NOTICE: version "1.5" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
REVOKE
GRANT
ALTER POLICY
REVOKE
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
NOTICE: extension "file_fdw" already exists, skipping
CREATE EXTENSION
DO
NOTICE: relation "postgres_log" already exists, skipping
CREATE TABLE
GRANT
NOTICE: column "backend_type" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE: column "leader_pid" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE: column "query_id" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE: relation "postgres_log_0" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_1" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_2" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_3" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_4" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_5" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_6" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE: relation "postgres_log_7" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE: drop cascades to 5 other objects
DETAIL: drop cascades to type zmon_utils.system_information
drop cascades to function zmon_utils.get_database_cluster_information()
drop cascades to function zmon_utils.get_database_cluster_system_information()
drop cascades to function zmon_utils.get_last_status_active_cronjobs()
drop cascades to view zmon_utils.last_status_active_cronjobs
DROP SCHEMA
NOTICE: extension "plpython3u" already exists, skipping
DO
NOTICE: language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE: function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE: function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE: function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
REVOKE
CREATE VIEW
REVOKE
GRANT
GRANT
You are now connected to database "postgres" as user "postgres".
NOTICE: schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
NOTICE: extension "pg_stat_statements" already exists, skipping
CREATE EXTENSION
NOTICE: extension "pg_stat_kcache" already exists, skipping
CREATE EXTENSION
NOTICE: extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE: version "3.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE: schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
You are now connected to database "template1" as user "postgres".
NOTICE: schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
NOTICE: extension "pg_stat_statements" already exists, skipping
CREATE EXTENSION
NOTICE: extension "pg_stat_kcache" already exists, skipping
CREATE EXTENSION
NOTICE: extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE: version "3.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE: schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
2023-09-04 09:17:29,917 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:17:41,173 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:17:50,012 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:17:59,737 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:09,861 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:19,849 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:24.902 UTC [60] LOG Starting pgqd 3.5
2023-09-04 09:18:24.902 UTC [60] LOG auto-detecting dbs ...
2023-09-04 09:18:30,361 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:40,085 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:51,458 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:54.932 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:18:59,653 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:09,720 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:20,911 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:24.937 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:19:30,140 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:39,858 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:50,245 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:54.966 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:19:59,896 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:10,038 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:19,835 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:24.971 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:20:31,033 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:40,096 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:51,082 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:55.005 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:20:59,887 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:10,007 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:21,008 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:25.006 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:21:29,765 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:40,422 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:50,094 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:55.036 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:21:59,747 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:09,830 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:20,097 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:25.047 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:22:30,209 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:39,853 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:49,807 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:55.077 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:22:59,720 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:10,020 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:19,848 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:25.082 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:23:30,012 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:39,955 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:51,392 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:55.111 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:23:59,825 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:09,955 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:20,586 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:25.117 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:24:29,707 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:39,791 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:50,486 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:55.146 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:24:59,901 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:11,196 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:19,769 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:25.151 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:25:30,007 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:39,815 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:49,970 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:55.180 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:26:00,151 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:09,759 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:19,766 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:25.185 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:26:29,654 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:39,728 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:49,768 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:55.215 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:26:59,851 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:09,770 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:19,774 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:25.220 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:27:29,768 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:40,045 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:49,851 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:55.250 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:27:59,720 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:10,547 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:20,069 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:25.259 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:28:29,966 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:39,762 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:49,971 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:55.288 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:28:59,663 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:09,884 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:21,186 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:25.294 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:29:30,431 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:40,693 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:50,708 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:55.323 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:30:00,522 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:10,499 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:19,831 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:25.329 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:30:29,922 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:39,952 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:50,410 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:55.359 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:30:59,654 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:09,741 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:19,713 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:25.363 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:31:29,707 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:39,745 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:49,854 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:55.392 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:32:00,071 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:09,741 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:20,392 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:25.401 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:32:30,275 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:39,675 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:50,035 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:55.431 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:33:00,034 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:10,070 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:20,751 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:25.437 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:33:30,546 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:39,838 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:50,362 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:55.467 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:33:59,929 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:09,789 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:19,890 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:25.489 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:34:30,090 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:39,925 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:49,876 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:55.519 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:34:59,722 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:09,804 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:19,862 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:25.524 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:35:29,984 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:40,091 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:49,675 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:55.553 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:35:59,788 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:12,913 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:20,957 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:25.559 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:36:29,824 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:40,283 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:49,671 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:55.588 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:36:59,827 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:09,784 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:20,169 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:25.593 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:37:30,003 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:39,743 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:49,883 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:55.622 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:37:59,815 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:09,898 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:21,365 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:25.626 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:38:30,278 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:40,228 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:50,733 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock