apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.11k stars 174 forks source link

[BUG]postgresql containers unready after restart after upgrade kb from 0.7.2 to 0.8.0 #6259

Closed ahjing99 closed 4 months ago

ahjing99 commented 10 months ago
  1. Create cluster
  2. upgrade kbcli and kubeblocks
    
    ➜  ~ kbcli kubeblocks upgrade --version 0.8.0-beta.28
    Current KubeBlocks version 0.7.2.
    Kubernetes version 1.27.3
    Kubernetes provider GKE
    kbcli version 0.8.0-beta.28
    Upgrade KubeBlocks from 0.7.2 to 0.8.0-beta.28
    Please type 'Yes/yes' to confirm your operation: yes
    Add and update repo kubeblocks                     OK
    Stop KubeBlocks 0.7.2                              OK
    Stop DataProtection                                OK
    Upgrading KubeBlocks to 0.8.0-beta.28              OK

KubeBlocks has been upgraded to 0.8.0-beta.28 SUCCESSFULLY!

-> Basic commands for cluster: kbcli cluster create -h # help information about creating a database cluster kbcli cluster list # list all database clusters kbcli cluster describe # get cluster information

-> Uninstall KubeBlocks: kbcli kubeblocks uninstall

3. Stop/Start and Restart, Pg cannot started

kbcli cluster stop postgres-ivtqbv --auto-approve --namespace default

OpsRequest postgres-ivtqbv-stop-zw9m4 created successfully, you can view the progress: kbcli cluster describe-ops postgres-ivtqbv-stop-zw9m4 -n default

kbcli cluster start postgres-ivtqbv --namespace default

OpsRequest postgres-ivtqbv-start-nx4dn created successfully, you can view the progress: kbcli cluster describe-ops postgres-ivtqbv-start-nx4dn -n default

kbcli cluster restart postgres-ivtqbv --auto-approve --namespace default

OpsRequest postgres-ivtqbv-restart-ktr4q created successfully, you can view the progress: kbcli cluster describe-ops postgres-ivtqbv-restart-ktr4q -n default

➜ ~ k get ops | grep postgres-ivtqbv postgres-ivtqbv-restart-ktr4q Restart postgres-ivtqbv Running 1/2 32m postgres-ivtqbv-start-nx4dn Start postgres-ivtqbv Succeed 2/2 33m postgres-ivtqbv-stop-zw9m4 Stop postgres-ivtqbv Succeed 2/2 34m

➜ ~ k describe ops postgres-ivtqbv-restart-ktr4q Name: postgres-ivtqbv-restart-ktr4q Namespace: default Labels: app.kubernetes.io/instance=postgres-ivtqbv app.kubernetes.io/managed-by=kubeblocks ops.kubeblocks.io/ops-type=Restart Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: OpsRequest Metadata: Creation Timestamp: 2023-12-27T05:13:07Z Finalizers: opsrequest.kubeblocks.io/finalizer Generate Name: postgres-ivtqbv-restart- Generation: 2 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:generateName: f:labels: .: f:app.kubernetes.io/instance: f:app.kubernetes.io/managed-by: f:spec: .: f:clusterRef: f:restart: .: k:{"componentName":"postgresql"}: .: f:componentName: f:ttlSecondsBeforeAbort: f:type: Manager: kbcli Operation: Update Time: 2023-12-27T05:13:07Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"opsrequest.kubeblocks.io/finalizer": f:labels: f:ops.kubeblocks.io/ops-type: f:ownerReferences: .: k:{"uid":"d7404eb5-1d4c-4eac-a143-fa10edd7acf0"}: Manager: manager Operation: Update Time: 2023-12-27T05:13:07Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterGeneration: f:components: .: f:postgresql: .: f:phase: f:progressDetails: f:conditions: .: k:{"type":"Restarting"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: k:{"type":"Validated"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: k:{"type":"WaitForProgressing"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: f:phase: f:progress: f:startTimestamp: Manager: manager Operation: Update Subresource: status Time: 2023-12-27T05:14:48Z Owner References: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Name: postgres-ivtqbv UID: d7404eb5-1d4c-4eac-a143-fa10edd7acf0 Resource Version: 2148480 UID: 45ab893c-6a0a-4c97-9285-7ac9a07b8a3f Spec: Cluster Ref: postgres-ivtqbv Restart: Component Name: postgresql Ttl Seconds Before Abort: 0 Type: Restart Status: Cluster Generation: 3 Components: Postgresql: Phase: Updating Progress Details: End Time: 2023-12-27T05:14:48Z Message: Successfully restart: Pod/postgres-ivtqbv-postgresql-1 in Component: postgresql Object Key: Pod/postgres-ivtqbv-postgresql-1 Start Time: 2023-12-27T05:13:07Z Status: Succeed Message: Start to restart: Pod/postgres-ivtqbv-postgresql-0 in Component: postgresql Object Key: Pod/postgres-ivtqbv-postgresql-0 Start Time: 2023-12-27T05:13:08Z Status: Processing Conditions: Last Transition Time: 2023-12-27T05:13:07Z Message: wait for the controller to process the OpsRequest: postgres-ivtqbv-restart-ktr4q in Cluster: postgres-ivtqbv Reason: WaitForProgressing Status: True Type: WaitForProgressing Last Transition Time: 2023-12-27T05:13:07Z Message: OpsRequest: postgres-ivtqbv-restart-ktr4q is validated Reason: ValidateOpsRequestPassed Status: True Type: Validated Last Transition Time: 2023-12-27T05:13:07Z Message: Start to restart database in Cluster: postgres-ivtqbv Reason: RestartStarted Status: True Type: Restarting Phase: Running Progress: 1/2 Start Timestamp: 2023-12-27T05:13:07Z Events: Type Reason Age From Message


Normal WaitForProgressing 32m (x2 over 32m) ops-request-controller wait for the controller to process the OpsRequest: postgres-ivtqbv-restart-ktr4q in Cluster: postgres-ivtqbv Normal ValidateOpsRequestPassed 32m ops-request-controller OpsRequest: postgres-ivtqbv-restart-ktr4q is validated Normal RestartStarted 32m ops-request-controller Start to restart database in Cluster: postgres-ivtqbv Normal Processing 32m ops-request-controller Start to restart: Pod/postgres-ivtqbv-postgresql-1 in Component: postgresql Normal Processing 32m ops-request-controller Start to restart: Pod/postgres-ivtqbv-postgresql-0 in Component: postgresql Normal Succeed 30m ops-request-controller Successfully restart: Pod/postgres-ivtqbv-postgresql-1 in Component: postgresql ➜ ~ ➜ ~ k get pod | grep postgres-ivtqbv postgres-ivtqbv-postgresql-0 4/5 Running 0 32m postgres-ivtqbv-postgresql-1 5/5 Running 0 32m ➜ ~ k describe pod postgres-ivtqbv-postgresql-0 Name: postgres-ivtqbv-postgresql-0 Namespace: default Priority: 0 Node: gke-yjtest-default-pool-ead468f1-drlg/10.128.0.25 Start Time: Wed, 27 Dec 2023 13:13:39 +0800 Labels: app.kubernetes.io/component=postgresql app.kubernetes.io/instance=postgres-ivtqbv app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=postgresql app.kubernetes.io/version=postgresql-14.8.0 apps.kubeblocks.io/component-name=postgresql apps.kubeblocks.io/workload-type=Replication apps.kubeblocks.postgres.patroni/scope=postgres-ivtqbv-postgresql-patroniedd7acf0 controller-revision-hash=postgres-ivtqbv-postgresql-58bc6898cf statefulset.kubernetes.io/pod-name=postgres-ivtqbv-postgresql-0 Annotations: apps.kubeblocks.io/component-replicas: 2 kubeblocks.io/restart: 2023-12-27T05:13:07Z status: {"conn_url":"postgres://10.104.5.34:5432/postgres","api_url":"http://10.104.5.34:8008/patroni","state":"starting","role":"replica","versio... Status: Running IP: 10.104.5.34 IPs: IP: 10.104.5.34 Controlled By: StatefulSet/postgres-ivtqbv-postgresql Init Containers: pg-init-container: Container ID: containerd://af2450c803cc2821b87695dc1ab6586ee9b49a7720985fd3b149bdca0a150b24 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/spilo:14.8.0-pgvector-v0.5.0 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/spilo@sha256:fcb43ae7be09ee319652334369f5ced5450566772ee0a54f9da941d4a0fe4238 Port: Host Port: Command: /kb-scripts/init_container.sh State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 27 Dec 2023 13:13:53 +0800 Finished: Wed, 27 Dec 2023 13:13:53 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment: KB_NAMESPACE: default KB_CLUSTER_NAME: postgres-ivtqbv KB_CLUSTER_UID: d7404eb5-1d4c-4eac-a143-fa10edd7acf0 KB_CLUSTER_COMP_NAME: postgres-ivtqbv-postgresql KB_COMP_NAME: postgresql KB_CLUSTER_UID_POSTFIX_8: edd7acf0 KB_POD_NAME: postgres-ivtqbv-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_SA_NAME: (v1:spec.serviceAccountName) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc Mounts: /home/postgres/conf from postgresql-config (rw) /home/postgres/pgdata from data (rw) /kb-podinfo from pod-info (rw) /kb-scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hppjp (ro) Containers: postgresql: Container ID: containerd://c9f86a50e9e77af2eeb9b7317d03ea6caf3b0c908e37962809263d14f10c5452 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/spilo:14.8.0-pgvector-v0.5.0 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/spilo@sha256:fcb43ae7be09ee319652334369f5ced5450566772ee0a54f9da941d4a0fe4238 Ports: 5432/TCP, 8008/TCP Host Ports: 0/TCP, 0/TCP Command: /kb-scripts/setup.sh State: Running Started: Wed, 27 Dec 2023 13:13:54 +0800 Ready: False Restart Count: 0 Limits: cpu: 100m memory: 512Mi Requests: cpu: 100m memory: 512Mi Readiness: exec [/bin/sh -c -ee exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432 [ -f /postgresql/tmp/.initialized ] || [ -f /postgresql/.initialized ] ] delay=10s timeout=5s period=30s #success=1 #failure=3 Environment Variables from: postgres-ivtqbv-postgresql-rsm-env ConfigMap Optional: false Environment: KB_NAMESPACE: default KB_CLUSTER_NAME: postgres-ivtqbv KB_CLUSTER_UID: d7404eb5-1d4c-4eac-a143-fa10edd7acf0 KB_CLUSTER_COMP_NAME: postgres-ivtqbv-postgresql KB_COMP_NAME: postgresql KB_CLUSTER_UID_POSTFIX_8: edd7acf0 KB_POD_NAME: postgres-ivtqbv-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_SA_NAME: (v1:spec.serviceAccountName) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc SERVICE_PORT: 5432 DCS_ENABLE_KUBERNETES_API: true KUBERNETES_USE_CONFIGMAPS: true SCOPE: $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni$(KB_CLUSTER_UID_POSTFIX_8) KUBERNETES_SCOPE_LABEL: apps.kubeblocks.postgres.patroni/scope KUBERNETES_ROLE_LABEL: apps.kubeblocks.postgres.patroni/role KUBERNETES_LABELS: {"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)"} RESTORE_DATA_DIR: /home/postgres/pgdata/kb_restore KB_PG_CONFIG_PATH: /home/postgres/conf/postgresql.conf SPILO_CONFIGURATION: bootstrap: initdb:

➜ ~ kbcli report cluster postgres-ivtqbv --with-logs --all-containers reporting cluster information to report-cluster-postgres-ivtqbv-2023-12-27-13-49-09.zip processing manifests OK processing events OK process pod logs OK ➜ ~ kbcli report kubeblocks --with-logs --all-containers --output yaml reporting KubeBlocks information to report-kubeblocks-2023-12-27-13-49-31.zip processing manifests OK processing events OK process pod logs OK report-kubeblocks-2023-12-27-13-49-31.zip report-cluster-postgres-ivtqbv-2023-12-27-13-49-09.zip

Y-Rookie commented 10 months ago
root@postgres-ivtqbv-postgresql-0:/home/postgres/pgdata/pgroot# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   2676   636 ?        Ss   05:13   0:00 /usr/bin/dumb-init -c --rewrite 1:0 -- /bin/sh /launch.sh
root          48  0.0  0.0   2924   604 ?        S    05:14   0:00 /bin/sh /launch.sh
root          62  0.0  0.0   2844   904 ?        S    05:14   0:00 /usr/bin/runsvdir -P /etc/service
root          63  0.0  0.0   2692   832 ?        Ss   05:14   0:00 runsv pgqd
root          64  0.0  0.0   2692   856 ?        Ss   05:14   0:00 runsv patroni
postgres      65  0.0  0.2 566652 37204 ?        Sl   05:14   0:03 /usr/bin/python3 /usr/local/bin/patroni /home/postgres/postgres.yml
postgres      66  0.0  0.0   4360  2964 ?        S    05:14   0:00 /bin/bash /scripts/patroni_wait.sh --role master -- /usr/bin/pgqd /home/postgres/pgq_ticker.ini
postgres      97  0.2  0.3 241616 50296 ?        S    05:14   0:07 /usr/lib/postgresql/14/bin/postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/conf/postgresql.conf --listen_addresses=* --port=5432 -
postgres      99  0.0  0.0  75036  6472 ?        Ss   05:14   0:00 postgres: postgres-ivtqbv-postgresql-patroniedd7acf0: logger
postgres     100  0.0  0.0 241804  8596 ?        Ss   05:14   0:00 postgres: postgres-ivtqbv-postgresql-patroniedd7acf0: startup waiting for 000000040000000000000008
postgres     101  0.1  0.1 342148 18788 ?        Ssl  05:14   0:04 postgres: postgres-ivtqbv-postgresql-patroniedd7acf0: bg_mon
$$ LANGUAGE sql IMMUTABLE SECURITY DEFINER STRICT;",,,,,,,,,"psql","client backend",,0
2023-12-27 05:12:58.853 GMT,"postgres","template1",266,"[local]",658bb259.10a,51,"idle",2023-12-27 05:12:57 GMT,9/170,0,LOG,00000,"statement: CREATE OR REPLACE VIEW pg_stat_statements AS SELECT * FROM pg_stat_statements(true);",,,,
,,,,,"psql","client backend",,0
2023-12-27 05:12:58.858 GMT,"postgres","template1",266,"[local]",658bb259.10a,52,"idle",2023-12-27 05:12:57 GMT,9/171,0,LOG,00000,"statement: REVOKE ALL ON ALL TABLES IN SCHEMA metric_helpers FROM public;",,,,,,,,,"psql","client ba
ckend",,0
2023-12-27 05:12:58.860 GMT,"postgres","template1",266,"[local]",658bb259.10a,53,"idle",2023-12-27 05:12:57 GMT,9/172,0,LOG,00000,"statement: GRANT SELECT ON ALL TABLES IN SCHEMA metric_helpers TO admin, robot_zmon;",,,,,,,,,"psql"
,"client backend",,0
2023-12-27 05:12:58.861 GMT,"postgres","template1",266,"[local]",658bb259.10a,54,"idle",2023-12-27 05:12:57 GMT,9/173,0,LOG,00000,"statement: REVOKE ALL ON ALL FUNCTIONS IN SCHEMA metric_helpers FROM public;",,,,,,,,,"psql","client
 backend",,0
2023-12-27 05:12:58.863 GMT,"postgres","template1",266,"[local]",658bb259.10a,55,"idle",2023-12-27 05:12:57 GMT,9/174,0,LOG,00000,"statement: GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA metric_helpers TO admin, robot_zmon;",,,,,,,,,"p
sql","client backend",,0
2023-12-27 05:14:35.054 GMT,,,97,,658bb2ba.61,1,,2023-12-27 05:14:34 GMT,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster",,0
2023-12-27 05:14:35.054 GMT,,,97,,658bb2ba.61,2,,2023-12-27 05:14:34 GMT,,0,LOG,00000,"starting PostgreSQL 14.10 (Ubuntu 14.10-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit",,,,,,,,,"","postmaster",,0
2023-12-27 05:14:35.054 GMT,,,97,,658bb2ba.61,3,,2023-12-27 05:14:34 GMT,,0,LOG,00000,"listening on IPv4 address ""0.0.0.0"", port 5432",,,,,,,,,"","postmaster",,0
2023-12-27 05:14:35.054 GMT,,,97,,658bb2ba.61,4,,2023-12-27 05:14:34 GMT,,0,LOG,00000,"listening on IPv6 address ""::"", port 5432",,,,,,,,,"","postmaster",,0
2023-12-27 05:14:35.152 GMT,,,97,,658bb2ba.61,5,,2023-12-27 05:14:34 GMT,,0,LOG,00000,"listening on Unix socket ""/var/run/postgresql/.s.PGSQL.5432""",,,,,,,,,"","postmaster",,0
2023-12-27 05:14:35.158 GMT,,,100,,658bb2bb.64,1,,2023-12-27 05:14:35 GMT,,0,LOG,00000,"database system was shut down in recovery at 2023-12-27 05:12:41 GMT",,,,,,,,,"","startup",,0
2023-12-27 05:14:35.158 GMT,,,100,,658bb2bb.64,2,,2023-12-27 05:14:35 GMT,,0,WARNING,01000,"specified neither primary_conninfo nor restore_command",,"The database server will regularly poll the pg_wal subdirectory to check for files placed there.",,,,,,,"","startup",,0
2023-12-27 05:14:35.158 GMT,,,100,,658bb2bb.64,3,,2023-12-27 05:14:35 GMT,,0,LOG,00000,"entering standby mode",,,,,,,,,"","startup",,0
2023-12-27 05:14:35.677 GMT,"postgres","postgres",104,"127.0.0.1:49460",658bb2bb.68,1,"",2023-12-27 05:14:35 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 05:14:35.749 GMT,"postgres","postgres",105,"127.0.0.1:49464",658bb2bb.69,1,"",2023-12-27 05:14:35 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 05:14:36.058 GMT,"postgres","postgres",107,"[local]",658bb2bc.6b,1,"",2023-12-27 05:14:36 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 05:14:36.150 GMT,"postgres","postgres",109,"[local]",658bb2bc.6d,1,"",2023-12-27 05:14:36 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
....
....
2023-12-27 06:19:01.676 GMT,"postgres","postgres",11618,"127.0.0.1:58014",658bc1d5.2d62,1,"",2023-12-27 06:19:01 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:01.678 GMT,"postgres","postgres",11619,"127.0.0.1:58020",658bc1d5.2d63,1,"",2023-12-27 06:19:01 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:01.945 GMT,"postgres","postgres",11620,"[local]",658bc1d5.2d64,1,"",2023-12-27 06:19:01 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:02.677 GMT,"postgres","postgres",11621,"127.0.0.1:58036",658bc1d6.2d65,1,"",2023-12-27 06:19:02 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:02.678 GMT,"postgres","postgres",11622,"127.0.0.1:58040",658bc1d6.2d66,1,"",2023-12-27 06:19:02 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:03.677 GMT,"postgres","postgres",11623,"127.0.0.1:58054",658bc1d7.2d67,1,"",2023-12-27 06:19:03 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:03.679 GMT,"postgres","postgres",11624,"127.0.0.1:58066",658bc1d7.2d68,1,"",2023-12-27 06:19:03 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:04.678 GMT,"postgres","postgres",11625,"127.0.0.1:47178",658bc1d8.2d69,1,"",2023-12-27 06:19:04 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:04.679 GMT,"postgres","postgres",11626,"127.0.0.1:47190",658bc1d8.2d6a,1,"",2023-12-27 06:19:04 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:05.677 GMT,"postgres","postgres",11627,"127.0.0.1:47224",658bc1d9.2d6b,1,"",2023-12-27 06:19:05 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:05.679 GMT,"postgres","postgres",11628,"127.0.0.1:47238",658bc1d9.2d6c,1,"",2023-12-27 06:19:05 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:06.677 GMT,"postgres","postgres",11629,"127.0.0.1:47250",658bc1da.2d6d,1,"",2023-12-27 06:19:06 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:06.678 GMT,"postgres","postgres",11630,"127.0.0.1:47264",658bc1da.2d6e,1,"",2023-12-27 06:19:06 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:07.677 GMT,"postgres","postgres",11631,"127.0.0.1:47274",658bc1db.2d6f,1,"",2023-12-27 06:19:07 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:07.679 GMT,"postgres","postgres",11632,"127.0.0.1:47290",658bc1db.2d70,1,"",2023-12-27 06:19:07 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:08.677 GMT,"postgres","postgres",11633,"127.0.0.1:47294",658bc1dc.2d71,1,"",2023-12-27 06:19:08 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:08.678 GMT,"postgres","postgres",11634,"127.0.0.1:47308",658bc1dc.2d72,1,"",2023-12-27 06:19:08 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:09.676 GMT,"postgres","postgres",11635,"127.0.0.1:47318",658bc1dd.2d73,1,"",2023-12-27 06:19:09 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:09.678 GMT,"postgres","postgres",11636,"127.0.0.1:47320",658bc1dd.2d74,1,"",2023-12-27 06:19:09 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:10.148 GMT,"postgres","postgres",11637,"127.0.0.1:47324",658bc1de.2d75,1,"",2023-12-27 06:19:10 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:10.148 GMT,"postgres","postgres",11638,"127.0.0.1:47326",658bc1de.2d76,1,"",2023-12-27 06:19:10 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:10.677 GMT,"postgres","postgres",11639,"127.0.0.1:47336",658bc1de.2d77,1,"",2023-12-27 06:19:10 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:10.679 GMT,"postgres","postgres",11640,"127.0.0.1:47352",658bc1de.2d78,1,"",2023-12-27 06:19:10 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:11.145 GMT,"postgres","postgres",11642,"[local]",658bc1df.2d7a,1,"",2023-12-27 06:19:11 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:11.151 GMT,"postgres","postgres",11643,"[local]",658bc1df.2d7b,1,"",2023-12-27 06:19:11 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:11.677 GMT,"postgres","postgres",11644,"127.0.0.1:47356",658bc1df.2d7c,1,"",2023-12-27 06:19:11 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:11.679 GMT,"postgres","postgres",11645,"127.0.0.1:47370",658bc1df.2d7d,1,"",2023-12-27 06:19:11 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:11.774 GMT,"postgres","postgres",11646,"[local]",658bc1df.2d7e,1,"",2023-12-27 06:19:11 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:12.677 GMT,"postgres","postgres",11647,"127.0.0.1:47380",658bc1e0.2d7f,1,"",2023-12-27 06:19:12 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:12.679 GMT,"postgres","postgres",11648,"127.0.0.1:47388",658bc1e0.2d80,1,"",2023-12-27 06:19:12 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:13.677 GMT,"postgres","postgres",11649,"127.0.0.1:47392",658bc1e1.2d81,1,"",2023-12-27 06:19:13 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:13.679 GMT,"postgres","postgres",11650,"127.0.0.1:47402",658bc1e1.2d82,1,"",2023-12-27 06:19:13 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:14.678 GMT,"postgres","postgres",11651,"127.0.0.1:55374",658bc1e2.2d83,1,"",2023-12-27 06:19:14 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:14.679 GMT,"postgres","postgres",11652,"127.0.0.1:55380",658bc1e2.2d84,1,"",2023-12-27 06:19:14 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:15.678 GMT,"postgres","postgres",11653,"127.0.0.1:55386",658bc1e3.2d85,1,"",2023-12-27 06:19:15 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:15.680 GMT,"postgres","postgres",11654,"127.0.0.1:55394",658bc1e3.2d86,1,"",2023-12-27 06:19:15 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:16.676 GMT,"postgres","postgres",11655,"127.0.0.1:55398",658bc1e4.2d87,1,"",2023-12-27 06:19:16 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:16.678 GMT,"postgres","postgres",11656,"127.0.0.1:55412",658bc1e4.2d88,1,"",2023-12-27 06:19:16 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:17.677 GMT,"postgres","postgres",11657,"127.0.0.1:55420",658bc1e5.2d89,1,"",2023-12-27 06:19:17 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:17.679 GMT,"postgres","postgres",11658,"127.0.0.1:55430",658bc1e5.2d8a,1,"",2023-12-27 06:19:17 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:18.677 GMT,"postgres","postgres",11659,"127.0.0.1:55440",658bc1e6.2d8b,1,"",2023-12-27 06:19:18 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:18.678 GMT,"postgres","postgres",11660,"127.0.0.1:55456",658bc1e6.2d8c,1,"",2023-12-27 06:19:18 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:19.677 GMT,"postgres","postgres",11661,"127.0.0.1:55464",658bc1e7.2d8d,1,"",2023-12-27 06:19:19 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:19.679 GMT,"postgres","postgres",11662,"127.0.0.1:55470",658bc1e7.2d8e,1,"",2023-12-27 06:19:19 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:20.677 GMT,"postgres","postgres",11663,"127.0.0.1:55486",658bc1e8.2d8f,1,"",2023-12-27 06:19:20 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:20.679 GMT,"postgres","postgres",11664,"127.0.0.1:55488",658bc1e8.2d90,1,"",2023-12-27 06:19:20 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:21.168 GMT,"postgres","postgres",11666,"[local]",658bc1e9.2d92,1,"",2023-12-27 06:19:21 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:21.251 GMT,"postgres","postgres",11667,"[local]",658bc1e9.2d93,1,"",2023-12-27 06:19:21 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:21.353 GMT,"postgres","postgres",11668,"[local]",658bc1e9.2d94,1,"",2023-12-27 06:19:21 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:21.677 GMT,"postgres","postgres",11669,"127.0.0.1:55500",658bc1e9.2d95,1,"",2023-12-27 06:19:21 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:21.678 GMT,"postgres","postgres",11670,"127.0.0.1:55508",658bc1e9.2d96,1,"",2023-12-27 06:19:21 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:22.677 GMT,"postgres","postgres",11671,"127.0.0.1:55520",658bc1ea.2d97,1,"",2023-12-27 06:19:22 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
2023-12-27 06:19:22.679 GMT,"postgres","postgres",11672,"127.0.0.1:55522",658bc1ea.2d98,1,"",2023-12-27 06:19:22 GMT,,0,FATAL,57P03,"the database system is starting up",,,,,,,,,"","client backend",,0
ldming commented 10 months ago

https://github.com/zalando/patroni/issues/2717 https://github.com/zalando/patroni/issues/1251

There are some scenarios that may lead to this situation, such as inconsistencies in the timeline between primary and standby servers. In such cases, it is best to delete the standby and perform a rebuild of the standby to fix the issue.