[BUG] patroni postgresql do stop and start Failed

Describe the bug patroni postgresql do stop and start Failed.

kbcli version                                                   
Kubernetes: v1.25.6-eks-48e63af
KubeBlocks: 0.5.0-alpha.7
kbcli: 0.5.0-alpha.7

To Reproduce Steps to reproduce the behavior:

install kubeblocks

create pg cluster

kbcli cluster create test-cluster --termination-policy=WipeOut --cluster-definition=postgresql --set cpu=100m,memory=500Mi,replicas=2,storage=1Gi --namespace default

stop
```
kbcli cluster stop test-cluster
```
start
```
kbcli cluster start test-cluster
```

See error


kubectl get pod,ops,sts -l app.kubernetes.io/instance=test-cluster
NAME                              READY   STATUS    RESTARTS   AGE
pod/test-cluster-postgresql-0     3/4     Running   0          14m
pod/test-cluster-postgresql-1-0   3/4     Running   0          14m

NAME TYPE CLUSTER STATUS PROGRESS AGE opsrequest.apps.kubeblocks.io/test-cluster-start-rrc6p Start test-cluster Failed 2/2 14m opsrequest.apps.kubeblocks.io/test-cluster-stop-zlzhv Stop test-cluster Succeed 2/2 14m

NAME READY AGE statefulset.apps/test-cluster-postgresql 0/1 14m statefulset.apps/test-cluster-postgresql-1 0/1 14m

6.  describe cluster

kbcli cluster describe test-cluster Name: test-cluster Created Time: Apr 13,2023 18:33 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default postgresql postgresql-15.2.0 Failed WipeOut

Endpoints: COMPONENT MODE INTERNAL EXTERNAL
postgresql ReadWrite test-cluster-postgresql.default.svc.cluster.local:5432
test-cluster-postgresql.default.svc.cluster.local:9187

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
postgresql test-cluster-postgresql-0 primary Running cn-northwest-1a ip-172-31-13-48.cn-northwest-1.compute.internal/172.31.13.48 Apr 13,2023 18:36 UTC+0800
postgresql test-cluster-postgresql-1-0 secondary Running cn-northwest-1c ip-172-31-44-8.cn-northwest-1.compute.internal/172.31.44.8 Apr 13,2023 18:37 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
postgresql false 100m / 100m 500Mi / 500Mi data:1Gi ebs-sc

Images: COMPONENT TYPE IMAGE
postgresql postgresql registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:15.2.0

Events(last 5 warnings, see more:kbcli cluster list-events -n default test-cluster): TIME TYPE REASON OBJECT MESSAGE
Apr 13,2023 18:36 UTC+0800 Warning ApplyResourcesFailed Cluster/test-cluster Operation cannot be fulfilled on statefulsets.apps "test-cluster-postgresql-1": StorageError: invalid object, Code: 4, Key: /registry/statefulsets/default/test-cluster-postgresql-1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 138d0e95-794e-44a6-b9d6-8c8fd01e9b1c, UID in object meta:
Apr 13,2023 18:37 UTC+0800 Warning Unhealthy Cluster/test-cluster Pod test-cluster-postgresql-0: Readiness probe failed: 127.0.0.1:5432 - no response

Apr 13,2023 18:38 UTC+0800 Warning Unhealthy Cluster/test-cluster Pod test-cluster-postgresql-1-0: Readiness probe failed: 127.0.0.1:5432 - no response

Apr 13,2023 18:47 UTC+0800 Warning Unhealthy Instance/test-cluster-postgresql-0 Readiness probe failed: 127.0.0.1:5432 - no response

Apr 13,2023 18:47 UTC+0800 Warning Unhealthy Instance/test-cluster-postgresql-1-0 Readiness probe failed: 127.0.0.1:5432 - no response

7.  logs pod

➜ ~ kubectl logs test-cluster-postgresql-0 Defaulted container "postgresql" out of: postgresql, metrics, kb-checkrole, config-manager, pg-init-container (init)

KB_PRIMARY_POD_NAME_PREFIX=test-cluster-postgresql-0
'[' test-cluster-postgresql-0 '!=' test-cluster-postgresql-0 ']'
python3 /kb-scripts/generate_patroni_yaml.py tmp_patroni.yaml ++ cat tmp_patroni.yaml
export 'SPILO_CONFIGURATION=bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- local replication all trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
SPILO_CONFIGURATION='bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- local replication all trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
exec /launch.sh init 2023-04-13 10:37:10,402 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?) 2023-04-13 10:37:10,509 - bootstrapping - INFO - Looks like you are running aws 2023-04-13 10:37:10,902 - bootstrapping - INFO - Configuring pgqd 2023-04-13 10:37:10,903 - bootstrapping - INFO - Configuring standby-cluster 2023-04-13 10:37:10,903 - bootstrapping - INFO - Configuring pam-oauth2 2023-04-13 10:37:10,903 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping 2023-04-13 10:37:10,903 - bootstrapping - INFO - Configuring crontab 2023-04-13 10:37:10,903 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability 2023-04-13 10:37:10,903 - bootstrapping - INFO - Configuring certificate 2023-04-13 10:37:10,903 - bootstrapping - INFO - Generating ssl self-signed certificate 2023-04-13 10:37:14,602 - bootstrapping - INFO - Configuring bootstrap 2023-04-13 10:37:14,603 - bootstrapping - INFO - Configuring wal-e 2023-04-13 10:37:14,603 - bootstrapping - INFO - Configuring patroni 2023-04-13 10:37:14,700 - bootstrapping - INFO - Writing to file /run/postgres.yml 2023-04-13 10:37:14,701 - bootstrapping - INFO - Configuring log 2023-04-13 10:37:14,701 - bootstrapping - INFO - Configuring pgbouncer 2023-04-13 10:37:14,701 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping 2023-04-13 10:37:17,208 INFO: Selected new K8s API server endpoint https://172.31.44.109:443 2023-04-13 10:37:17,403 INFO: No PostgreSQL configuration items changed, nothing to reload. 2023-04-13 10:37:17,501 INFO: Lock owner: None; I am test-cluster-postgresql-0 2023-04-13 10:37:17,605 INFO: waiting for leader to bootstrap 2023-04-13 10:37:28,139 INFO: Lock owner: None; I am test-cluster-postgresql-0 2023-04-13 10:37:28,139 INFO: waiting for leader to bootstrap 2023-04-13 10:37:38,008 INFO: Lock owner: None; I am test-cluster-postgresql-0 2023-04-13 10:37:38,008 INFO: waiting for leader to bootstrap ... 2023-04-13 10:52:08,010 INFO: waiting for leader to bootstrap 2023-04-13 10:52:18,006 INFO: Lock owner: None; I am test-cluster-postgresql-0 2023-04-13 10:52:18,007 INFO: waiting for leader to bootstrap 2023-04-13 10:52:28,040 INFO: Lock owner: None; I am test-cluster-postgresql-0 2023-04-13 10:52:28,041 INFO: waiting for leader to bootstrap 2023-04-13 10:52:38,007 INFO: Lock owner: None; I am test-cluster-postgresql-0 2023-04-13 10:52:38,007 INFO: waiting for leader to bootstrap ➜ ~

➜ ~ kubectl logs test-cluster-postgresql-1-0 Defaulted container "postgresql" out of: postgresql, metrics, kb-checkrole, config-manager, pg-init-container (init)

KB_PRIMARY_POD_NAME_PREFIX=test-cluster-postgresql-0
'[' test-cluster-postgresql-0 '!=' test-cluster-postgresql-1-0 ']'
sleep 3
python3 /kb-scripts/generate_patroni_yaml.py tmp_patroni.yaml ++ cat tmp_patroni.yaml
export 'SPILO_CONFIGURATION=bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- local replication all trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
SPILO_CONFIGURATION='bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- local replication all trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
exec /launch.sh init 2023-04-13 10:37:16,819 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?) 2023-04-13 10:37:17,015 - bootstrapping - INFO - Looks like you are running aws 2023-04-13 10:37:17,319 - bootstrapping - INFO - Configuring patroni 2023-04-13 10:37:17,417 - bootstrapping - INFO - Writing to file /run/postgres.yml 2023-04-13 10:37:17,418 - bootstrapping - INFO - Configuring certificate 2023-04-13 10:37:17,418 - bootstrapping - INFO - Generating ssl self-signed certificate 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring wal-e 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring log 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring pam-oauth2 2023-04-13 10:37:24,812 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring crontab 2023-04-13 10:37:24,812 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring bootstrap 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring standby-cluster 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring pgbouncer 2023-04-13 10:37:24,812 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping 2023-04-13 10:37:24,812 - bootstrapping - INFO - Configuring pgqd 2023-04-13 10:37:27,317 INFO: Selected new K8s API server endpoint https://172.31.44.109:443 2023-04-13 10:37:27,515 INFO: No PostgreSQL configuration items changed, nothing to reload. 2023-04-13 10:37:27,612 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 2023-04-13 10:37:27,714 INFO: waiting for leader to bootstrap 2023-04-13 10:37:38,117 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 2023-04-13 10:37:38,117 INFO: waiting for leader to bootstrap 2023-04-13 10:37:48,116 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 2023-04-13 10:37:48,117 INFO: waiting for leader to bootstrap 2023-04-13 10:37:58,119 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 2023-04-13 10:37:58,120 INFO: waiting for leader to bootstrap 2023-04-13 10:38:08,116 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 ... 2023-04-13 10:53:08,119 INFO: waiting for leader to bootstrap 2023-04-13 10:53:18,117 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 2023-04-13 10:53:18,117 INFO: waiting for leader to bootstrap 2023-04-13 10:53:28,118 INFO: Lock owner: None; I am test-cluster-postgresql-1-0 2023-04-13 10:53:28,118 INFO: waiting for leader to bootstrap ➜ ~


cluster yaml

➜ ~ kubectl get cluster test-cluster -oyaml apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: cluster.kubeblocks.io/component-class: '{}' creationTimestamp: "2023-04-13T10:33:50Z" finalizers:

cluster.kubeblocks.io/finalizer generation: 3 labels: clusterdefinition.kubeblocks.io/name: postgresql clusterversion.kubeblocks.io/name: postgresql-15.2.0 name: test-cluster namespace: default resourceVersion: "73530572" uid: 09d24196-d265-44e7-baf7-4d0ffd089d9f spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: postgresql clusterVersionRef: postgresql-15.2.0 componentSpecs:
componentDefRef: postgresql enabledLogs:
- running monitor: true name: postgresql replicas: 2 resources: limits: cpu: 100m memory: 500Mi requests: cpu: 100m memory: 500Mi switchPolicy: type: Noop volumeClaimTemplates:
- name: data spec: accessModes:
  - ReadWriteOnce resources: requests: storage: 1Gi terminationPolicy: WipeOut status: clusterDefGeneration: 12 components: postgresql: message: Pod/test-cluster-postgresql-0: |- Readiness probe failed: 127.0.0.1:5432 - no response ; Pod/test-cluster-postgresql-1-0: |- Readiness probe failed: 127.0.0.1:5432 - no response ; phase: Failed podsReady: false replicationSetStatus: primary: pod: test-cluster-postgresql-0 secondaries:
  - pod: test-cluster-postgresql-1-0 conditions:
lastTransitionTime: "2023-04-13T10:37:57Z" message: 'Start opsRequest: test-cluster-start-rrc6p has been processed' reason: Processed status: "True" type: LatestOpsRequestProcessed
lastTransitionTime: "2023-04-13T10:33:50Z" message: 'The operator has started the provisioning of Cluster: test-cluster' observedGeneration: 3 reason: PreCheckSucceed status: "True" type: ProvisioningStarted
lastTransitionTime: "2023-04-13T10:36:59Z" message: Successfully applied for resources observedGeneration: 3 reason: ApplyResourcesSucceed status: "True" type: ApplyResources
lastTransitionTime: "2023-04-13T10:36:31Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady
lastTransitionTime: "2023-04-13T10:36:31Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 3 phase: Failed ➜ ~


**Expected behavior**
patroni postgresql do stop and start succeed.

**Screenshots**

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.

apecloud / kubeblocks

[BUG] patroni postgresql do stop and start Failed #2572