[BUG] patroni postgresql config --> stop --> start cluster is Failed

Describe the bug patroni postgresql config --> stop --> start cluster Failed.

kbcli version
Kubernetes: v1.25.6-eks-48e63af
KubeBlocks: 0.5.0-alpha.6
kbcli: 0.5.0-alpha.6

To Reproduce Steps to reproduce the behavior:

create pg cluster

kbcli cluster create pg-cluster --termination-policy=WipeOut --cluster-definition=postgresql --set replicas=2,cpu=300m,memory=500Mi,storage=1Gi

config

echo yes | kbcli cluster configure pg-cluster\
--component postgresql \
--config-spec postgresql-configuration \
--config-file postgresql.conf \
--set max_connections=300,shared_buffers=512MB

stop --> start
```
kbcli cluster stop pg-cluster
```

kbcli cluster start pg-cluster

4. See error

kbcli cluster describe pg-cluster Name: pg-cluster Created Time: Apr 11,2023 15:50 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default postgresql postgresql-15.2.0 Failed WipeOut

Endpoints: COMPONENT MODE INTERNAL EXTERNAL
postgresql ReadWrite pg-cluster-postgresql.default.svc.cluster.local:5432
pg-cluster-postgresql.default.svc.cluster.local:9187

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
postgresql pg-cluster-postgresql-0 primary Running cn-northwest-1a ip-172-31-13-48.cn-northwest-1.compute.internal/172.31.13.48 Apr 11,2023 16:30 UTC+0800
postgresql pg-cluster-postgresql-1-0 secondary Running cn-northwest-1c ip-172-31-44-8.cn-northwest-1.compute.internal/172.31.44.8 Apr 11,2023 16:30 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
postgresql false 300m / 300m 500Mi / 500Mi data:1Gi ebs-sc

Images: COMPONENT TYPE IMAGE
postgresql postgresql registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:15.2.0

Events(last 5 warnings, see more:kbcli cluster list-events -n default pg-cluster): TIME TYPE REASON OBJECT MESSAGE
Apr 11,2023 16:29 UTC+0800 Warning ApplyResourcesFailed Cluster/pg-cluster Operation cannot be fulfilled on statefulsets.apps "pg-cluster-postgresql": StorageError: invalid object, Code: 4, Key: /registry/statefulsets/default/pg-cluster-postgresql, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 84df5a84-ee75-446c-be7e-85dbc440991e, UID in object meta:
Apr 11,2023 16:30 UTC+0800 Warning Unhealthy Cluster/pg-cluster Pod pg-cluster-postgresql-1-0: Readiness probe failed: 127.0.0.1:5432 - no response

Apr 11,2023 16:31 UTC+0800 Warning Unhealthy Cluster/pg-cluster Pod pg-cluster-postgresql-0: Readiness probe failed: 127.0.0.1:5432 - no response

Apr 11,2023 16:35 UTC+0800 Warning Unhealthy Instance/pg-cluster-postgresql-1-0 Readiness probe failed: 127.0.0.1:5432 - no response

Apr 11,2023 16:35 UTC+0800 Warning Unhealthy Instance/pg-cluster-postgresql-0 Readiness probe failed: 127.0.0.1:5432 - no response

kubectl get pod,ops,cm -l app.kubernetes.io/instance=pg-cluster
NAME READY STATUS RESTARTS AGE pod/pg-cluster-postgresql-0 3/4 Running 0 5m24s pod/pg-cluster-postgresql-1-0 3/4 Running 0 5m24s

NAME TYPE CLUSTER STATUS PROGRESS AGE opsrequest.apps.kubeblocks.io/pg-cluster-reconfiguring-pmr6n Reconfiguring pg-cluster Succeed 2/2 40m opsrequest.apps.kubeblocks.io/pg-cluster-start-bbbdg Start pg-cluster Failed 2/2 5m25s opsrequest.apps.kubeblocks.io/pg-cluster-stop-xjx4f Stop pg-cluster Succeed 2/2 6m10s

NAME DATA AGE configmap/pg-cluster-postgresql-env 5 5m24s configmap/pg-cluster-postgresql-patroni-config 0 44m configmap/pg-cluster-postgresql-patroni-leader 0 44m configmap/pg-cluster-postgresql-postgresql-configuration 3 5m24s configmap/pg-cluster-postgresql-postgresql-custom-metrics 1 5m24s configmap/pg-cluster-postgresql-postgresql-scripts 3 5m24s


logs pod

kubectl logs pg-cluster-postgresql-0 Defaulted container "postgresql" out of: postgresql, metrics, kb-checkrole, config-manager, pg-init-container (init)

KB_PRIMARY_POD_NAME_PREFIX=pg-cluster-postgresql-0
'[' pg-cluster-postgresql-0 '!=' pg-cluster-postgresql-0 ']'
python3 /kb-scripts/generate_patroni_yaml.py tmp_patroni.yaml ++ cat tmp_patroni.yaml
export 'SPILO_CONFIGURATION=bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
SPILO_CONFIGURATION='bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
exec /launch.sh init 2023-04-11 08:30:16,625 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?) 2023-04-11 08:30:16,710 - bootstrapping - INFO - Looks like you are running aws 2023-04-11 08:30:16,814 - bootstrapping - INFO - Configuring certificate 2023-04-11 08:30:16,814 - bootstrapping - INFO - Generating ssl self-signed certificate 2023-04-11 08:30:19,120 - bootstrapping - INFO - Configuring pgqd 2023-04-11 08:30:19,120 - bootstrapping - INFO - Configuring log 2023-04-11 08:30:19,120 - bootstrapping - INFO - Configuring pam-oauth2 2023-04-11 08:30:19,120 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping 2023-04-11 08:30:19,121 - bootstrapping - INFO - Configuring bootstrap 2023-04-11 08:30:19,121 - bootstrapping - INFO - Configuring standby-cluster 2023-04-11 08:30:19,121 - bootstrapping - INFO - Configuring crontab 2023-04-11 08:30:19,121 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability 2023-04-11 08:30:19,121 - bootstrapping - INFO - Configuring pgbouncer 2023-04-11 08:30:19,121 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping 2023-04-11 08:30:19,121 - bootstrapping - INFO - Configuring wal-e 2023-04-11 08:30:19,121 - bootstrapping - INFO - Configuring patroni 2023-04-11 08:30:19,128 - bootstrapping - INFO - Writing to file /run/postgres.yml 2023-04-11 08:30:21,020 INFO: Selected new K8s API server endpoint https://172.31.14.56:443 2023-04-11 08:30:21,225 ERROR: Exception when setting dynamic_configuration Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 199, in set_dynamic_configuration self.__effective_configuration = self._build_effective_configuration(configuration, File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 407, in _build_effective_configuration config = self._safe_copy_dynamic_configuration(dynamic_configuration) File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 237, in _safe_copy_dynamic_configuration config['postgresql'][name].update(self._process_postgresql_parameters(value)) File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 223, in _process_postgresql_parameters return {name: value for name, value in (parameters or {}).items() File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 225, in not is_local and ConfigHandler.CMDLINE_OPTIONS[name]1} File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/config.py", line 274, in 'max_connections': (100, lambda v: int(v) >= 25, 90100), ValueError: invalid literal for int() with base 10: "'55'" 2023-04-11 08:30:21,227 INFO: No PostgreSQL configuration items changed, nothing to reload. 2023-04-11 08:30:21,229 INFO: Lock owner: None; I am pg-cluster-postgresql-0 2023-04-11 08:30:21,313 INFO: waiting for leader to bootstrap 2023-04-11 08:30:31,735 INFO: Lock owner: None; I am pg-cluster-postgresql-0 2023-04-11 08:30:31,736 INFO: waiting for leader to bootstrap ... 2023-04-11 08:36:01,735 INFO: waiting for leader to bootstrap 2023-04-11 08:36:11,735 INFO: Lock owner: None; I am pg-cluster-postgresql-0 2023-04-11 08:36:11,736 INFO: waiting for leader to bootstrap 2023-04-11 08:36:21,735 INFO: Lock owner: None; I am pg-cluster-postgresql-0 2023-04-11 08:36:21,735 INFO: waiting for leader to bootstrap ➜ ~

➜ ~ kubectl logs pg-cluster-postgresql-1-0 Defaulted container "postgresql" out of: postgresql, metrics, kb-checkrole, config-manager, pg-init-container (init)

KB_PRIMARY_POD_NAME_PREFIX=pg-cluster-postgresql-0
'[' pg-cluster-postgresql-0 '!=' pg-cluster-postgresql-1-0 ']'
sleep 3
python3 /kb-scripts/generate_patroni_yaml.py tmp_patroni.yaml ++ cat tmp_patroni.yaml
export 'SPILO_CONFIGURATION=bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
SPILO_CONFIGURATION='bootstrap: initdb:
- auth-host: md5
- auth-local: trust postgresql: config_dir: /home/postgres/pgdata/conf custom_conf: /home/postgres/conf/postgresql.conf pg_hba:
- host all all 0.0.0.0/0 trust
- host all all ::/0 trust
- local all all trust
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
- host replication all 0.0.0.0/0 md5
- host replication all ::/0 md5'
exec /launch.sh init 2023-04-11 08:30:12,450 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?) 2023-04-11 08:30:12,523 - bootstrapping - INFO - Looks like you are running aws 2023-04-11 08:30:12,626 - bootstrapping - INFO - Configuring wal-e 2023-04-11 08:30:12,626 - bootstrapping - INFO - Configuring pgbouncer 2023-04-11 08:30:12,626 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping 2023-04-11 08:30:12,626 - bootstrapping - INFO - Configuring pgqd 2023-04-11 08:30:12,627 - bootstrapping - INFO - Configuring pam-oauth2 2023-04-11 08:30:12,627 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping 2023-04-11 08:30:12,627 - bootstrapping - INFO - Configuring bootstrap 2023-04-11 08:30:12,627 - bootstrapping - INFO - Configuring patroni 2023-04-11 08:30:12,634 - bootstrapping - INFO - Writing to file /run/postgres.yml 2023-04-11 08:30:12,635 - bootstrapping - INFO - Configuring log 2023-04-11 08:30:12,635 - bootstrapping - INFO - Configuring crontab 2023-04-11 08:30:12,635 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability 2023-04-11 08:30:12,635 - bootstrapping - INFO - Configuring standby-cluster 2023-04-11 08:30:12,635 - bootstrapping - INFO - Configuring certificate 2023-04-11 08:30:12,635 - bootstrapping - INFO - Generating ssl self-signed certificate 2023-04-11 08:30:16,030 INFO: Selected new K8s API server endpoint https://172.31.44.109:443 2023-04-11 08:30:16,118 ERROR: Exception when setting dynamic_configuration Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 199, in set_dynamic_configuration self.__effective_configuration = self._build_effective_configuration(configuration, File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 407, in _build_effective_configuration config = self._safe_copy_dynamic_configuration(dynamic_configuration) File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 237, in _safe_copy_dynamic_configuration config['postgresql'][name].update(self._process_postgresql_parameters(value)) File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 223, in _process_postgresql_parameters return {name: value for name, value in (parameters or {}).items() File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 225, in not is_local and ConfigHandler.CMDLINE_OPTIONS[name]1} File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/config.py", line 274, in 'max_connections': (100, lambda v: int(v) >= 25, 90100), ValueError: invalid literal for int() with base 10: "'55'" 2023-04-11 08:30:16,119 INFO: No PostgreSQL configuration items changed, nothing to reload. 2023-04-11 08:30:16,121 INFO: Lock owner: None; I am pg-cluster-postgresql-1-0 2023-04-11 08:30:16,218 INFO: waiting for leader to bootstrap 2023-04-11 08:30:26,629 INFO: Lock owner: None; I am pg-cluster-postgresql-1-0 2023-04-11 08:30:26,630 INFO: waiting for leader to bootstrap ... 2023-04-11 08:36:36,627 INFO: waiting for leader to bootstrap 2023-04-11 08:36:46,627 INFO: Lock owner: None; I am pg-cluster-postgresql-1-0 2023-04-11 08:36:46,627 INFO: waiting for leader to bootstrap 2023-04-11 08:36:56,628 INFO: Lock owner: None; I am pg-cluster-postgresql-1-0 2023-04-11 08:36:56,628 INFO: waiting for leader to bootstrap ➜ ~



**Expected behavior**
patroni postgresql config --> stop --> start cluster is OK.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.

apecloud / kubeblocks

[BUG] patroni postgresql config --> stop --> start cluster is Failed #2512