Y-Rookie
commented
1 year ago There are currently three ways to create a pg cluster：
- Create a pg cluster through helm and kbcli, and the required rbac resources (serviceAccount, role, rolebinding) will be automatically created. Each cluster corresponds to a set of rbac resources.
- If it is created by kubectl apply, you need to manually create rbac resources (serviceAccount, role, rolebinding), and you need to specify a specific serviceAccountName on the cluster yaml
Describe the bug pg cluster create failed：Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace.
Warning Unhealthy 14m kubelet Readiness probe failed: {"event":"Failed","message":"error executing select pg_is_in_recovery();: failed to connect to
host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: connection refused)","originalRole":""}To Reproduce Steps to reproduce the behavior:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/data-postgresql-cluster-postgresql-0 Bound d-bp1cs832jejcrmglo7io 300Gi RWO alicloud-disk-essd 13m persistentvolumeclaim/data-postgresql-cluster-postgresql-1 Bound d-bp1bldgclx2els837vmv 300Gi RWO alicloud-disk-essd 13m
NAME DATA AGE configmap/kube-root-ca.crt 1 26m configmap/patroni-reload-script-postgresql-cluster 3 15m configmap/postgresql-cluster-postgresql-env 5 15m configmap/postgresql-cluster-postgresql-postgresql-configuration 4 15m configmap/postgresql-cluster-postgresql-postgresql-custom-metrics 1 15m configmap/postgresql-cluster-postgresql-postgresql-scripts 4 15m ➜ ~
➜ ~ kubectl describe pod postgresql-cluster-postgresql-0 Name: postgresql-cluster-postgresql-0 Namespace: default Priority: 0 Service Account: default Node: cn-hangzhou.192.168.0.112/192.168.0.112 Start Time: Thu, 04 May 2023 10:30:44 +0800 Labels: app.kubernetes.io/component=postgresql app.kubernetes.io/instance=postgresql-cluster app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=postgresql app.kubernetes.io/version=postgresql-12.14.0 apps.kubeblocks.io/component-name=postgresql apps.kubeblocks.io/workload-type=Replication apps.kubeblocks.postgres.patroni/scope=postgresql-cluster-postgresql-patroni controller-revision-hash=postgresql-cluster-postgresql-54469c8864 kubeblocks.io/role=primary statefulset.kubernetes.io/pod-name=postgresql-cluster-postgresql-0 Annotations: k8s.aliyun.com/pod-ips: 192.168.0.132 kubernetes.io/psp: ack.privileged Status: Running IP: 192.168.0.132 IPs: IP: 192.168.0.132 Controlled By: StatefulSet/postgresql-cluster-postgresql Init Containers: pg-init-container: Container ID: containerd://1d5e183eb490a25189321e4aff4b820bb3e781c6ccf8cf4f436730711f6ce9bc Image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.14.0 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo@sha256:5e0b1211207b158ed43c109e5ff1be830e1bf5e7aff1f0dd3c90966804c5a143 Port:
Host Port:
Command:
/kb-scripts/init_container.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 04 May 2023 10:31:32 +0800
Finished: Thu, 04 May 2023 10:31:32 +0800
Ready: True
Restart Count: 0
Environment Variables from:
postgresql-cluster-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgresql-cluster-postgresql-0 (v1:metadata.name)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgresql-cluster
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgresql-cluster-postgresql
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
Mounts:
/home/postgres/conf from postgresql-config (rw)
/home/postgres/pgdata from data (rw)
/kb-scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cfzn9 (ro)
Containers:
postgresql:
Container ID: containerd://9947bd034761a1f5409bcd3d7448cb77bc05704bb9ed375547a596618a37ec1f
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.14.0
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo@sha256:5e0b1211207b158ed43c109e5ff1be830e1bf5e7aff1f0dd3c90966804c5a143
Ports: 5432/TCP, 8008/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/kb-scripts/setup.sh
State: Running
Started: Thu, 04 May 2023 10:31:36 +0800
Ready: False
Restart Count: 0
Readiness: exec [/bin/sh -c -ee exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
[ -f /postgresql/tmp/.initialized ] || [ -f /postgresql/.initialized ]
] delay=25s timeout=5s period=30s #success=1 #failure=3
Environment Variables from:
postgresql-cluster-postgresql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: postgresql-cluster-postgresql-0 (v1:metadata.name)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: postgresql-cluster
KB_COMP_NAME: postgresql
KB_CLUSTER_COMP_NAME: postgresql-cluster-postgresql
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
DCS_ENABLE_KUBERNETES_API: true
KUBERNETES_USE_CONFIGMAPS: true
SCOPE: $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni
KUBERNETES_SCOPE_LABEL: apps.kubeblocks.postgres.patroni/scope
KUBERNETES_ROLE_LABEL: apps.kubeblocks.postgres.patroni/role
KUBERNETES_LABELS: {"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)"}
RESTORE_DATA_DIR: /home/postgres/pgdata/kb_restore
KB_PG_CONFIG_PATH: /home/postgres/conf/postgresql.conf
SPILO_CONFIGURATION: bootstrap:
initdb:
auth-local: trust
ALLOW_NOSSL: true PGROOT: /home/postgres/pgdata/pgroot POD_IP: (v1:status.podIP) POD_NAMESPACE: default (v1:metadata.namespace) PGUSER_SUPERUSER: <set to the key 'username' in secret 'postgresql-cluster-conn-credential'> Optional: false PGPASSWORD_SUPERUSER: <set to the key 'password' in secret 'postgresql-cluster-conn-credential'> Optional: false PGUSER_ADMIN: superadmin PGPASSWORD_ADMIN: <set to the key 'password' in secret 'postgresql-cluster-conn-credential'> Optional: false PGUSER_STANDBY: standby PGPASSWORD_STANDBY: <set to the key 'password' in secret 'postgresql-cluster-conn-credential'> Optional: false PGUSER: <set to the key 'username' in secret 'postgresql-cluster-conn-credential'> Optional: false PGPASSWORD: <set to the key 'password' in secret 'postgresql-cluster-conn-credential'> Optional: false Mounts: /dev/shm from dshm (rw) /home/postgres/conf from postgresql-config (rw) /home/postgres/pgdata from data (rw) /kb-scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cfzn9 (ro) metrics: Container ID: containerd://eb5567d9f1d4697a51eefa759063e6ef3adaf90e057c25714abfc1a0e5ec182e Image: registry.cn-hangzhou.aliyuncs.com/apecloud/postgres-exporter:0.11.1-debian-11-r66 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/postgres-exporter@sha256:17c0bf751b9db5476a83a252caab6f26109a786b93fd83d4a73a2ea9c33e1e69 Port: 9187/TCP Host Port: 0/TCP Command: /opt/bitnami/postgres-exporter/bin/postgres_exporter --auto-discover-databases --extend.query-path=/opt/conf/custom-metrics.yaml --exclude-databases=template0,template1 --log.level=info State: Running Started: Thu, 04 May 2023 10:31:45 +0800 Ready: True Restart Count: 0 Liveness: http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6 Readiness: http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6 Environment Variables from: postgresql-cluster-postgresql-env ConfigMap Optional: false Environment: KB_POD_NAME: postgresql-cluster-postgresql-0 (v1:metadata.name) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: postgresql-cluster KB_COMP_NAME: postgresql KB_CLUSTER_COMP_NAME: postgresql-cluster-postgresql KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc DATA_SOURCE_URI: 127.0.0.1:5432/postgres?sslmode=disable DATA_SOURCE_PASS: <set to the key 'password' in secret 'postgresql-cluster-conn-credential'> Optional: false DATA_SOURCE_USER: <set to the key 'username' in secret 'postgresql-cluster-conn-credential'> Optional: false Mounts: /opt/conf from postgresql-custom-metrics (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cfzn9 (ro) kb-checkrole: Container ID: containerd://84eb3a0a69d2969d62f85c6edf24404154636f79b576a1ce332f04ff5b7ac583 Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.0-beta.15 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:c983538b5cf64e1ca5a55382067bee3bf2f275f6afe9c5c3eefd3caa141820a4 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: probe --app-id batch-sdk --dapr-http-port 3501 --dapr-grpc-port 50001 --app-protocol http --log-level info --config /config/probe/config.yaml --components-path /config/probe/components State: Running Started: Thu, 04 May 2023 10:31:46 +0800 Ready: True Restart Count: 0 Readiness: exec [curl -X POST --max-time 1 --fail-with-body --silent -H Content-ComponentDefRef: application/json http://localhost:3501/v1.0/bindings/postgresql -d {"operation": "checkRole", "metadata":{"sql":""}}] delay=0s timeout=1s period=1s #success=1 #failure=2 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: postgresql-cluster-postgresql-env ConfigMap Optional: false Environment: KB_POD_NAME: postgresql-cluster-postgresql-0 (v1:metadata.name) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: postgresql-cluster KB_COMP_NAME: postgresql KB_CLUSTER_COMP_NAME: postgresql-cluster-postgresql KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc KB_SERVICE_USER: <set to the key 'username' in secret 'postgresql-cluster-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'postgresql-cluster-conn-credential'> Optional: false KB_SERVICE_PORT: 5432 KB_SERVICE_ROLES: {} KB_SERVICE_CHARACTER_TYPE: postgresql Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cfzn9 (ro) config-manager: Container ID: containerd://558f6f8e9f7c32e400b52d5ab758a7da0171e4dac8c4bab65208be0ce6ad297c Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.0-beta.15 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:c983538b5cf64e1ca5a55382067bee3bf2f275f6afe9c5c3eefd3caa141820a4 Port:
Host Port:
Command:
/bin/reloader
Args:
--operator-update-enable
--log-level
info
--tcp
9901
--notify-type
tpl
--tpl-config
/opt/config/reload/reload.yaml
State: Running
Started: Thu, 04 May 2023 10:31:46 +0800
Ready: True
Restart Count: 0
Environment:
CONFIG_MANAGER_POD_IP: (v1:status.podIP)
DB_TYPE: postgresql
Mounts:
/home/postgres/conf from postgresql-config (rw)
/opt/config/reload from reload-manager-reload (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cfzn9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-postgresql-cluster-postgresql-0
ReadOnly: false
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit:
postgresql-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgresql-cluster-postgresql-postgresql-configuration
Optional: false
postgresql-custom-metrics:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgresql-cluster-postgresql-postgresql-custom-metrics
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: postgresql-cluster-postgresql-postgresql-scripts
Optional: false
reload-manager-reload:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: patroni-reload-script-postgresql-cluster
Optional: false
kube-api-access-cfzn9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Warning FailedScheduling 2m17s default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Normal Scheduled 2m15s default-scheduler Successfully assigned default/postgresql-cluster-postgresql-0 to cn-hangzhou.192.168.0.112 Normal SuccessfulAttachVolume 2m15s attachdetach-controller AttachVolume.Attach succeeded for volume "d-bp1cs832jejcrmglo7io" Normal AllocIPSucceed 118s terway-daemon Alloc IP 192.168.0.132/24 Normal Pulling 118s kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.14.0" Normal Pulled 87s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.14.0" in 30.566761317s (30.566769571s including waiting) Normal Created 87s kubelet Created container pg-init-container Normal Started 87s kubelet Started container pg-init-container Normal Pulled 83s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.14.0" already present on machine Normal Created 83s kubelet Created container postgresql Normal Started 83s kubelet Started container postgresql Normal Pulling 83s kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/postgres-exporter:0.11.1-debian-11-r66" Normal Pulled 74s kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/postgres-exporter:0.11.1-debian-11-r66" in 8.927890494s (8.927897719s including waiting) Normal Created 74s kubelet Created container metrics Normal Started 74s kubelet Started container metrics Normal Pulled 74s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.0-beta.15" already present on machine Normal Created 74s kubelet Created container kb-checkrole Normal Started 73s kubelet Started container kb-checkrole Normal Pulled 73s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.0-beta.15" already present on machine Normal Created 73s kubelet Created container config-manager Normal Started 73s kubelet Started container config-manager Warning Unhealthy 66s kubelet Readiness probe failed: {"event":"Failed","message":"error executing select pg_is_in_recovery();: failed to connect to
host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: connection refused)","originalRole":""} Warning Unhealthy 28s (x2 over 58s) kubelet Readiness probe failed: 127.0.0.1:5432 - no response ➜ ~pod logs
Expected behavior pg cluster create succeed.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context Add any other context about the problem here.