search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.1k stars 170 forks source link

[BUG]PG cluster keeps ConditionsError status for 800s after pod is Running after upgrade kb from 0.5.1 to 0.6.2 #5004

Closed ahjing99 closed 4 months ago

ahjing99 commented 1 year ago
  1. install kb 0.5.1, create clusters
  2. upgrade kb to 0.6.2-beta.1
  3. PG cluster status is ConditionsError for about 800 seconds after pod is Running
➜  ~ k describe pod postgres-2g757-postgresql-0
Name:         postgres-2g757-postgresql-0
Namespace:    default
Priority:     0
Node:         gke-cluster-1-default-pool-56d1d4e0-fttg/10.128.0.49
Start Time:   Mon, 04 Sep 2023 17:16:39 +0800
Labels:       app.kubernetes.io/component=postgresql
              app.kubernetes.io/instance=postgres-2g757
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=postgresql
              app.kubernetes.io/version=postgresql-14.7.2
              apps.kubeblocks.io/component-name=postgresql
              apps.kubeblocks.io/workload-type=Replication
              apps.kubeblocks.postgres.patroni/role=master
              apps.kubeblocks.postgres.patroni/scope=postgres-2g757-postgresql-patroni5f2bfe19
              controller-revision-hash=postgres-2g757-postgresql-79fcff654c
              kubeblocks.io/role=primary
              statefulset.kubernetes.io/pod-name=postgres-2g757-postgresql-0
Annotations:  apps.kubeblocks.io/component-replicas: 1
              apps.kubeblocks.io/last-role-changed-event-timestamp: 2023-09-04T09:17:28Z
              rs.apps.kubeblocks.io/primary: postgres-2g757-postgresql-0
              status:
                {"conn_url":"postgres://10.4.0.196:5432/postgres","api_url":"http://10.4.0.196:8008/patroni","state":"running","role":"master","version":"...
Status:       Running
IP:           10.4.0.196
IPs:
  IP:           10.4.0.196
Controlled By:  StatefulSet/postgres-2g757-postgresql
Init Containers:
  pg-init-container:
    Container ID:  containerd://d458d65e9698d480f2f8893de6e457dc015076643f7e6e4771e8dcbbff8ec3c5
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/spilo@sha256:796331813dec29254a87faefeca0b3a2c9c4ac3c462f89d17a06c2d4ecde39c7
    Port:          <none>
    Host Port:     <none>
    Command:
      /kb-scripts/init_container.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 04 Sep 2023 17:16:50 +0800
      Finished:     Mon, 04 Sep 2023 17:16:50 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      postgres-2g757-postgresql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               postgres-2g757-postgresql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           postgres-2g757
      KB_COMP_NAME:              postgresql
      KB_CLUSTER_COMP_NAME:      postgres-2g757-postgresql
      KB_CLUSTER_UID_POSTFIX_8:  5f2bfe19
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    Mounts:
      /home/postgres/conf from postgresql-config (rw)
      /home/postgres/pgdata from data (rw)
      /kb-podinfo from pod-info (rw)
      /kb-scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
Containers:
  postgresql:
    Container ID:  containerd://a31c3bbf43cf6c4921030d17f0d7a98fed908ff9715bd25d577c00add9fa0767
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/spilo@sha256:796331813dec29254a87faefeca0b3a2c9c4ac3c462f89d17a06c2d4ecde39c7
    Ports:         5432/TCP, 8008/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /kb-scripts/setup.sh
    State:          Running
      Started:      Mon, 04 Sep 2023 17:16:53 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   102Mi
    Readiness:  exec [/bin/sh -c -ee exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
[ -f /postgresql/tmp/.initialized ] || [ -f /postgresql/.initialized ]
] delay=10s timeout=5s period=30s #success=1 #failure=3
    Environment Variables from:
      postgres-2g757-postgresql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                postgres-2g757-postgresql-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            postgres-2g757
      KB_COMP_NAME:               postgresql
      KB_CLUSTER_COMP_NAME:       postgres-2g757-postgresql
      KB_CLUSTER_UID_POSTFIX_8:   5f2bfe19
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      DCS_ENABLE_KUBERNETES_API:  true
      KUBERNETES_USE_CONFIGMAPS:  true
      SCOPE:                      $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni$(KB_CLUSTER_UID_POSTFIX_8)
      KUBERNETES_SCOPE_LABEL:     apps.kubeblocks.postgres.patroni/scope
      KUBERNETES_ROLE_LABEL:      apps.kubeblocks.postgres.patroni/role
      KUBERNETES_LABELS:          {"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)"}
      RESTORE_DATA_DIR:           /home/postgres/pgdata/kb_restore
      KB_PG_CONFIG_PATH:          /home/postgres/conf/postgresql.conf
      SPILO_CONFIGURATION:        bootstrap:
                                    initdb:
                                      - auth-host: md5
                                      - auth-local: trust

      ALLOW_NOSSL:                true
      PGROOT:                     /home/postgres/pgdata/pgroot
      POD_IP:                      (v1:status.podIP)
      POD_NAMESPACE:              default (v1:metadata.namespace)
      PGUSER_SUPERUSER:           <set to the key 'username' in secret 'postgres-2g757-conn-credential'>  Optional: false
      PGPASSWORD_SUPERUSER:       <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
      PGUSER_ADMIN:               superadmin
      PGPASSWORD_ADMIN:           <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
      PGUSER_STANDBY:             standby
      PGPASSWORD_STANDBY:         <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
      PGUSER:                     <set to the key 'username' in secret 'postgres-2g757-conn-credential'>  Optional: false
      PGPASSWORD:                 <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
    Mounts:
      /dev/shm from dshm (rw)
      /home/postgres/conf from postgresql-config (rw)
      /home/postgres/pgdata from data (rw)
      /kb-podinfo from pod-info (rw)
      /kb-scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
  pgbouncer:
    Container ID:  containerd://4cd7d4b99ee3472f0372811fed403f5cfc69a74e113293660f4e4f8659855ab9
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer@sha256:c788fda7436d178e6ea94cf7b8fa767005f0ef483edd90b328c560e1935b26b5
    Port:          6432/TCP
    Host Port:     0/TCP
    Command:
      /kb-scripts/pgbouncer_setup.sh
    State:          Running
      Started:      Mon, 04 Sep 2023 17:16:53 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Liveness:   tcp-socket :tcp-pgbouncer delay=15s timeout=5s period=30s #success=1 #failure=3
    Readiness:  tcp-socket :tcp-pgbouncer delay=15s timeout=5s period=30s #success=1 #failure=3
    Environment Variables from:
      postgres-2g757-postgresql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               postgres-2g757-postgresql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           postgres-2g757
      KB_COMP_NAME:              postgresql
      KB_CLUSTER_COMP_NAME:      postgres-2g757-postgresql
      KB_CLUSTER_UID_POSTFIX_8:  5f2bfe19
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      PGBOUNCER_AUTH_TYPE:       md5
      POSTGRESQL_USERNAME:       <set to the key 'username' in secret 'postgres-2g757-conn-credential'>  Optional: false
      POSTGRESQL_PASSWORD:       <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
      POSTGRESQL_PORT:           5432
      POSTGRESQL_HOST:            (v1:status.podIP)
      PGBOUNCER_PORT:            6432
      PGBOUNCER_BIND_ADDRESS:    0.0.0.0
    Mounts:
      /home/pgbouncer/conf from pgbouncer-config (rw)
      /kb-scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
  metrics:
    Container ID:  containerd://33e24a2285a81a91a5012a182a2960a6f76c24d38271cccdc37664e34b1906d0
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
    Port:          9187/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/agamotto/agamotto-config.yaml
    State:          Running
      Started:      Mon, 04 Sep 2023 17:16:55 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      postgres-2g757-postgresql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               postgres-2g757-postgresql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           postgres-2g757
      KB_COMP_NAME:              postgresql
      KB_CLUSTER_COMP_NAME:      postgres-2g757-postgresql
      KB_CLUSTER_UID_POSTFIX_8:  5f2bfe19
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      ENDPOINT:                  127.0.0.1:5432
      DATA_SOURCE_PASS:          <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
      DATA_SOURCE_USER:          <set to the key 'username' in secret 'postgres-2g757-conn-credential'>  Optional: false
    Mounts:
      /opt/agamotto from agamotto-configuration (rw)
      /opt/conf from postgresql-custom-metrics (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
  kb-checkrole:
    Container ID:  containerd://c17f0a5b8b2424f808b4bc90745f2eb495aacef25b7aca71912685399cac1106
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6760e31e4094714bbe126134d43d19c7d4bd82c1619695908c1c6c696474cb9f
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      probe
      --app-id
      batch-sdk
      --dapr-http-port
      3501
      --dapr-grpc-port
      50001
      --log-level
      info
      --config
      /config/probe/config.yaml
      --components-path
      /config/probe/components
    State:          Running
      Started:      Mon, 04 Sep 2023 17:16:56 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:3501/v1.0/bindings/postgresql%3Foperation=checkRole&workloadType=Replication delay=0s timeout=1s period=1s #success=1 #failure=2
    Startup:    tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      postgres-2g757-postgresql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                postgres-2g757-postgresql-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            postgres-2g757
      KB_COMP_NAME:               postgresql
      KB_CLUSTER_COMP_NAME:       postgres-2g757-postgresql
      KB_CLUSTER_UID_POSTFIX_8:   5f2bfe19
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'postgres-2g757-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'postgres-2g757-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            5432
      KB_SERVICE_ROLES:           {}
      KB_SERVICE_CHARACTER_TYPE:  postgresql
      KB_WORKLOAD_TYPE:           Replication
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
  config-manager:
    Container ID:  containerd://ee803697fd6060e4cac6ab3b54d203253fcf567c7ce72cc8d60fff22c0349124
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6760e31e4094714bbe126134d43d19c7d4bd82c1619695908c1c6c696474cb9f
    Port:          <none>
    Host Port:     <none>
    Command:
      env
    Args:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
      /bin/reloader
      --log-level
      info
      --operator-update-enable
      --tcp
      9901
      --config
      /opt/config-manager/config-manager.yaml
    State:          Running
      Started:      Mon, 04 Sep 2023 17:16:56 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      postgres-2g757-postgresql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               postgres-2g757-postgresql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           postgres-2g757
      KB_COMP_NAME:              postgresql
      KB_CLUSTER_COMP_NAME:      postgres-2g757-postgresql
      KB_CLUSTER_UID_POSTFIX_8:  5f2bfe19
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CONFIG_MANAGER_POD_IP:      (v1:status.podIP)
      DB_TYPE:                   postgresql
      TOOLS_PATH:                /opt/kb-tools/reload/postgresql-configuration:/opt/config-manager
    Mounts:
      /home/postgres/conf from postgresql-config (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/postgresql-configuration from cm-script-postgresql-configuration (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcqkm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-postgres-2g757-postgresql-0
    ReadOnly:   false
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels['kubeblocks.io/role'] -> pod-role
      metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  agamotto-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      postgres-2g757-postgresql-agamotto-configuration
    Optional:  false
  pgbouncer-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      postgres-2g757-postgresql-pgbouncer-configuration
    Optional:  false
  postgresql-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      postgres-2g757-postgresql-postgresql-configuration
    Optional:  false
  postgresql-custom-metrics:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      postgres-2g757-postgresql-postgresql-custom-metrics
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      postgres-2g757-postgresql-postgresql-scripts
    Optional:  false
  cm-script-postgresql-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-patroni-reload-script-postgres-2g757
    Optional:  false
  config-manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-postgres-2g757-postgresql-config-manager-config
    Optional:  false
  kube-api-access-fcqkm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    27m                default-scheduler  Successfully assigned default/postgres-2g757-postgresql-0 to gke-cluster-1-default-pool-56d1d4e0-fttg
  Warning  FailedMount  27m                kubelet            MountVolume.SetUp failed for volume "pgbouncer-config" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulled       27m                kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2" already present on machine
  Normal   Created      27m                kubelet            Created container pg-init-container
  Normal   Started      27m                kubelet            Started container pg-init-container
  Normal   Pulled       27m                kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:14.7.2" already present on machine
  Normal   Created      27m                kubelet            Created container postgresql
  Normal   Started      27m                kubelet            Started container postgresql
  Normal   Pulled       27m                kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0" already present on machine
  Normal   Created      27m                kubelet            Created container pgbouncer
  Normal   Started      27m                kubelet            Started container pgbouncer
  Normal   Pulled       27m                kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" already present on machine
  Normal   Created      27m                kubelet            Created container metrics
  Normal   Started      27m                kubelet            Started container metrics
  Normal   Pulled       27m                kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1" already present on machine
  Normal   Created      27m                kubelet            Created container kb-checkrole
  Normal   Started      27m                kubelet            Started container kb-checkrole
  Normal   Pulled       27m                kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1" already present on machine
  Normal   Created      27m                kubelet            Created container config-manager
  Normal   Started      27m                kubelet            Started container config-manager
  Normal   checkRole    27m                sqlchannel         {"event":"Failed","message":"error executing select pg_is_in_recovery();: failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)","operation":"checkRole","originalRole":""}
  Warning  Unhealthy    26m (x3 over 26m)  kubelet            Readiness probe failed: 127.0.0.1:5432 - no response
  Normal   checkRole    26m                sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"","role":"secondary"}
  Normal   checkRole    26m                sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}

➜  ~ k describe cluster postgres-2g757
Name:         postgres-2g757
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=postgresql
              clusterversion.kubeblocks.io/name=postgresql-14.7.2
Annotations:  kubeblocks.io/reconcile: 2023-09-04T09:16:39.574547783Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-09-04T08:12:18Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generate Name:  postgres-
  Generation:     1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        .:
        f:affinity:
          .:
          f:nodeLabels:
          f:podAntiAffinity:
          f:tenancy:
          f:topologyKeys:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"postgresql"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:switchPolicy:
              .:
              f:type:
            f:volumeClaimTemplates:
        f:terminationPolicy:
        f:tolerations:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2023-09-04T08:12:18Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:postgresql:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
            f:replicationSetStatus:
              .:
              f:primary:
                .:
                f:pod:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-09-04T09:07:29Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
    Manager:         manager
    Operation:       Update
    Time:            2023-09-04T09:16:39Z
  Resource Version:  442678
  UID:               fba0a375-be2a-42f3-aa94-ebde5f2bfe19
Spec:
  Affinity:
    Node Labels:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
    Topology Keys:
  Cluster Definition Ref:  postgresql
  Cluster Version Ref:     postgresql-14.7.2
  Component Specs:
    Component Def Ref:  postgresql
    Monitor:            true
    Name:               postgresql
    No Create PDB:      false
    Replicas:           1
    Resources:
      Limits:
        Cpu:     1000m
        Memory:  1024Mi
      Requests:
        Cpu:               100m
        Memory:            102Mi
    Service Account Name:  dbname
    Switch Policy:
      Type:  Noop
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         3Gi
        Storage Class Name:  standard-rwo
  Termination Policy:        WipeOut
  Tolerations:
Status:
  Cluster Def Generation:  2
  Components:
    Postgresql:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-09-04T08:46:41Z
      Replication Set Status:
        Primary:
          Pod:  postgres-2g757-postgresql-0
  Conditions:
    Last Transition Time:  2023-09-04T08:12:49Z
    Message:               The operator has started the provisioning of Cluster: postgres-2g757
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-09-04T09:07:27Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-09-04T08:14:04Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-09-04T08:14:04Z
    Message:               Cluster: postgres-2g757 is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Running
Events:
  Type     Reason                  Age                From                Message
  ----     ------                  ----               ----                -------
  Warning  ApplyResourcesFailed    51m                cluster-controller  Operation cannot be fulfilled on statefulsets.apps "postgres-2g757-postgresql": the object has been modified; please apply your changes to the latest version and try again
  Normal   ApplyResourcesSucceed   37m                cluster-controller  Successfully applied for resources
  Normal   WaitingForProbeSuccess  28m (x2 over 51m)  cluster-controller  Waiting for probe success
  Warning  RecreatingFailedPod     28m                event-controller    StatefulSet default/postgres-2g757-postgresql is recreating failed Pod postgres-2g757-postgresql-0

➜  ~ k logs postgres-2g757-postgresql-0
Defaulted container "postgresql" out of: postgresql, pgbouncer, metrics, kb-checkrole, config-manager, pg-init-container (init)
2023-09-04 09:17:23,483 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2023-09-04 09:17:23,494 - bootstrapping - INFO - Looks like you are running google
2023-09-04 09:17:23,518 - bootstrapping - INFO - kubeblocks generate local configuration:
bootstrap:
  dcs:
    postgresql:
      parameters:
        archive_command: /bin/true
        archive_mode: 'on'
        autovacuum_analyze_scale_factor: '0.05'
        autovacuum_max_workers: '1'
        autovacuum_vacuum_scale_factor: '0.1'
        checkpoint_completion_target: '0.95'
        hot_standby: 'on'
        log_autovacuum_min_duration: 1s
        log_checkpoints: 'True'
        log_lock_waits: 'True'
        log_min_duration_statement: '100'
        log_statement: ddl
        max_connections: '112'
        max_locks_per_transaction: '128'
        max_prepared_transactions: '0'
        max_replication_slots: '16'
        max_wal_senders: '24'
        max_worker_processes: '8'
        track_commit_timestamp: 'True'
        track_functions: all
        wal_keep_size: 20GB
        wal_level: replica
        wal_log_hints: 'on'
  initdb:
  - auth-host: md5
  - auth-local: trust
postgresql:
  config_dir: /home/postgres/pgdata/conf
  custom_conf: /home/postgres/conf/postgresql.conf
  parameters:
    pg_stat_statements.track_utility: 'False'
    shared_buffers: 256MB
  pg_hba:
  - host     all             all             0.0.0.0/0               trust
  - host     all             all             ::/0                    trust
  - local    all             all                                     trust
  - host     all             all             127.0.0.1/32            trust
  - host     all             all             ::1/128                 trust
  - local     replication     all                                    trust
  - host      replication     all             0.0.0.0/0               md5
  - host      replication     all             ::/0                    md5

2023-09-04 09:17:23,532 - bootstrapping - INFO - Configuring crontab
2023-09-04 09:17:23,532 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2023-09-04 09:17:23,532 - bootstrapping - INFO - Configuring bootstrap
2023-09-04 09:17:23,532 - bootstrapping - INFO - Configuring standby-cluster
2023-09-04 09:17:23,533 - bootstrapping - INFO - Configuring certificate
2023-09-04 09:17:23,533 - bootstrapping - INFO - Generating ssl self-signed certificate
2023-09-04 09:17:23,823 - bootstrapping - INFO - Configuring pgqd
2023-09-04 09:17:23,823 - bootstrapping - INFO - Configuring pam-oauth2
2023-09-04 09:17:23,824 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring pgbouncer
2023-09-04 09:17:23,824 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring wal-e
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring log
2023-09-04 09:17:23,824 - bootstrapping - INFO - Configuring patroni
2023-09-04 09:17:23,833 - bootstrapping - INFO - Writing to file /run/postgres.yml
2023-09-04 09:17:25,516 INFO: Selected new K8s API server endpoint https://10.128.0.48:443
2023-09-04 09:17:25,783 INFO: No PostgreSQL configuration items changed, nothing to reload.
2023-09-04 09:17:25,790 WARNING: Postgresql is not running.
2023-09-04 09:17:25,790 INFO: Lock owner: None; I am postgres-2g757-postgresql-0
2023-09-04 09:17:25,791 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7274880837373194364
  Database cluster state: shut down
  pg_control last modified: Mon Sep  4 09:16:25 2023
  Latest checkpoint location: 0/4000028
  Latest checkpoint's REDO location: 0/4000028
  Latest checkpoint's REDO WAL file: 000000020000000000000004
  Latest checkpoint's TimeLineID: 2
  Latest checkpoint's PrevTimeLineID: 2
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:1287
  Latest checkpoint's NextOID: 16818
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 0
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 734
  Latest checkpoint's newestCommitTsXid: 1286
  Time of latest checkpoint: Mon Sep  4 09:16:25 2023
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 112
  max_worker_processes setting: 8
  max_wal_senders setting: 24
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 128
  track_commit_timestamp setting: on
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 0
  Mock authentication nonce: 7f0a2c2e5bd8aa26aded1b3e7c036431e75d1960203084f0900ef2fe3cb69fe9

2023-09-04 09:17:25,799 INFO: Lock owner: None; I am postgres-2g757-postgresql-0
2023-09-04 09:17:26,054 INFO: starting as a secondary
2023-09-04 09:17:26 GMT [91]: [1-1] 64f5a0a6.5b 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2023-09-04 09:17:26 GMT [91]: [2-1] 64f5a0a6.5b 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2023-09-04 09:17:26,305 INFO: postmaster pid=91
/var/run/postgresql:5432 - no response
2023-09-04 09:17:26 GMT [91]: [3-1] 64f5a0a6.5b 0     LOG:  redirecting log output to logging collector process
2023-09-04 09:17:26 GMT [91]: [4-1] 64f5a0a6.5b 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2023-09-04 09:17:27,347 INFO: establishing a new patroni connection to the postgres cluster
2023-09-04 09:17:27,433 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2023-09-04 09:17:28,079 INFO: promoted self to leader by acquiring session lock
server promoting
2023-09-04 09:17:28,081 INFO: cleared rewind state after becoming the leader
DO
NOTICE:  role "admin" is already a member of role "cron_admin"
GRANT ROLE
DO
DO
NOTICE:  extension "pg_auth_mon" already exists, skipping
CREATE EXTENSION
NOTICE:  version "1.1" of extension "pg_auth_mon" is already installed
ALTER EXTENSION
GRANT
NOTICE:  extension "pg_cron" already exists, skipping
CREATE EXTENSION
DO
NOTICE:  version "1.5" of extension "pg_cron" is already installed
ALTER EXTENSION
ALTER POLICY
REVOKE
GRANT
REVOKE
GRANT
ALTER POLICY
REVOKE
GRANT
CREATE FUNCTION
REVOKE
GRANT
REVOKE
GRANT
NOTICE:  extension "file_fdw" already exists, skipping
CREATE EXTENSION
DO
NOTICE:  relation "postgres_log" already exists, skipping
CREATE TABLE
GRANT
NOTICE:  column "backend_type" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE:  column "leader_pid" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE:  column "query_id" of relation "postgres_log" already exists, skipping
ALTER TABLE
NOTICE:  relation "postgres_log_0" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_1" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_2" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_3" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_4" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_5" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_6" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
NOTICE:  relation "postgres_log_7" already exists, skipping
CREATE FOREIGN TABLE
GRANT
CREATE VIEW
ALTER VIEW
GRANT
RESET
SET
NOTICE:  drop cascades to 5 other objects
DETAIL:  drop cascades to type zmon_utils.system_information
drop cascades to function zmon_utils.get_database_cluster_information()
drop cascades to function zmon_utils.get_database_cluster_system_information()
drop cascades to function zmon_utils.get_last_status_active_cronjobs()
drop cascades to view zmon_utils.last_status_active_cronjobs
DROP SCHEMA
NOTICE:  extension "plpython3u" already exists, skipping
DO
NOTICE:  language "plpythonu" does not exist, skipping
DROP LANGUAGE
NOTICE:  function plpython_call_handler() does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_inline_handler(internal) does not exist, skipping
DROP FUNCTION
NOTICE:  function plpython_validator(oid) does not exist, skipping
DROP FUNCTION
CREATE SCHEMA
GRANT
SET
CREATE TYPE
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
REVOKE
CREATE VIEW
REVOKE
GRANT
GRANT
You are now connected to database "postgres" as user "postgres".
NOTICE:  schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
NOTICE:  extension "pg_stat_statements" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pg_stat_kcache" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE:  version "3.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE:  schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
You are now connected to database "template1" as user "postgres".
NOTICE:  schema "user_management" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
CREATE FUNCTION
REVOKE
GRANT
COMMENT
GRANT
RESET
NOTICE:  extension "pg_stat_statements" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "pg_stat_kcache" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "set_user" already exists, skipping
CREATE EXTENSION
NOTICE:  version "3.0" of extension "set_user" is already installed
ALTER EXTENSION
GRANT
GRANT
GRANT
NOTICE:  schema "metric_helpers" already exists, skipping
CREATE SCHEMA
GRANT
SET
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
REVOKE
GRANT
REVOKE
GRANT
RESET
2023-09-04 09:17:29,917 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:17:41,173 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:17:50,012 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:17:59,737 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:09,861 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:19,849 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:24.902 UTC [60] LOG Starting pgqd 3.5
2023-09-04 09:18:24.902 UTC [60] LOG auto-detecting dbs ...
2023-09-04 09:18:30,361 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:40,085 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:51,458 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:18:54.932 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:18:59,653 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:09,720 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:20,911 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:24.937 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:19:30,140 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:39,858 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:50,245 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:19:54.966 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:19:59,896 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:10,038 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:19,835 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:24.971 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:20:31,033 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:40,096 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:51,082 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:20:55.005 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:20:59,887 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:10,007 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:21,008 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:25.006 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:21:29,765 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:40,422 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:50,094 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:21:55.036 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:21:59,747 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:09,830 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:20,097 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:25.047 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:22:30,209 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:39,853 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:49,807 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:22:55.077 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:22:59,720 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:10,020 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:19,848 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:25.082 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:23:30,012 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:39,955 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:51,392 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:23:55.111 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:23:59,825 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:09,955 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:20,586 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:25.117 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:24:29,707 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:39,791 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:50,486 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:24:55.146 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:24:59,901 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:11,196 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:19,769 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:25.151 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:25:30,007 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:39,815 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:49,970 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:25:55.180 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:26:00,151 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:09,759 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:19,766 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:25.185 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:26:29,654 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:39,728 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:49,768 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:26:55.215 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:26:59,851 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:09,770 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:19,774 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:25.220 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:27:29,768 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:40,045 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:49,851 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:27:55.250 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:27:59,720 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:10,547 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:20,069 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:25.259 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:28:29,966 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:39,762 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:49,971 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:28:55.288 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:28:59,663 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:09,884 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:21,186 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:25.294 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:29:30,431 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:40,693 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:50,708 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:29:55.323 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:30:00,522 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:10,499 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:19,831 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:25.329 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:30:29,922 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:39,952 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:50,410 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:30:55.359 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:30:59,654 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:09,741 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:19,713 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:25.363 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:31:29,707 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:39,745 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:49,854 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:31:55.392 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:32:00,071 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:09,741 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:20,392 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:25.401 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:32:30,275 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:39,675 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:50,035 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:32:55.431 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:33:00,034 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:10,070 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:20,751 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:25.437 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:33:30,546 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:39,838 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:50,362 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:33:55.467 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:33:59,929 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:09,789 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:19,890 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:25.489 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:34:30,090 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:39,925 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:49,876 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:34:55.519 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:34:59,722 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:09,804 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:19,862 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:25.524 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:35:29,984 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:40,091 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:49,675 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:35:55.553 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:35:59,788 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:12,913 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:20,957 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:25.559 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:36:29,824 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:40,283 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:49,671 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:36:55.588 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:36:59,827 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:09,784 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:20,169 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:25.593 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:37:30,003 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:39,743 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:49,883 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:37:55.622 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:37:59,815 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:09,898 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:21,365 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:25.626 UTC [60] LOG {ticks: 0, maint: 0, retry: 0}
2023-09-04 09:38:30,278 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:40,228 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
2023-09-04 09:38:50,733 INFO: no action. I am (postgres-2g757-postgresql-0), the leader with the lock
github-actions[bot] commented 1 year ago

This issue has been marked as stale because it has been open for 30 days with no activity