apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.09k stars 170 forks source link

[BUG] postgresql 16.4.0 pg_basebackup: error: incompatible server version 16.4 (Ubuntu 16.4-1.pgdg22.04+1) #8258

Closed JashBook closed 1 week ago

JashBook commented 1 week ago

Describe the bug

kbcli version
Kubernetes: v1.31.0
KubeBlocks: 0.9.1
kbcli: 0.9.1

To Reproduce Steps to reproduce the behavior:

  1. create cluster with postgresql-16.4.0
    
    kbcli cluster create  postgres-sqldnk --termination-policy=Halt --cluster-definition=postgresql --enable-all-logs=false --cluster-version=postgresql-16.4.0 --set cpu=100m,memory=0.5Gi,replicas=2,storage=3Gi  --namespace default

apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: labels: clusterdefinition.kubeblocks.io/name: postgresql clusterversion.kubeblocks.io/name: postgresql-16.4.0 name: postgres-sqldnk namespace: default spec: clusterDefinitionRef: postgresql clusterVersionRef: postgresql-16.4.0 componentSpecs:

kubectl get backup NAME POLICY METHOD REPO STATUS TOTAL-SIZE DURATION CREATION-TIME COMPLETION-TIME EXPIRATION-TIME backup-default-postgres-sqldnk-20241010185004 postgres-sqldnk-postgresql-backup-policy pg-basebackup backuprepo-kbcli-test Failed 2024-10-10T10:50:04Z

logs error pod 

kubectl logs dp-backup-0-backup-default-postgres-sqldnk-20241010185004-mpvnw Defaulted container "backupdata" out of: backupdata, manager, dp-copy-datasafed (init) Password: pg_basebackup: error: incompatible server version 16.4 (Ubuntu 16.4-1.pgdg22.04+1) failed with exit code 1

describe error pod 

kubectl describe pod dp-backup-0-backup-default-postgres-sqldnk-20241010185004-mpvnw Name: dp-backup-0-backup-default-postgres-sqldnk-20241010185004-mpvnw Namespace: default Priority: 0 Service Account: kubeblocks-dataprotection-worker Node: minikube/192.168.49.2 Start Time: Thu, 10 Oct 2024 18:50:04 +0800 Labels: app.kubernetes.io/instance=postgres-sqldnk app.kubernetes.io/managed-by=kubeblocks-dataprotection apps.kubeblocks.io/sharding-name= batch.kubernetes.io/controller-uid=fefbfd37-140e-4abd-8584-1327afe95a24 batch.kubernetes.io/job-name=dp-backup-0-backup-default-postgres-sqldnk-20241010185004-00ff6 controller-uid=fefbfd37-140e-4abd-8584-1327afe95a24 dataprotection.kubeblocks.io/backup-name=backup-default-postgres-sqldnk-20241010185004 dataprotection.kubeblocks.io/backup-policy=postgres-sqldnk-postgresql-backup-policy dataprotection.kubeblocks.io/backup-repo-name=backuprepo-kbcli-test dataprotection.kubeblocks.io/backup-type=Full dataprotection.kubeblocks.io/cluster-uid=bbc03428-2589-4e96-8a2e-2a6cc5093fdd job-name=dp-backup-0-backup-default-postgres-sqldnk-20241010185004-00ff6 kubeblocks.io/backup-protection=Retain ops.kubeblocks.io/ops-name=backup-default-postgres-sqldnk-20241010185004 ops.kubeblocks.io/ops-type=Backup Annotations: Status: Failed IP: 10.244.0.98 IPs: IP: 10.244.0.98 Controlled By: Job/dp-backup-0-backup-default-postgres-sqldnk-20241010185004-00ff6 Init Containers: dp-copy-datasafed: Container ID: docker://ad1cf6dfa74ae975413bab94d1a4f9e9dc9ae751bf1084850dc6df1a375ee16c Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/datasafed:0.2.0 Image ID: docker-pullable://apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/datasafed@sha256:ee48f0674aa4001ac0ce959c31f74dbe0159f5ddb317ef6a9f09454bdcfc418b Port: Host Port: Command: /bin/sh -c /scripts/install-datasafed.sh /bin/datasafed State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 10 Oct 2024 18:50:05 +0800 Finished: Thu, 10 Oct 2024 18:50:05 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment: Mounts: /bin/datasafed from dp-datasafed-bin (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfxvc (ro) Containers: backupdata: Container ID: docker://5cc6ac19cf06b6868783c9d156263a7ea2b0b192836ff23c2cfd53183bd18fe2 Image: docker.io/apecloud/spilo:14.8.0-pgvector-v0.6.1 Image ID: docker-pullable://apecloud/spilo@sha256:a2450aec1532c97a8a4c6b47cbf9c79e1bb11383595beff924410361ea53ae6e Port: Host Port: Command: bash -c function get_current_time() { curr_time=$(psql -U ${DP_DB_USER} -h ${DP_DB_HOST} -d postgres -t -c "SELECT now() AT TIME ZONE 'UTC'") echo $curr_time }

  function stat_and_save_backup_info() {
    export PATH="$PATH:$DP_DATASAFED_BIN_PATH"
    export DATASAFED_BACKEND_BASE_PATH="$DP_BACKUP_BASE_PATH"
    START_TIME=$1
    STOP_TIME=$2
    if [ -z $STOP_TIME ]; then
      STOP_TIME=$(get_current_time)
    fi
    START_TIME=$(date -d "${START_TIME}" -u '+%Y-%m-%dT%H:%M:%SZ')
    STOP_TIME=$(date -d "${STOP_TIME}" -u '+%Y-%m-%dT%H:%M:%SZ')
    TOTAL_SIZE=$(datasafed stat / | grep TotalSize | awk '{print $2}')
    echo "{\"totalSize\":\"$TOTAL_SIZE\",\"timeRange\":{\"start\":\"${START_TIME}\",\"end\":\"${STOP_TIME}\"}}" >"${DP_BACKUP_INFO_FILE}"
  }

  # if the script exits with a non-zero exit code, touch a file to indicate that the backup failed,
  # the sync progress container will check this file and exit if it exists
  function handle_exit() {
    exit_code=$?
    if [ $exit_code -ne 0 ]; then
      echo "failed with exit code $exit_code"
      touch "${DP_BACKUP_INFO_FILE}.exit"
      exit 1
    fi
  }

  set -e
  set -o pipefail
  export PATH="$PATH:$DP_DATASAFED_BIN_PATH"
  export DATASAFED_BACKEND_BASE_PATH="$DP_BACKUP_BASE_PATH"
  export PGPASSWORD=${DP_DB_PASSWORD}

  trap handle_exit EXIT

  START_TIME=`get_current_time`
  echo ${DP_DB_PASSWORD} | pg_basebackup -Ft -Pv -c fast -Xf -D - -h ${DP_DB_HOST} -U ${DP_DB_USER} -W | datasafed push -z zstd-fastest - "/${DP_BACKUP_NAME}.tar.zst"

  # stat and save the backup information
  stat_and_save_backup_info "$START_TIME"

State:          Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 10 Oct 2024 18:51:05 +0800
  Finished:     Thu, 10 Oct 2024 18:51:05 +0800
Ready:          False
Restart Count:  0
Limits:
  cpu:     0
  memory:  0
Requests:
  cpu:     0
  memory:  0
Environment Variables from:
  postgres-sqldnk-postgresql-env      ConfigMap  Optional: false
  postgres-sqldnk-postgresql-rsm-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:                dp-backup-0-backup-default-postgres-sqldnk-20241010185004-mpvnw (v1:metadata.name)
  KB_POD_UID:                  (v1:metadata.uid)
  KB_NAMESPACE:               default (v1:metadata.namespace)
  KB_SA_NAME:                  (v1:spec.serviceAccountName)
  KB_NODENAME:                 (v1:spec.nodeName)
  KB_HOST_IP:                  (v1:status.hostIP)
  KB_POD_IP:                   (v1:status.podIP)
  KB_POD_IPS:                  (v1:status.podIPs)
  KB_HOSTIP:                   (v1:status.hostIP)
  KB_PODIP:                    (v1:status.podIP)
  KB_PODIPS:                   (v1:status.podIPs)
  KB_POD_FQDN:                $(KB_POD_NAME).postgres-sqldnk-postgresql-headless.$(KB_NAMESPACE).svc
  SERVICE_PORT:               5432
  DCS_ENABLE_KUBERNETES_API:  true
  KUBERNETES_USE_CONFIGMAPS:  true
  SCOPE:                      $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni$(KB_CLUSTER_UID_POSTFIX_8)
  KUBERNETES_SCOPE_LABEL:     apps.kubeblocks.postgres.patroni/scope
  KUBERNETES_ROLE_LABEL:      apps.kubeblocks.postgres.patroni/role
  KUBERNETES_LABELS:          {"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)"}
  RESTORE_DATA_DIR:           /home/postgres/pgdata/kb_restore
  KB_PG_CONFIG_PATH:          /home/postgres/conf/postgresql.conf
  SPILO_CONFIGURATION:        bootstrap:
                                initdb:
                                  - auth-host: md5
                                  - auth-local: trust

  ALLOW_NOSSL:                true
  PGROOT:                     /home/postgres/pgdata/pgroot
  POD_IP:                      (v1:status.podIP)
  POD_NAMESPACE:              default (v1:metadata.namespace)
  PGUSER_SUPERUSER:           <set to the key 'username' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGPASSWORD_SUPERUSER:       <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGUSER_ADMIN:               superadmin
  PGPASSWORD_ADMIN:           <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGUSER_STANDBY:             standby
  PGPASSWORD_STANDBY:         <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGUSER:                     <set to the key 'username' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGPASSWORD:                 <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  DP_BACKUP_NAME:             backup-default-postgres-sqldnk-20241010185004
  DP_PARENT_BACKUP_NAME:      
  DP_TARGET_POD_NAME:         postgres-sqldnk-postgresql-1
  DP_TARGET_POD_ROLE:         secondary
  DP_BACKUP_BASE_PATH:        /default/postgres-sqldnk-bbc03428-2589-4e96-8a2e-2a6cc5093fdd/postgresql/backup-default-postgres-sqldnk-20241010185004
  DP_BACKUP_INFO_FILE:        /dp-manager/backup.info
  DP_TTL:                     
  DP_DB_HOST:                 postgres-sqldnk-postgresql-1.postgres-sqldnk-postgresql-headless
  DP_DB_PASSWORD:             <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  DP_DB_USER:                 <set to the key 'username' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  DP_DB_PORT:                 5432
  DATA_DIR:                   /home/postgres/pgdata/pgroot/data
  IMAGE_TAG:                  14.8.0-pgvector-v0.6.1
  KB_CLUSTER_UID:             bbc03428-2589-4e96-8a2e-2a6cc5093fdd
  KB_CLUSTER_NAME:            postgres-sqldnk
  KB_COMP_NAME:               postgresql
  KB_NAMESPACE:               default
  DP_DATASAFED_BIN_PATH:      /bin/datasafed
Mounts:
  /bin/datasafed from dp-datasafed-bin (rw)
  /dp-manager from manager-shared-volume (rw)
  /etc/datasafed from dp-datasafed-config (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfxvc (ro)

manager: Container ID: docker://b5633529ebbc051ff9c11fad88c9cebc714923890a6519d25a76bfd2ea8119c9 Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1 Image ID: docker-pullable://apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6afd857c62229b58eec81cfd51deb27cedd03630b471fad57a46c3d7da9662fd Port: Host Port: Command: sh -c Args:

  set -o errexit
  set -o nounset

  export PATH="$PATH:$DP_DATASAFED_BIN_PATH"
  export DATASAFED_BACKEND_BASE_PATH="$DP_BACKUP_BASE_PATH"

  backup_info_file="${DP_BACKUP_INFO_FILE}"
  sleep_seconds="${DP_CHECK_INTERVAL}"
  namespace="default"
  backup_name="backup-default-postgres-sqldnk-20241010185004"

  if [ "$sleep_seconds" -le 0 ]; then
    sleep_seconds=30
  fi

  exit_file="${backup_info_file}.exit"
  while true; do
    if [ -f "$exit_file" ]; then
      echo "exit file $exit_file exists, exit"
      exit 1
    fi
    if [ -f "$backup_info_file" ]; then
      break
    fi
    echo "backup info file not exists, wait for ${sleep_seconds}s"
    sleep "$sleep_seconds"
  done

  backup_info=$(cat "$backup_info_file")
  echo "backupInfo:${backup_info}"

  status="{\"status\":${backup_info}}"
  kubectl -n "$namespace" patch backups.dataprotection.kubeblocks.io "$backup_name" --subresource=status --type=merge --patch "${status}"

  # save the backup CR object to the backup repo
  kubectl -n "$namespace" get backups.dataprotection.kubeblocks.io "$backup_name" -o json | datasafed push - "/kubeblocks-backup.json"

State:          Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 10 Oct 2024 18:51:05 +0800
  Finished:     Thu, 10 Oct 2024 18:51:10 +0800
Ready:          False
Restart Count:  0
Limits:
  cpu:     0
  memory:  0
Requests:
  cpu:     0
  memory:  0
Environment Variables from:
  postgres-sqldnk-postgresql-env      ConfigMap  Optional: false
  postgres-sqldnk-postgresql-rsm-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:                dp-backup-0-backup-default-postgres-sqldnk-20241010185004-mpvnw (v1:metadata.name)
  KB_POD_UID:                  (v1:metadata.uid)
  KB_NAMESPACE:               default (v1:metadata.namespace)
  KB_SA_NAME:                  (v1:spec.serviceAccountName)
  KB_NODENAME:                 (v1:spec.nodeName)
  KB_HOST_IP:                  (v1:status.hostIP)
  KB_POD_IP:                   (v1:status.podIP)
  KB_POD_IPS:                  (v1:status.podIPs)
  KB_HOSTIP:                   (v1:status.hostIP)
  KB_PODIP:                    (v1:status.podIP)
  KB_PODIPS:                   (v1:status.podIPs)
  KB_POD_FQDN:                $(KB_POD_NAME).postgres-sqldnk-postgresql-headless.$(KB_NAMESPACE).svc
  SERVICE_PORT:               5432
  DCS_ENABLE_KUBERNETES_API:  true
  KUBERNETES_USE_CONFIGMAPS:  true
  SCOPE:                      $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni$(KB_CLUSTER_UID_POSTFIX_8)
  KUBERNETES_SCOPE_LABEL:     apps.kubeblocks.postgres.patroni/scope
  KUBERNETES_ROLE_LABEL:      apps.kubeblocks.postgres.patroni/role
  KUBERNETES_LABELS:          {"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)"}
  RESTORE_DATA_DIR:           /home/postgres/pgdata/kb_restore
  KB_PG_CONFIG_PATH:          /home/postgres/conf/postgresql.conf
  SPILO_CONFIGURATION:        bootstrap:
                                initdb:
                                  - auth-host: md5
                                  - auth-local: trust

  ALLOW_NOSSL:                true
  PGROOT:                     /home/postgres/pgdata/pgroot
  POD_IP:                      (v1:status.podIP)
  POD_NAMESPACE:              default (v1:metadata.namespace)
  PGUSER_SUPERUSER:           <set to the key 'username' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGPASSWORD_SUPERUSER:       <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGUSER_ADMIN:               superadmin
  PGPASSWORD_ADMIN:           <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGUSER_STANDBY:             standby
  PGPASSWORD_STANDBY:         <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGUSER:                     <set to the key 'username' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  PGPASSWORD:                 <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  DP_BACKUP_NAME:             backup-default-postgres-sqldnk-20241010185004
  DP_PARENT_BACKUP_NAME:      
  DP_TARGET_POD_NAME:         postgres-sqldnk-postgresql-1
  DP_TARGET_POD_ROLE:         secondary
  DP_BACKUP_BASE_PATH:        /default/postgres-sqldnk-bbc03428-2589-4e96-8a2e-2a6cc5093fdd/postgresql/backup-default-postgres-sqldnk-20241010185004
  DP_BACKUP_INFO_FILE:        /dp-manager/backup.info
  DP_TTL:                     
  DP_DB_HOST:                 postgres-sqldnk-postgresql-1.postgres-sqldnk-postgresql-headless
  DP_DB_PASSWORD:             <set to the key 'password' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  DP_DB_USER:                 <set to the key 'username' in secret 'postgres-sqldnk-conn-credential'>  Optional: false
  DP_DB_PORT:                 5432
  DATA_DIR:                   /home/postgres/pgdata/pgroot/data
  IMAGE_TAG:                  14.8.0-pgvector-v0.6.1
  KB_CLUSTER_UID:             bbc03428-2589-4e96-8a2e-2a6cc5093fdd
  KB_CLUSTER_NAME:            postgres-sqldnk
  KB_COMP_NAME:               postgresql
  KB_NAMESPACE:               default
  DP_DATASAFED_BIN_PATH:      /bin/datasafed
  DP_CHECK_INTERVAL:          5
Mounts:
  /bin/datasafed from dp-datasafed-bin (rw)
  /dp-manager from manager-shared-volume (rw)
  /etc/datasafed from dp-datasafed-config (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfxvc (ro)

Conditions: Type Status PodReadyToStartContainers False Initialized True Ready False ContainersReady False PodScheduled True Volumes: manager-shared-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: dp-datasafed-config: Type: Secret (a volume populated by a Secret) SecretName: tool-config-backuprepo-kbcli-test-ktq4wm Optional: false dp-datasafed-bin: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: kube-api-access-gfxvc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: kb-controller=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message


Normal Scheduled 3m13s default-scheduler Successfully assigned default/dp-backup-0-backup-default-postgres-sqldnk-20241010185004-mpvnw to minikube Normal Pulled 3m13s kubelet Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/datasafed:0.2.0" already present on machine Normal Created 3m13s kubelet Created container dp-copy-datasafed Normal Started 3m13s kubelet Started container dp-copy-datasafed Normal Pulling 3m13s kubelet Pulling image "docker.io/apecloud/spilo:14.8.0-pgvector-v0.6.1" Normal Pulled 2m14s kubelet Successfully pulled image "docker.io/apecloud/spilo:14.8.0-pgvector-v0.6.1" in 58.568s (58.568s including waiting). Image size: 689401295 bytes. Normal Created 2m13s kubelet Created container backupdata Normal Started 2m13s kubelet Started container backupdata Normal Pulled 2m13s kubelet Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1" already present on machine Normal Created 2m13s kubelet Created container manager Normal Started 2m13s kubelet Started container manager


get bpt yaml

kubectl get backuppolicytemplates postgresql-backup-policy-template -oyaml apiVersion: apps.kubeblocks.io/v1alpha1 kind: BackupPolicyTemplate metadata: annotations: dataprotection.kubeblocks.io/reconfigure-ref: | { "name": "postgresql-configuration", "key": "postgresql.conf", "enable": { "archive-wal": [{"key": "archive_command","value": "''"}] }, "disable": { "archive-wal": [{ "key": "archive_command","value": "'/bin/true'"}] } } meta.helm.sh/release-name: kb-addon-postgresql meta.helm.sh/release-namespace: kb-system creationTimestamp: "2024-10-10T10:42:02Z" generation: 2 labels: app.kubernetes.io/instance: kb-addon-postgresql app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: postgresql app.kubernetes.io/version: 14.7.2 clusterdefinition.kubeblocks.io/name: postgresql helm.sh/chart: postgresql-0.9.0 postgresql-12: postgresql-12 postgresql-14: postgresql-14 service-kind/postgresql: "true" name: postgresql-backup-policy-template resourceVersion: "68564" uid: 5df68965-68e1-4be0-bbf5-7f57620d116b spec: backupPolicies:



**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
wangyelei commented 1 week ago

fixed