apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
1.75k stars 154 forks source link

[BUG] pulsar component bookies CrashLoopBackOff upgrade KB from 0.8.3 to 0.9.0 #7672

Open JashBook opened 5 days ago

JashBook commented 5 days ago

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. install kb 0.8.3
    
    curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.8.4-beta.1

kbcli kubeblocks install --create-namespace --version 0.8.3 --set image.registry=docker.io --set dataProtection.image.registry=docker.io --set addonChartsImage.registry=docker.io --set dataProtection.image.datasafed.tag=0.1.0 --namespace kb-cluster

2. create cluster pulsar

apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: pulsar-cluster namespace: ns-cluster annotations: "kubeblocks.io/extra-env": '"KB_PULSAR_BROKER_NODEPORT": "true"' spec: services:

kbcli kubeblocks upgrade --auto-approve --set upgradeAddons=true --version 0.9.0-beta.41 --set image.registry=docker.io --set dataProtection.image.registry=docker.io --set addonChartsImage.registry=docker.io --set dataProtection.image.datasafed.tag=0.2.0 --namespace kb-cluster

4. `kbcli cluster hscale pulsar-cluster -n ns-cluster --replicas 5 --components bookies --auto-approve`
5. See error

kubectl get pod -l app.kubernetes.io/instance=pulsar-cluster -n ns-cluster NAME READY STATUS RESTARTS AGE pulsar-cluster-bookies-0 2/2 Running 0 141m pulsar-cluster-bookies-1 2/2 Running 0 142m pulsar-cluster-bookies-2 2/2 Running 0 142m pulsar-cluster-bookies-3 1/1 Running 0 63m pulsar-cluster-bookies-4 0/1 CrashLoopBackOff 16 (2m53s ago) 63m pulsar-cluster-pulsar-broker-0 2/2 Running 0 61m pulsar-cluster-pulsar-broker-1 2/2 Running 0 62m pulsar-cluster-pulsar-proxy-0 1/1 Running 0 63m pulsar-cluster-zookeeper-0 1/1 Running 0 62m pulsar-cluster-zookeeper-1 1/1 Running 0 62m pulsar-cluster-zookeeper-2 1/1 Running 0 63m

6.  describe pod

kubectl describe pod -n ns-cluster pulsar-cluster-bookies-4
Name: pulsar-cluster-bookies-4 Namespace: ns-cluster Priority: 0 Node: ip-172-31-15-87.us-west-2.compute.internal/172.31.15.87 Start Time: Fri, 28 Jun 2024 17:11:08 +0800 Labels: app.kubernetes.io/component=bookies app.kubernetes.io/instance=pulsar-cluster app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=pulsar app.kubernetes.io/version= apps.kubeblocks.io/cluster-uid=cd4e8dd9-49af-4526-bab3-42ced2f771f1 apps.kubeblocks.io/component-name=bookies apps.kubeblocks.io/pod-name=pulsar-cluster-bookies-4 clusterdefinition.kubeblocks.io/name=pulsar clusterversion.kubeblocks.io/name=pulsar-3.0.2 controller-revision-hash=7b784bd9bf workloads.kubeblocks.io/instance=pulsar-cluster-bookies workloads.kubeblocks.io/managed-by=InstanceSet Annotations: apps.kubeblocks.io/component-replicas: 5 Status: Running IP: 172.31.10.63 IPs: IP: 172.31.10.63 Controlled By: InstanceSet/pulsar-cluster-bookies Init Containers: init-bookies: Container ID: containerd://85ab489d6e4a0f6d4cffb3786a253006674aa191093c7574ef5f39d8b72309a7 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:3.0.2 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:f73e9f139b0c0dfa3419401556359c3ca205e7f1732b3dc7372a40f36c2c0b7a Port: Host Port: Command: /kb-scripts/init-bookies.sh State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 28 Jun 2024 17:11:17 +0800 Finished: Fri, 28 Jun 2024 17:11:43 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: pulsar-cluster-bookies-env ConfigMap Optional: false pulsar-cluster-bookies-bookies-env-envfrom ConfigMap Optional: false Environment: KB_POD_NAME: pulsar-cluster-bookies-4 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-cluster (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).pulsar-cluster-bookies-headless.$(KB_NAMESPACE).svc Mounts: /kb-scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dssbp (ro) Containers: bookies: Container ID: containerd://b8e240c0d1c07b8f34ad45366c24beeb36e90a595afccaa528764fcb3e9fd644 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:3.0.2 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar@sha256:f73e9f139b0c0dfa3419401556359c3ca205e7f1732b3dc7372a40f36c2c0b7a Ports: 8000/TCP, 3181/TCP Host Ports: 0/TCP, 0/TCP Command: /kb-scripts/start-bookies.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Fri, 28 Jun 2024 18:11:06 +0800 Finished: Fri, 28 Jun 2024 18:11:16 +0800 Ready: False Restart Count: 16 Limits: cpu: 600m memory: 644245094400m Requests: cpu: 600m memory: 644245094400m Liveness: http-get http://:http/api/v1/bookie/state delay=10s timeout=5s period=30s #success=1 #failure=60 Readiness: http-get http://:http/api/v1/bookie/is_ready delay=10s timeout=5s period=30s #success=1 #failure=60 Environment Variables from: pulsar-cluster-bookies-env ConfigMap Optional: false pulsar-cluster-bookies-bookies-env-envfrom ConfigMap Optional: false pulsar-cluster-bookies-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: pulsar-cluster-bookies-4 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-cluster (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).pulsar-cluster-bookies-headless.$(KB_NAMESPACE).svc SERVICE_PORT: 8000 cluster_domain: .cluster.local Mounts: /etc/annotations from annotations (rw) /kb-scripts from scripts (rw) /opt/pulsar/conf from pulsar-bookies-config (rw) /opt/pulsar/none from bookies-env (rw) /pulsar/data/bookkeeper/journal from journal (rw) /pulsar/data/bookkeeper/ledgers from ledgers (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dssbp (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: annotations: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas pulsar-bookies-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsar-cluster-bookies-bookies-config Optional: false bookies-env: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsar-cluster-bookies-bookies-env Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsar-cluster-bookies-pulsar-scripts Optional: false journal: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: journal-pulsar-cluster-bookies-4 ReadOnly: false ledgers: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: ledgers-pulsar-cluster-bookies-4 ReadOnly: false kube-api-access-dssbp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message


Warning BackOff 4m4s (x273 over 63m) kubelet Back-off restarting failed container bookies in pod pulsar-cluster-bookies-4_ns-cluster(02d7799d-9f79-446b-b5c6-757742178025)

6. logs pod

kubectl logs -n ns-cluster pulsar-cluster-bookies-4 --previous

2024-06-28T10:11:14,423+0000 [main] INFO org.apache.bookkeeper.server.Main - Using configuration file /pulsar/conf/bookkeeper.conf 2024-06-28T10:11:14,443+0000 [main] INFO org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is , listening on port 3181. Metadata service uri is zk+null://pulsar-cluster-zookeeper.ns-cluster.svc:2181/ledgers. Journals are in [/pulsar/data/bookkeeper/journal]. Ledgers are stored in [/pulsar/data/bookkeeper/ledgers]. Indexes are stored in [/pulsar/data/bookkeeper/ledgers]. 2024-06-28T10:11:14,542+0000 [main] INFO org.apache.bookkeeper.server.EmbeddedServer$Builder - Load lifecycle component : stats-provider 2024-06-28T10:11:14,733+0000 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2024-06-28T10:11:14,741+0000 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsar-cluster-zookeeper.ns-cluster.svc:2181/ledgers : zkServers = pulsar-cluster-zookeeper.ns-cluster.svc:2181, ledgersRootPath = /ledgers. 2024-06-28T10:11:14,747+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC 2024-06-28T10:11:14,747+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-cluster-bookies-4.pulsar-cluster-bookies-headless.ns-cluster.svc.cluster.local 2024-06-28T10:11:14,748+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7 2024-06-28T10:11:14,748+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian 2024-06-28T10:11:14,820+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64 2024-06-28T10:11:14,821+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/com.auth0-java-jwt-4.3.0.jar:/pulsar/lib/com.auth0-jwks-rsa-0.22.0.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/com.google.re2j-re2j-1.6.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/com.squareup.okio-okio-3.4.0.jar:/pulsar/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/io.etcd-jetcd-api-0.7.5.jar:/pulsar/lib/io.etcd-jetcd-common-0.7.5.jar:/pulsar/lib/io.etcd-jetcd-core-0.7.5.jar:/pulsar/lib/io.etcd-jetcd-grpc-0.7.5.jar:/pulsar/lib/io.grpc-grpc-all-1.55.3.jar:/pulsar/lib/io.grpc-grpc-alts-1.55.3.jar:/pulsar/lib/io.grpc-grpc-api-1.55.3.jar:/pulsar/lib/io.grpc-grpc-auth-1.55.3.jar:/pulsar/lib/io.grpc-grpc-context-1.55.3.jar:/pulsar/lib/io.grpc-grpc-core-1.55.3.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.55.3.jar:/pulsar/lib/io.grpc-grpc-netty-1.55.3.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.55.3.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/pulsar/lib/io.grpc-grpc-rls-1.55.3.jar:/pulsar/lib/io.grpc-grpc-services-1.55.3.jar:/pulsar/lib/io.grpc-grpc-servlet-1.55.3.jar:/pulsar/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/pulsar/lib/io.grpc-grpc-stub-1.55.3.jar:/pulsar/lib/io.grpc-grpc-xds-1.55.3.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/io.netty-netty-buffer-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-common-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-handler-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-classes-macos-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-transport-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/io.perfmark-perfmark-api-0.26.0.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/io.vertx-vertx-auth-common-4.3.8.jar:/pulsar/lib/io.vertx-vertx-bridge-common-4.3.8.jar:/pulsar/lib/io.vertx-vertx-core-4.3.8.jar:/pulsar/lib/io.vertx-vertx-grpc-4.3.5.jar:/pulsar/lib/io.vertx-vertx-web-4.3.8.jar:/pulsar/lib/io.vertx-vertx-web-common-4.3.8.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/pulsar/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/org.apache.avro-avro-1.11.3.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.11.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-native-io-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.16.3.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3-tests.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.16.3.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-3.0.2.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-3.0.2-pkg.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-3.0.2.jar:/pulsar/lib/org.apache.pulsar-testmocks-3.0.2.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.3.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.53.v20231009.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar: 2024-06-28T10:11:14,823+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2024-06-28T10:11:14,823+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2024-06-28T10:11:14,823+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2024-06-28T10:11:14,824+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2024-06-28T10:11:14,824+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2024-06-28T10:11:14,824+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.10.219-208.866.amzn2.x86_64 2024-06-28T10:11:14,824+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2024-06-28T10:11:14,825+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2024-06-28T10:11:14,825+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar 2024-06-28T10:11:14,825+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=12MB 2024-06-28T10:11:14,825+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=308MB 2024-06-28T10:11:14,825+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=30MB 2024-06-28T10:11:14,834+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-cluster-zookeeper.ns-cluster.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@3d08f3f5 2024-06-28T10:11:14,843+0000 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2024-06-28T10:11:14,924+0000 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes 2024-06-28T10:11:14,938+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2024-06-28T10:11:15,043+0000 [main-SendThread(pulsar-cluster-zookeeper.ns-cluster.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-cluster-zookeeper.ns-cluster.svc/10.100.171.105:2181. 2024-06-28T10:11:15,044+0000 [main-SendThread(pulsar-cluster-zookeeper.ns-cluster.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2024-06-28T10:11:15,123+0000 [main-SendThread(pulsar-cluster-zookeeper.ns-cluster.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.31.10.63:54602, server: pulsar-cluster-zookeeper.ns-cluster.svc/10.100.171.105:2181 2024-06-28T10:11:15,132+0000 [main-SendThread(pulsar-cluster-zookeeper.ns-cluster.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-cluster-zookeeper.ns-cluster.svc/10.100.171.105:2181, session id = 0x100007c0f000009, negotiated timeout = 30000 2024-06-28T10:11:15,138+0000 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2024-06-28T10:11:16,229+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current] 2024-06-28T10:11:16,229+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:113) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3] at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:408) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3] at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3] at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]


**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

kbcli version Kubernetes: v1.27.14-eks-1de2ab1 KubeBlocks: 0.9.0-beta.41 kbcli: 0.9.0-beta.1



**Additional context**
Add any other context about the problem here.