search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.12k stars 175 forks source link

[BUG] pulsar cluster bookies pod CrashLoopBackOff when all ack nodes scale to 0 then recovery #4358

Closed JashBook closed 10 months ago

JashBook commented 1 year ago

Describe the bug pulsar cluster bookies pod CrashLoopBackOff when all ack nodes scale to 0 then recovery.

To Reproduce Steps to reproduce the behavior:

  1. create pulsar cluster 
    kbcli cluster create pulsar-cluster \
--cluster-definition=pulsar \
--set type=bookies,replicas=3,storage=20Gi \
--set type=zookeeper,replicas=3,storage=20Gi \
--set type=pulsar-broker,replicas=1,storage=20Gi \
--set type=pulsar-proxy,replicas=3,storage=20Gi
  2. all ack nodes scale to 0
  3. recovery ack nodes
  4. See error 
    
kubectl get cluster
NAME                             CLUSTER-DEFINITION   VERSION             TERMINATION-POLICY   STATUS    AGE
pulsar-cluster                   pulsar               pulsar-2.11.1       Delete               Failed    21h

kubectl get pod pulsar-cluster-bookies-0 1/2 CrashLoopBackOff 33 (2m16s ago) 17h pulsar-cluster-bookies-1 1/2 CrashLoopBackOff 33 (2m31s ago) 17h pulsar-cluster-bookies-2 1/2 CrashLoopBackOff 7 (4m18s ago) 15m pulsar-cluster-bookies-recovery-0 2/2 Running 0 17h pulsar-cluster-pulsar-broker-0 3/3 Running 0 17h pulsar-cluster-pulsar-proxy-0 2/2 Running 0 17h pulsar-cluster-pulsar-proxy-1 2/2 Running 0 17h pulsar-cluster-pulsar-proxy-2 2/2 Running 0 17h pulsar-cluster-zookeeper-0 2/2 Running 0 17h pulsar-cluster-zookeeper-1 2/2 Running 0 17h pulsar-cluster-zookeeper-2 2/2 Running 0 17h

describe cluster

kubectl describe cluster pulsar-cluster Name: pulsar-cluster Namespace: default Labels: clusterdefinition.kubeblocks.io/name=pulsar clusterversion.kubeblocks.io/name=pulsar-2.11.1 Annotations: kubeblocks.io/reconcile: 2023-07-19T06:42:00.672431501Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-07-18T09:09:55Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 3 Resource Version: 628396547 UID: 753f041b-48ea-4f4b-a60e-b562d8ee448a Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: pulsar Cluster Version Ref: pulsar-2.11.1 Component Specs: Component Def Ref: pulsar-broker Monitor: true Name: pulsar-broker No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 1 Memory: 1Gi Requests: Cpu: 1 Memory: 1Gi Service Account Name: kb-pulsar-cluster Services: Annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: internet Name: internet Service Type: LoadBalancer Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: pulsar-proxy Monitor: true Name: pulsar-proxy No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 1 Memory: 1Gi Requests: Cpu: 1 Memory: 1Gi Service Account Name: kb-pulsar-cluster Services: Annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: internet Name: internet Service Type: LoadBalancer Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: bookies Monitor: true Name: bookies No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 1 Memory: 1Gi Requests: Cpu: 1 Memory: 1Gi Service Account Name: kb-pulsar-cluster Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: bookies-recovery Monitor: true Name: bookies-recovery No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 1 Memory: 1Gi Requests: Cpu: 1 Memory: 1Gi Service Account Name: kb-pulsar-cluster Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: zookeeper Monitor: true Name: zookeeper No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 1 Memory: 1Gi Requests: Cpu: 1 Memory: 1Gi Service Account Name: kb-pulsar-cluster Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Status: Cluster Def Generation: 2 Components: Bookies: Message: Pod/pulsar-cluster-bookies-0: back-off 5m0s restarting failed container=bookies pod=pulsar-cluster-bookies-0_default(dae2d35c-f13b-4de0-85c6-208425204d23) Pod/pulsar-cluster-bookies-1: back-off 5m0s restarting failed container=bookies pod=pulsar-cluster-bookies-1_default(a87bae7f-9886-4dba-8596-3ee89c415f61) Pod/pulsar-cluster-bookies-2: back-off 5m0s restarting failed container=bookies pod=pulsar-cluster-bookies-2_default(118cb3df-6b37-4b50-82f2-304fbfca06e1) Phase: Failed Pods Ready: false Bookies - Recovery: Phase: Running Pods Ready: true Pods Ready Time: 2023-07-19T04:14:34Z Pulsar - Broker: Phase: Running Pods Ready: true Pods Ready Time: 2023-07-19T04:15:09Z Pulsar - Proxy: Phase: Running Pods Ready: true Pods Ready Time: 2023-07-19T04:15:47Z Zookeeper: Phase: Running Pods Ready: true Pods Ready Time: 2023-07-18T09:10:52Z Conditions: Last Transition Time: 2023-07-19T06:26:58Z Message: Restart opsRequest: pulsar-cluster-restart-kws8w has been processed Reason: Processed Status: True Type: LatestOpsRequestProcessed Last Transition Time: 2023-07-18T09:09:55Z Message: The operator has started the provisioning of Cluster: pulsar-cluster Observed Generation: 3 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-07-19T04:15:10Z Message: Successfully applied for resources Observed Generation: 3 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2023-07-19T04:14:23Z Message: pods are not ready in Components: [bookies], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2023-07-19T04:14:23Z Message: pods are unavailable in Components: [bookies], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 3 Phase: Failed Events: Type Reason Age From Message

Warning BackOff 48m (x23 over 147m) event-controller Pod pulsar-cluster-bookies-0: Back-off restarting failed container Normal Restart 16m ops-request-controller Start to process the Restart opsRequest "pulsar-cluster-restart-kws8w" in Cluster: pulsar-cluster Normal ComponentPhaseTransition 16m cluster-controller Running: false, PodsReady: false, PodsTimedout: true Warning Failed 16m (x2 over 148m) cluster-controller Cluster: pulsar-cluster is Failed, check according to the components message Normal Processed 16m cluster-controller Restart opsRequest: pulsar-cluster-restart-kws8w has been processed Warning BackOff 15m (x28 over 147m) event-controller Pod pulsar-cluster-bookies-2: Back-off restarting failed container Warning BackOff 3m34s (x27 over 143m) event-controller Pod pulsar-cluster-bookies-1: Back-off restarting failed container


logs pod

kubectl logs pulsar-cluster-bookies-0 Defaulted container "bookies" out of: bookies, metrics, init-bookies (init) [conf/bookkeeper.conf] Applying config autoRecoveryDaemonEnabled = true [conf/bookkeeper.conf] Applying config compactionRateByBytes = 52428800 [conf/bookkeeper.conf] Applying config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Applying config enableStatistics = true [conf/bookkeeper.conf] Applying config httpServerEnabled = true [conf/bookkeeper.conf] Applying config httpServerPort = 8000 [conf/bookkeeper.conf] Applying config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Applying config journalMaxBackups = 0 [conf/bookkeeper.conf] Applying config ledgerDirectories = /pulsar/data/bookkeeper/ledgers [conf/bookkeeper.conf] Applying config lostBookieRecoveryDelay = 300 [conf/bookkeeper.conf] Applying config statsProviderClass = org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider [conf/bookkeeper.conf] Applying config useHostNameAsBookieID = true [conf/bookkeeper.conf] Applying config zkLedgersRootPath = /ledgers [conf/bookkeeper.conf] Applying config zkServers = pulsar-cluster-zookeeper.default.svc:2181 [conf/bookkeeper.conf] Updating config autoRecoveryDaemonEnabled = true [conf/bookkeeper.conf] Updating config compactionRateByBytes = 52428800 [conf/bookkeeper.conf] Updating config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Adding config useTransactionalCompaction = true VM settings: Max. Heap Size (Estimated): 512.00M Using VM: OpenJDK 64-Bit Server VM

2023-07-19T06:40:07,634+0000 [main] INFO org.apache.bookkeeper.server.Main - Using configuration file /pulsar/conf/bookkeeper.conf 2023-07-19T06:40:07,644+0000 [main] INFO org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is , listening on port 3181. Metadata service uri is zk+null://pulsar-cluster-zookeeper.default.svc:2181/ledgers. Journals are in [/pulsar/data/bookkeeper/journal]. Ledgers are stored in [/pulsar/data/bookkeeper/ledgers]. 2023-07-19T06:40:07,655+0000 [main] INFO org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService 2023-07-19T06:40:07,726+0000 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2023-07-19T06:40:07,731+0000 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsar-cluster-zookeeper.default.svc:2181/ledgers : zkServers = pulsar-cluster-zookeeper.default.svc:2181, ledgersRootPath = /ledgers. 2023-07-19T06:40:07,736+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC 2023-07-19T06:40:07,736+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-cluster-bookies-0.pulsar-cluster-bookies-headless.default.svc.cluster.local 2023-07-19T06:40:07,736+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7 2023-07-19T06:40:07,737+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian 2023-07-19T06:40:07,737+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64 2023-07-19T06:40:07,737+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-2.11.1.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.48.v20220622.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-2.11.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.13.4.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.54.Final.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.15.4-tests.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/io.perfmark-perfmark-api-0.19.0.jar:/pulsar/lib/io.grpc-grpc-xds-1.45.1.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-2.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-2.11.1.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.15.4.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-2.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-2.11.1.jar:/pulsar/lib/com.google.auto.service-auto-service-annotations-1.0.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.54.Final-linux-x86_64.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.15.4.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.45.1.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.11.1.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.48.v20220622.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.8.4.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.4.32.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.48.v20220622.jar:/pulsar/lib/io.swagger-swagger-models-1.6.2.jar:/pulsar/lib/io.vertx-vertx-web-common-3.9.8.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-2.11.1.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-2.11.1.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.13.4.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/io.netty-netty-common-4.1.86.Final.jar:/pulsar/lib/io.grpc-grpc-context-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.15.4.jar:/pulsar/lib/io.grpc-grpc-rls-1.45.1.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.15.4.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.86.Final-linux-x86_64.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.11.1.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.15.4.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/io.netty-netty-codec-4.1.86.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-2.11.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.48.v20220622.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.15.4.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.15.4.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.54.Final.jar:/pulsar/lib/com.google.guava-guava-31.0.1-jre.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.48.v20220622.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.48.v20220622.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.13.4.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.apache.avro-avro-1.10.2.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.15.4.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-2.11.1.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.86.Final.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.45.1.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/io.etcd-jetcd-core-0.5.11.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-2.11.1.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.15.4.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/io.kubernetes-client-java-12.0.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/io.kubernetes-client-java-proto-12.0.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.48.v20220622.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/org.yaml-snakeyaml-1.32.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.54.Final-linux-aarch_64.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.45.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.54.Final-osx-x86_64.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-2.11.1.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-2.11.1.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-2.11.1.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.13.4.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-2.11.1.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.86.Final.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.9.8.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.13.4.jar:/pulsar/lib/io.grpc-grpc-services-1.45.1.jar:/pulsar/lib/com.squareup.okio-okio-2.8.0.jar:/pulsar/lib/io.grpc-grpc-stub-1.45.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.48.v20220622.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.13.4.jar:/pulsar/lib/io.grpc-grpc-all-1.45.1.jar:/pulsar/lib/io.netty-netty-resolver-4.1.86.Final.jar:/pulsar/lib/io.netty-netty-transport-4.1.86.Final.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.86.Final.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.4.32.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/io.vertx-vertx-web-3.9.8.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.48.v20220622.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/io.grpc-grpc-api-1.45.1.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.15.4.jar:/pulsar/lib/io.grpc-grpc-alts-1.45.1.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-2.11.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.15.4.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.15.4.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.48.v20220622.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.48.v20220622.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-2.11.1.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.86.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.48.v20220622.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.15.4.jar:/pulsar/lib/io.kubernetes-client-java-api-12.0.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.48.v20220622.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.48.v20220622.jar:/pulsar/lib/io.grpc-grpc-auth-1.45.1.jar:/pulsar/lib/io.etcd-jetcd-common-0.5.11.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.15.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-2.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-2.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-2.11.1.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-2.11.1.jar:/pulsar/lib/org.checkerframework-checker-qual-3.12.0.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/io.vertx-vertx-bridge-common-3.9.8.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.rocksdb-rocksdbjni-6.29.4.1.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.15.4.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-2.11.1.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.0.1.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.15.4.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.86.Final.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.48.v20220622.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.86.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.15.4.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.48.v20220622.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.15.4.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.54.Final-windows-x86_64.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/io.swagger-swagger-core-1.6.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-2.11.1.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.54.Final-osx-aarch_64.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.7.6.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.86.Final.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.15.4.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-2.11.1.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.86.Final.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.15.4.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.4.32.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-2.11.1-pkg.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-2.11.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-2.11.1.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.15.4.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.13.4.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-2.11.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.48.v20220622.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.15.4.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.1.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/io.vertx-vertx-core-3.9.8.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.86.Final.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.4.32.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-2.11.1.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.13.4.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.48.v20220622.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-common-2.11.1.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/io.grpc-grpc-core-1.45.1.jar:/pulsar/lib/io.netty-netty-buffer-4.1.86.Final.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.15.4.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/io.grpc-grpc-netty-1.45.1.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.netty-netty-handler-4.1.86.Final.jar:/pulsar/lib/com.google.re2j-re2j-1.5.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.1.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/org.apache.pulsar-testmocks-2.11.1.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.1.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.86.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.86.Final.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-2.11.1.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.48.v20220622.jar: 2023-07-19T06:40:07,738+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2023-07-19T06:40:07,738+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2023-07-19T06:40:07,738+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2023-07-19T06:40:07,738+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2023-07-19T06:40:07,740+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2023-07-19T06:40:07,740+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.19.91-27.1.al7.x86_64 2023-07-19T06:40:07,740+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2023-07-19T06:40:07,740+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2023-07-19T06:40:07,740+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar 2023-07-19T06:40:07,740+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=12MB 2023-07-19T06:40:07,741+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=512MB 2023-07-19T06:40:07,741+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=31MB 2023-07-19T06:40:07,744+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-cluster-zookeeper.default.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@1ce61929 2023-07-19T06:40:07,748+0000 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2023-07-19T06:40:07,753+0000 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes 2023-07-19T06:40:07,760+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2023-07-19T06:40:07,827+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181. 2023-07-19T06:40:07,827+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2023-07-19T06:40:07,833+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.0.0.219:57508, server: pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181 2023-07-19T06:40:07,838+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181, session id = 0x1000006ce97002c, negotiated timeout = 30000 2023-07-19T06:40:07,840+0000 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2023-07-19T06:40:08,221+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current] 2023-07-19T06:40:08,221+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:422) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.doMain(Main.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.main(Main.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4]

➜
➜
➜ kubectl logs pulsar-cluster-bookies-1 Defaulted container "bookies" out of: bookies, metrics, init-bookies (init) [conf/bookkeeper.conf] Applying config autoRecoveryDaemonEnabled = true [conf/bookkeeper.conf] Applying config compactionRateByBytes = 52428800 [conf/bookkeeper.conf] Applying config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Applying config enableStatistics = true [conf/bookkeeper.conf] Applying config httpServerEnabled = true [conf/bookkeeper.conf] Applying config httpServerPort = 8000 [conf/bookkeeper.conf] Applying config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Applying config journalMaxBackups = 0 [conf/bookkeeper.conf] Applying config ledgerDirectories = /pulsar/data/bookkeeper/ledgers [conf/bookkeeper.conf] Applying config lostBookieRecoveryDelay = 300 [conf/bookkeeper.conf] Applying config statsProviderClass = org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider [conf/bookkeeper.conf] Applying config useHostNameAsBookieID = true [conf/bookkeeper.conf] Applying config zkLedgersRootPath = /ledgers [conf/bookkeeper.conf] Applying config zkServers = pulsar-cluster-zookeeper.default.svc:2181 [conf/bookkeeper.conf] Updating config autoRecoveryDaemonEnabled = true [conf/bookkeeper.conf] Updating config compactionRateByBytes = 52428800 [conf/bookkeeper.conf] Updating config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Adding config useTransactionalCompaction = true VM settings: Max. Heap Size (Estimated): 512.00M Using VM: OpenJDK 64-Bit Server VM

2023-07-19T06:39:52,754+0000 [main] INFO org.apache.bookkeeper.server.Main - Using configuration file /pulsar/conf/bookkeeper.conf 2023-07-19T06:39:52,820+0000 [main] INFO org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is , listening on port 3181. Metadata service uri is zk+null://pulsar-cluster-zookeeper.default.svc:2181/ledgers. Journals are in [/pulsar/data/bookkeeper/journal]. Ledgers are stored in [/pulsar/data/bookkeeper/ledgers]. 2023-07-19T06:39:52,829+0000 [main] INFO org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService 2023-07-19T06:39:52,841+0000 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2023-07-19T06:39:52,844+0000 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsar-cluster-zookeeper.default.svc:2181/ledgers : zkServers = pulsar-cluster-zookeeper.default.svc:2181, ledgersRootPath = /ledgers. 2023-07-19T06:39:52,848+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC 2023-07-19T06:39:52,848+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-cluster-bookies-1.pulsar-cluster-bookies-headless.default.svc.cluster.local 2023-07-19T06:39:52,848+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7 2023-07-19T06:39:52,848+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian 2023-07-19T06:39:52,849+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64 2023-07-19T06:39:52,849+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.grpc-grpc-grpclb-... 2023-07-19T06:39:52,849+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.19.91-27.1.al7.x86_64 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2023-07-19T06:39:52,850+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2023-07-19T06:39:52,851+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar 2023-07-19T06:39:52,851+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=8MB 2023-07-19T06:39:52,851+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=512MB 2023-07-19T06:39:52,851+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=36MB 2023-07-19T06:39:52,854+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-cluster-zookeeper.default.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@1ce61929 2023-07-19T06:39:52,913+0000 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2023-07-19T06:39:52,917+0000 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes 2023-07-19T06:39:52,924+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2023-07-19T06:39:52,940+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181. 2023-07-19T06:39:52,941+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2023-07-19T06:39:52,946+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.0.0.224:60248, server: pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181 2023-07-19T06:39:52,951+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181, session id = 0x1000006ce97002b, negotiated timeout = 30000 2023-07-19T06:39:53,014+0000 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2023-07-19T06:39:53,316+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current] 2023-07-19T06:39:53,316+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:422) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.doMain(Main.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.main(Main.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4]

➜
➜
➜ kubectl logs pulsar-cluster-bookies-2 Defaulted container "bookies" out of: bookies, metrics, init-bookies (init) [conf/bookkeeper.conf] Applying config autoRecoveryDaemonEnabled = true [conf/bookkeeper.conf] Applying config compactionRateByBytes = 52428800 [conf/bookkeeper.conf] Applying config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Applying config enableStatistics = true [conf/bookkeeper.conf] Applying config httpServerEnabled = true [conf/bookkeeper.conf] Applying config httpServerPort = 8000 [conf/bookkeeper.conf] Applying config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Applying config journalMaxBackups = 0 [conf/bookkeeper.conf] Applying config ledgerDirectories = /pulsar/data/bookkeeper/ledgers [conf/bookkeeper.conf] Applying config lostBookieRecoveryDelay = 300 [conf/bookkeeper.conf] Applying config statsProviderClass = org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider [conf/bookkeeper.conf] Applying config useHostNameAsBookieID = true [conf/bookkeeper.conf] Applying config zkLedgersRootPath = /ledgers [conf/bookkeeper.conf] Applying config zkServers = pulsar-cluster-zookeeper.default.svc:2181 [conf/bookkeeper.conf] Updating config autoRecoveryDaemonEnabled = true [conf/bookkeeper.conf] Updating config compactionRateByBytes = 52428800 [conf/bookkeeper.conf] Updating config journalDirectories = /pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Adding config useTransactionalCompaction = true VM settings: Max. Heap Size (Estimated): 512.00M Using VM: OpenJDK 64-Bit Server VM

2023-07-19T06:43:11,206+0000 [main] INFO org.apache.bookkeeper.server.Main - Using configuration file /pulsar/conf/bookkeeper.conf 2023-07-19T06:43:11,272+0000 [main] INFO org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is , listening on port 3181. Metadata service uri is zk+null://pulsar-cluster-zookeeper.default.svc:2181/ledgers. Journals are in [/pulsar/data/bookkeeper/journal]. Ledgers are stored in [/pulsar/data/bookkeeper/ledgers]. 2023-07-19T06:43:11,283+0000 [main] INFO org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService 2023-07-19T06:43:11,295+0000 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2023-07-19T06:43:11,300+0000 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsar-cluster-zookeeper.default.svc:2181/ledgers : zkServers = pulsar-cluster-zookeeper.default.svc:2181, ledgersRootPath = /ledgers. 2023-07-19T06:43:11,304+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC 2023-07-19T06:43:11,304+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-cluster-bookies-2.pulsar-cluster-bookies-headless.default.svc.cluster.local 2023-07-19T06:43:11,304+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7 2023-07-19T06:43:11,304+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian 2023-07-19T06:43:11,304+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64 2023-07-19T06:43:11,304+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.grpc-grpc-grpclb-... 2023-07-19T06:43:11,305+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2023-07-19T06:43:11,305+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2023-07-19T06:43:11,305+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.19.91-27.1.al7.x86_64 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=8MB 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=512MB 2023-07-19T06:43:11,306+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=30MB 2023-07-19T06:43:11,310+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-cluster-zookeeper.default.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@1ce61929 2023-07-19T06:43:11,313+0000 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2023-07-19T06:43:11,371+0000 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes 2023-07-19T06:43:11,378+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2023-07-19T06:43:11,387+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181. 2023-07-19T06:43:11,387+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2023-07-19T06:43:11,392+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.0.0.14:55352, server: pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181 2023-07-19T06:43:11,397+0000 [main-SendThread(pulsar-cluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-cluster-zookeeper.default.svc/192.168.93.187:2181, session id = 0x1000006ce97002d, negotiated timeout = 30000 2023-07-19T06:43:11,400+0000 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2023-07-19T06:43:11,784+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current] 2023-07-19T06:43:11,785+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:422) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.doMain(Main.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.main(Main.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4]



**Expected behavior**
pulsar cluster status ok.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
JashBook commented 1 year ago

pvc and pv is Bound,but ack volume is Unattached

kubectl get pvc,pv
NAME                                                           STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/data-pulsar-cluster-bookies-0            Bound    d-8vb3bicldxhsc9btszat   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-bookies-1            Bound    d-8vb1woicfilu99qq6ib1   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-bookies-2            Bound    d-8vb6z5a61mpcskxttd9w   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-bookies-recovery-0   Bound    d-8vbdbtsinp5iw3cym207   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-pulsar-broker-0      Bound    d-8vb0ntztdjer43yusfaw   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-pulsar-proxy-0       Bound    d-8vb75dgg1vuatsq3do1n   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-pulsar-proxy-1       Bound    d-8vbdwec3vst734mro9xi   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-pulsar-proxy-2       Bound    d-8vbdwec3vst734mro9xj   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-zookeeper-0          Bound    d-8vb7wngs19qmjvjrjn9s   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-zookeeper-1          Bound    d-8vb8cade9q5f550f3yql   20Gi       RWO            kb-default-sc   22h
persistentvolumeclaim/data-pulsar-cluster-zookeeper-2          Bound    d-8vbavo4gy7teo7dn180r   20Gi       RWO            kb-default-sc   22h

NAME                                      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                            STORAGECLASS    REASON   AGE
persistentvolume/d-8vb0ntztdjer43yusfaw   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-pulsar-broker-0      kb-default-sc            22h
persistentvolume/d-8vb1woicfilu99qq6ib1   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-bookies-1            kb-default-sc            22h
persistentvolume/d-8vb3bicldxhsc9btszat   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-bookies-0            kb-default-sc            22h
persistentvolume/d-8vb6z5a61mpcskxttd9w   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-bookies-2            kb-default-sc            22h
persistentvolume/d-8vb75dgg1vuatsq3do1n   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-pulsar-proxy-0       kb-default-sc            22h
persistentvolume/d-8vb7wngs19qmjvjrjn9s   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-zookeeper-0          kb-default-sc            22h
persistentvolume/d-8vb8cade9q5f550f3yql   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-zookeeper-1          kb-default-sc            22h
persistentvolume/d-8vbavo4gy7teo7dn180r   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-zookeeper-2          kb-default-sc            22h
persistentvolume/d-8vbdbtsinp5iw3cym207   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-bookies-recovery-0   kb-default-sc            22h
persistentvolume/d-8vbdwec3vst734mro9xi   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-pulsar-proxy-1       kb-default-sc            22h
persistentvolume/d-8vbdwec3vst734mro9xj   20Gi       RWO            Delete           Bound    default/data-pulsar-cluster-pulsar-proxy-2       kb-default-sc            22h

image

github-actions[bot] commented 1 year ago

This issue has been marked as stale because it has been open for 30 days with no activity