[BUG]Pulsar bookies pod crash after vscale again

kbcli version Kubernetes: v1.27.2-gke.1200 KubeBlocks: 0.6.0-beta.29 kbcli: 0.6.0-beta.29

create cluster


apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: pulsarcluster
namespace: default
spec:
clusterDefinitionRef: pulsar
clusterVersionRef: pulsar-2.11.2
terminationPolicy: WipeOut
affinity:
topologyKeys:
  - kubernetes.io/hostname
componentSpecs:
- name: pulsar-broker
  componentDefRef: pulsar-broker
  monitor: false
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi
  volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
- name: bookies
  componentDefRef: bookies
  monitor: false
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi
  volumeClaimTemplates:
    - name: ledgers
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    - name: journal
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
- name: zookeeper
  componentDefRef: zookeeper
  monitor: false
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi
  volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
- name: pulsar-proxy
  componentDefRef: pulsar-proxy
  monitor: false
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi

  `kubectl apply -f test_create_pulsarcluster.yaml`

cluster.apps.kubeblocks.io/pulsarcluster created

2. Hscale and 1st vscale success

  `kbcli cluster hscale pulsarcluster --auto-approve --components bookies --replicas 5 --namespace default `

OpsRequest pulsarcluster-horizontalscaling-sbkw5 created successfully, you can view the progress: kbcli cluster describe-ops pulsarcluster-horizontalscaling-sbkw5 -n default

kbcli cluster vscale pulsarcluster --auto-approve --components pulsar-broker --cpu 200m --memory 0.8Gi --namespace default

OpsRequest pulsarcluster-verticalscaling-cxc87 created successfully, you can view the progress: kbcli cluster describe-ops pulsarcluster-verticalscaling-cxc87 -n default

3. Vscale again, bookies pod crash

kbcli cluster vscale pulsarcluster --auto-approve --components zookeeper --cpu 200m --memory 0.8Gi --namespace default

OpsRequest pulsarcluster-verticalscaling-8wxgh created successfully, you can view the progress: kbcli cluster describe-ops pulsarcluster-verticalscaling-8wxgh -n default

➜ ~ k get pod | grep pulsar pulsarcluster-bookies-0 2/2 Running 1 (5m17s ago) 14m pulsarcluster-bookies-1 2/2 Running 0 14m pulsarcluster-bookies-2 2/2 Running 1 (5m15s ago) 14m pulsarcluster-bookies-3 2/2 Running 0 12m pulsarcluster-bookies-4 1/2 CrashLoopBackOff 6 (37s ago) 12m pulsarcluster-pulsar-broker-0 3/3 Running 0 10m pulsarcluster-pulsar-proxy-0 2/2 Running 0 14m pulsarcluster-zookeeper-0 2/2 Running 0 5m49s pulsarcluster-zookeeper-1 2/2 Running 0 5m53s pulsarcluster-zookeeper-2 2/2 Running 0 7m

➜ ~ k describe pod pulsarcluster-bookies-4 Name: pulsarcluster-bookies-4 Namespace: default Priority: 0 Node: gke-yjtest-default-pool-8e798dc1-4z3z/10.128.15.226 Start Time: Tue, 08 Aug 2023 12:02:56 +0800 Labels: app.kubernetes.io/component=bookies app.kubernetes.io/instance=pulsarcluster app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=pulsar app.kubernetes.io/version=pulsar-2.11.2 apps.kubeblocks.io/component-name=bookies apps.kubeblocks.io/workload-type=Stateful controller-revision-hash=pulsarcluster-bookies-6564cdb7f5 statefulset.kubernetes.io/pod-name=pulsarcluster-bookies-4 Annotations: apps.kubeblocks.io/component-replicas: 5 config.kubeblocks.io/restart-bookies-config: 78cff7697b kubeblocks.io/restart: 2023-08-08T03:54:48Z Status: Running IP: 10.104.0.73 IPs: IP: 10.104.0.73 Controlled By: StatefulSet/pulsarcluster-bookies Init Containers: init-bookies: Container ID: containerd://8d9446efa610c86e425b600c12cd3d83c4f94134088ba3049786510627651d42 Image: docker.io/apecloud/pulsar:2.11.2 Image ID: docker.io/apecloud/pulsar@sha256:b50c2a1c83e6dd91287d5ec47da61544d104df546cd802b90ba8a01e55786792 Port: Host Port: Command: /kb-scripts/init-bookies.sh State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 08 Aug 2023 12:03:09 +0800 Finished: Tue, 08 Aug 2023 12:03:14 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: pulsarcluster-bookies-env ConfigMap Optional: false pulsarcluster-bookies-bookies-env-envfrom ConfigMap Optional: false Environment: KB_POD_NAME: pulsarcluster-bookies-4 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: pulsarcluster KB_COMP_NAME: bookies KB_CLUSTER_COMP_NAME: pulsarcluster-bookies KB_CLUSTER_UID_POSTFIX_8: 7400c864 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc Mounts: /kb-scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7nkmw (ro) Containers: bookies: Container ID: containerd://02a9221cae6f19acfcfd6189a86c24ff6c97bed6ea335ddc8d488bb07aa9d595 Image: docker.io/apecloud/pulsar:2.11.2 Image ID: docker.io/apecloud/pulsar@sha256:b50c2a1c83e6dd91287d5ec47da61544d104df546cd802b90ba8a01e55786792 Ports: 8000/TCP, 3181/TCP Host Ports: 0/TCP, 0/TCP Command: /kb-scripts/start-bookies.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Tue, 08 Aug 2023 12:14:32 +0800 Finished: Tue, 08 Aug 2023 12:14:37 +0800 Ready: False Restart Count: 6 Requests: cpu: 100m memory: 512Mi Liveness: http-get http://:http/api/v1/bookie/state delay=10s timeout=5s period=30s #success=1 #failure=60 Readiness: http-get http://:http/api/v1/bookie/is_ready delay=10s timeout=5s period=30s #success=1 #failure=60 Environment Variables from: pulsarcluster-bookies-env ConfigMap Optional: false pulsarcluster-bookies-bookies-env-envfrom ConfigMap Optional: false Environment: KB_POD_NAME: pulsarcluster-bookies-4 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: pulsarcluster KB_COMP_NAME: bookies KB_CLUSTER_COMP_NAME: pulsarcluster-bookies KB_CLUSTER_UID_POSTFIX_8: 7400c864 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc cluster_domain: .cluster.local Mounts: /etc/annotations from annotations (rw) /kb-scripts from scripts (rw) /opt/pulsar/conf from pulsar-bookies-config (rw) /opt/pulsar/none from bookies-env (rw) /pulsar/data/bookkeeper/journal from journal (rw) /pulsar/data/bookkeeper/ledgers from ledgers (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7nkmw (ro) metrics: Container ID: containerd://9ee2ddab702ff7ab0d4bdf1914e0c1375f779b666052c730cf2a4ee8ef3b4c0e Image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:a85d7d80d33de188ab39880d8d6013b663ebade9186371ce09a06c29dfa443ac Port: 1234/TCP Host Port: 0/TCP Command: /bin/agamotto --config=/opt/agamotto/agamotto-config.yaml --feature-gates=-pkg.translator.prometheus.NormalizeName State: Running Started: Tue, 08 Aug 2023 12:03:15 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: pulsarcluster-bookies-env ConfigMap Optional: false Environment: KB_POD_NAME: pulsarcluster-bookies-4 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: pulsarcluster KB_COMP_NAME: bookies KB_CLUSTER_COMP_NAME: pulsarcluster-bookies KB_CLUSTER_UID_POSTFIX_8: 7400c864 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc JOB_NAME: bookie SERVICE_PORT: 8000 POD_NAME: $(KB_POD_NAME) POD_NAMESPACE: $(KB_NAMESPACE) LOG_LEVEL: info Mounts: /opt/agamotto from agamotto-configuration (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7nkmw (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: ledgers: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: ledgers-pulsarcluster-bookies-4 ReadOnly: false journal: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: journal-pulsarcluster-bookies-4 ReadOnly: false annotations: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas agamotto-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsarcluster-bookies-agamotto-configuration Optional: false pulsar-bookies-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsarcluster-bookies-bookies-config Optional: false bookies-env: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsarcluster-bookies-bookies-env Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: pulsarcluster-bookies-pulsar-scripts Optional: false kube-api-access-7nkmw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 12m default-scheduler Successfully assigned default/pulsarcluster-bookies-4 to gke-yjtest-default-pool-8e798dc1-4z3z Normal SuccessfulAttachVolume 12m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-f41767d6-57d9-44cf-9bad-e68c1cabb96c" Normal SuccessfulAttachVolume 12m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c70d7c9d-eb55-4034-9ab7-c102d29fd375" Normal Pulled 12m kubelet Container image "docker.io/apecloud/pulsar:2.11.2" already present on machine Normal Created 12m kubelet Created container init-bookies Normal Started 12m kubelet Started container init-bookies Normal Pulled 12m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2" already present on machine Normal Started 12m kubelet Started container metrics Normal Created 12m kubelet Created container metrics Normal Pulled 6m33s (x4 over 12m) kubelet Container image "docker.io/apecloud/pulsar:2.11.2" already present on machine Normal Started 6m33s (x4 over 12m) kubelet Started container bookies Normal Created 6m33s (x4 over 12m) kubelet Created container bookies Warning BackOff 2m25s (x23 over 7m18s) kubelet Back-off restarting failed container bookies in pod pulsarcluster-bookies-4_default(49c57cf4-46c5-4aea-8c0f-d8b30f383ed3)

➜ ~ k logs pulsarcluster-bookies-4 Defaulted container "bookies" out of: bookies, metrics, init-bookies (init)

mkdir -p /current
mkdir -p /current ++ ls -A /current
journalRes= ++ ls -A /current
ledgerRes=
[[ -z '' ]]
[[ -z '' ]]
host_ip_port=pulsarcluster-bookies-4.pulsarcluster-bookies-headless.default.svc.cluster.local:3181
zNode=/cookies/pulsarcluster-bookies-4.pulsarcluster-bookies-headless.default.svc.cluster.local:3181
zkURL=pulsarcluster-zookeeper.default.svc:2181
python3 /kb-scripts/zookeeper.py get /cookies/pulsarcluster-bookies-4.pulsarcluster-bookies-headless.default.svc.cluster.local:3181 Tue, 08 Aug 2023 04:14:32 connection.py[line:624] INFO Connecting to pulsarcluster-zookeeper.default.svc(10.108.15.111):2181, use_ssl: False Tue, 08 Aug 2023 04:14:32 client.py[line:532] INFO Zookeeper connection established, state: CONNECTED Tue, 08 Aug 2023 04:14:32 zookeeper.py[line:59] ERROR Node /cookies/pulsarcluster-bookies-4.pulsarcluster-bookies-headless.default.svc.cluster.local:3181 does not exist. Tue, 08 Aug 2023 04:14:32 connection.py[line:591] INFO Closing connection to pulsarcluster-zookeeper.default.svc:2181 Tue, 08 Aug 2023 04:14:32 client.py[line:537] INFO Zookeeper session closed, state: CLOSED
python3 /kb-scripts/merge_pulsar_config.py conf/bookkeeper.conf /opt/pulsar/conf/bookkeeper.conf [conf/bookkeeper.conf] Adding config: journalDirectories=/pulsar/data/bookkeeper/journal [conf/bookkeeper.conf] Updating config compactionRateByBytes=52428800 [conf/bookkeeper.conf] Adding config: useTransactionalCompaction=true [conf/bookkeeper.conf] Updating config dbStorage_readAheadCacheBatchSize=100 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_blockSize=65536 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_writeBufferSizeMB=64 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_sstSizeInMB=64 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_bloomFilterBitsPerKey=10 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_numLevels=-1 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_numFilesInLevel0=10 [conf/bookkeeper.conf] Updating config dbStorage_rocksDB_maxSizeInLevel1MB=256 [conf/bookkeeper.conf] Updating config diskUsageThreshold=0.95 [conf/bookkeeper.conf] Adding config: diskUsageWarnThreshold=0.95 [conf/bookkeeper.conf] Adding config: diskUsageLwmThreshold=0.9 [conf/bookkeeper.conf] Updating config diskCheckInterval=10000 [conf/bookkeeper.conf] Updating config journalFormatVersionToWrite=5 [conf/bookkeeper.conf] Updating config journalMaxSizeMB=2048 [conf/bookkeeper.conf] Updating config journalMaxBackups=5 [conf/bookkeeper.conf] Updating config journalPreAllocSizeMB=16 [conf/bookkeeper.conf] Updating config journalWriteBufferSizeKB=64 [conf/bookkeeper.conf] Updating config journalRemoveFromPageCache=true [conf/bookkeeper.conf] Updating config journalSyncData=true [conf/bookkeeper.conf] Updating config journalAdaptiveGroupWrites=true [conf/bookkeeper.conf] Updating config journalMaxGroupWaitMSec=2 [conf/bookkeeper.conf] Updating config journalBufferedWritesThreshold=524288 [conf/bookkeeper.conf] Updating config journalFlushWhenQueueEmpty=false [conf/bookkeeper.conf] Updating config journalAlignmentSize=512 [conf/bookkeeper.conf] Adding config: journalBufferedEntriesThreshold=0 [conf/bookkeeper.conf] Adding config: journalQueueSize=10000 [conf/bookkeeper.conf] Updating config lostBookieRecoveryDelay=300 [conf/bookkeeper.conf] Updating config httpServerEnabled=true [conf/bookkeeper.conf] Updating config httpServerPort=8000 [conf/bookkeeper.conf] Updating config ledgerDirectories=/pulsar/data/bookkeeper/ledgers [conf/bookkeeper.conf] Updating config statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider [conf/bookkeeper.conf] Adding config: enableStatistics=true [conf/bookkeeper.conf] Updating config useHostNameAsBookieID=true [conf/bookkeeper.conf] Updating config zkLedgersRootPath=/ledgers [conf/bookkeeper.conf] Updating config autoRecoveryDaemonEnabled=true
bin/apply-config-from-env.py conf/bookkeeper.conf [conf/bookkeeper.conf] Applying config zkServers = pulsarcluster-zookeeper.default.svc:2181
OPTS=' -Dlog4j2.formatMsgNoLookups=true'
exec bin/pulsar bookie VM settings: Max. Heap Size (Estimated): 3.88G Using VM: OpenJDK 64-Bit Server VM

2023-08-08T04:14:35,760+0000 [main] INFO org.apache.bookkeeper.server.Main - Using configuration file /pulsar/conf/bookkeeper.conf 2023-08-08T04:14:35,782+0000 [main] INFO org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is , listening on port 3181. Metadata service uri is zk+null://pulsarcluster-zookeeper.default.svc:2181/ledgers. Journals are in [/pulsar/data/bookkeeper/journal]. Ledgers are stored in [/pulsar/data/bookkeeper/ledgers]. 2023-08-08T04:14:35,804+0000 [main] INFO org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService 2023-08-08T04:14:35,832+0000 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2023-08-08T04:14:35,842+0000 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsarcluster-zookeeper.default.svc:2181/ledgers : zkServers = pulsarcluster-zookeeper.default.svc:2181, ledgersRootPath = /ledgers. 2023-08-08T04:14:35,859+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC 2023-08-08T04:14:35,860+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsarcluster-bookies-4.pulsarcluster-bookies-headless.default.svc.cluster.local 2023-08-08T04:14:35,861+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7 2023-08-08T04:14:35,862+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian 2023-08-08T04:14:35,863+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64 2023-08-08T04:14:35,864+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.15.4.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.51.v20230217.jar:/pulsar/lib/io.grpc-grpc-api-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-2.11.2.jar:/pulsar/lib/io.grpc-grpc-alts-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-2.11.2.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-2.11.2.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.45.1.jar:/pulsar/lib/io.netty-netty-common-4.1.93.Final.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/io.netty-netty-codec-4.1.93.Final.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.20.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.15.4.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.15.4.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.93.Final.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.15.4.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/io.grpc-grpc-rls-1.45.1.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.15.4.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-2.11.2-pkg.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-2.11.2.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/org.rocksdb-rocksdbjni-6.29.4.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.51.v20230217.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.15.4.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-2.11.2.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-2.11.2.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.51.v20230217.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.11.2.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.15.4.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.45.1.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.45.1.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.9.8.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/org.apache.pulsar-testmocks-2.11.2.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.51.v20230217.jar:/pulsar/lib/io.netty-netty-buffer-4.1.93.Final.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.51.v20230217.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.51.v20230217.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.15.4.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.15.4.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/io.vertx-vertx-core-3.9.8.jar:/pulsar/lib/io.grpc-grpc-xds-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-2.11.2.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/com.google.auto.service-auto-service-annotations-1.0.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.20.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.93.Final.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.51.v20230217.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/io.vertx-vertx-web-common-3.9.8.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.51.v20230217.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.45.1.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/io.grpc-grpc-auth-1.45.1.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-2.11.2.jar:/pulsar/lib/io.grpc-grpc-core-1.45.1.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/org.apache.avro-avro-1.10.2.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.93.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-4.1.93.Final.jar:/pulsar/lib/io.grpc-grpc-netty-1.45.1.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-2.11.2.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.1.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.93.Final.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-2.11.2.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.15.4.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.51.v20230217.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-2.11.2.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.8.20.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-2.11.2.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.8.20.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/io.grpc-grpc-all-1.45.1.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.11.2.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.15.4.jar:/pulsar/lib/io.netty-netty-resolver-4.1.93.Final.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.93.Final.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.15.4.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.1.jar:/pulsar/lib/io.etcd-jetcd-common-0.5.11.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.93.Final.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.15.4.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-2.11.2.jar:/pulsar/lib/io.grpc-grpc-context-1.45.1.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.51.v20230217.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-2.11.2.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.15.4.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.93.Final.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.51.v20230217.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.1.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.15.4.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.93.Final-linux-x86_64.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-2.11.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.51.v20230217.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.93.Final.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.51.v20230217.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-2.11.2.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-2.11.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/com.squareup.okio-okio-2.8.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-2.11.2.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-2.11.2.jar:/pulsar/lib/io.perfmark-perfmark-api-0.19.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.15.4.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.93.Final.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-common-2.11.2.jar:/pulsar/lib/io.vertx-vertx-web-3.9.8.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.15.4.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-2.11.2.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.51.v20230217.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-2.11.2.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.15.4.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.93.Final.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.15.4.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.51.v20230217.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.15.4.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-2.11.2.jar:/pulsar/lib/io.grpc-grpc-services-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-2.11.2.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/io.grpc-grpc-stub-1.45.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-2.11.2.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.51.v20230217.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.15.4.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.15.4.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/io.etcd-jetcd-core-0.5.11.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-2.11.2.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.15.4-tests.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/io.vertx-vertx-bridge-common-3.9.8.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.11.2.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.15.4.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-2.11.2.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/com.google.re2j-re2j-1.5.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.1.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-2.11.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.51.v20230217.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/io.netty-netty-handler-4.1.93.Final.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.0.1.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar: 2023-08-08T04:14:35,867+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2023-08-08T04:14:35,868+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2023-08-08T04:14:35,869+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2023-08-08T04:14:35,870+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2023-08-08T04:14:35,870+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2023-08-08T04:14:35,871+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.109+ 2023-08-08T04:14:35,872+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2023-08-08T04:14:35,873+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2023-08-08T04:14:35,873+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar 2023-08-08T04:14:35,874+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=109MB 2023-08-08T04:14:35,875+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=3978MB 2023-08-08T04:14:35,876+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=130MB 2023-08-08T04:14:35,884+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsarcluster-zookeeper.default.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@28276e50 2023-08-08T04:14:35,895+0000 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2023-08-08T04:14:35,909+0000 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes 2023-08-08T04:14:35,927+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2023-08-08T04:14:36,050+0000 [main-SendThread(pulsarcluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsarcluster-zookeeper.default.svc/10.108.15.111:2181. 2023-08-08T04:14:36,051+0000 [main-SendThread(pulsarcluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2023-08-08T04:14:36,087+0000 [main-SendThread(pulsarcluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.104.0.73:47148, server: pulsarcluster-zookeeper.default.svc/10.108.15.111:2181 2023-08-08T04:14:36,098+0000 [main-SendThread(pulsarcluster-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsarcluster-zookeeper.default.svc/10.108.15.111:2181, session id = 0x10029740f200008, negotiated timeout = 30000 2023-08-08T04:14:36,103+0000 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2023-08-08T04:14:36,835+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - Cookie for this bookie is not stored in metadata store. Bookie failing to come up 2023-08-08T04:14:36,836+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:122) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:422) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.doMain(Main.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.main(Main.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4]

apecloud / kubeblocks

[BUG]Pulsar bookies pod crash after vscale again #4684