search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.21k stars 184 forks source link

[BUG]bookies hsacle 1-2 crash on lvm #4871

Closed ahjing99 closed 1 year ago

ahjing99 commented 1 year ago
  1. Create pulsar with lvm
  2. hscale 3-1 success 
    
kbcli cluster hscale pulsar --auto-approve --components bookies --replicas 1 --namespace default
``
  3. Hscale 1-2 failed 
    
kbcli cluster hscale pulsar --auto-approve --components bookies --replicas 2 --namespace default

[root@ip-172-31-25-255 ~]# k logs pulsar-bookies-1 bookies

2023-08-25T03:55:34,137+0000 [main] INFO org.apache.bookkeeper.server.Main - Using configuration file /pulsar/conf/bookkeeper.conf 2023-08-25T03:55:34,150+0000 [main] INFO org.apache.bookkeeper.server.Main - Hello, I'm your bookie, bookieId is , listening on port 3181. Metadata service uri is zk+null://pulsar-zookeeper.default.svc:2181/ledgers. Journals are in [/pulsar/data/bookkeeper/journal]. Ledgers are stored in [/pulsar/data/bookkeeper/ledgers]. 2023-08-25T03:55:34,165+0000 [main] INFO org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService 2023-08-25T03:55:34,186+0000 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2023-08-25T03:55:34,192+0000 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsar-zookeeper.default.svc:2181/ledgers : zkServers = pulsar-zookeeper.default.svc:2181, ledgersRootPath = /ledgers. 2023-08-25T03:55:34,198+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC 2023-08-25T03:55:34,198+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-bookies-1.pulsar-bookies-headless.default.svc.cluster.local 2023-08-25T03:55:34,198+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7 2023-08-25T03:55:34,198+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian 2023-08-25T03:55:34,198+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-arm64 2023-08-25T03:55:34,198+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.0.1.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/com.google.auto.service-auto-service-annotations-1.0.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/com.google.re2j-re2j-1.5.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/com.squareup.okio-okio-2.8.0.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/io.etcd-jetcd-common-0.5.11.jar:/pulsar/lib/io.etcd-jetcd-core-0.5.11.jar:/pulsar/lib/io.grpc-grpc-all-1.45.1.jar:/pulsar/lib/io.grpc-grpc-alts-1.45.1.jar:/pulsar/lib/io.grpc-grpc-api-1.45.1.jar:/pulsar/lib/io.grpc-grpc-auth-1.45.1.jar:/pulsar/lib/io.grpc-grpc-context-1.45.1.jar:/pulsar/lib/io.grpc-grpc-core-1.45.1.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.45.1.jar:/pulsar/lib/io.grpc-grpc-netty-1.45.1.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.45.1.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.45.1.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.45.1.jar:/pulsar/lib/io.grpc-grpc-rls-1.45.1.jar:/pulsar/lib/io.grpc-grpc-services-1.45.1.jar:/pulsar/lib/io.grpc-grpc-stub-1.45.1.jar:/pulsar/lib/io.grpc-grpc-xds-1.45.1.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/io.netty-netty-buffer-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-common-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-handler-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-resolver-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-transport-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.93.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.93.Final.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.93.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.93.Final.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/io.perfmark-perfmark-api-0.19.0.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/io.vertx-vertx-auth-common-3.9.8.jar:/pulsar/lib/io.vertx-vertx-bridge-common-3.9.8.jar:/pulsar/lib/io.vertx-vertx-core-3.9.8.jar:/pulsar/lib/io.vertx-vertx-web-3.9.8.jar:/pulsar/lib/io.vertx-vertx-web-common-3.9.8.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/net.java.dev.jna-jna-4.2.0.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/org.apache.avro-avro-1.10.2.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.10.2.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.15.4.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.15.4.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.15.4.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.15.4-tests.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.15.4.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.15.4.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-2.11.2-pkg.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-common-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-2.11.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-2.11.2.jar:/pulsar/lib/org.apache.pulsar-testmocks-2.11.2.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.1.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.1.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.51.v20230217.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.51.v20230217.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.8.20.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.8.20.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.20.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.20.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/org.rocksdb-rocksdbjni-6.29.4.1.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.1.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar: 2023-08-25T03:55:34,199+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/aarch64-linux-gnu/jni:/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=aarch64 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=6.1.41-63.114.amzn2023.aarch64 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar 2023-08-25T03:55:34,200+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=184MB 2023-08-25T03:55:34,201+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=7892MB 2023-08-25T03:55:34,201+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=256MB 2023-08-25T03:55:34,206+0000 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-zookeeper.default.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@28276e50 2023-08-25T03:55:34,212+0000 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2023-08-25T03:55:34,219+0000 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes 2023-08-25T03:55:34,228+0000 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2023-08-25T03:55:34,307+0000 [main-SendThread(pulsar-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-zookeeper.default.svc/10.96.3.222:2181. 2023-08-25T03:55:34,308+0000 [main-SendThread(pulsar-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2023-08-25T03:55:34,321+0000 [main-SendThread(pulsar-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /100.86.10.74:41446, server: pulsar-zookeeper.default.svc/10.96.3.222:2181 2023-08-25T03:55:34,329+0000 [main-SendThread(pulsar-zookeeper.default.svc:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-zookeeper.default.svc/10.96.3.222:2181, session id = 0x100005be9b90019, negotiated timeout = 30000 2023-08-25T03:55:34,333+0000 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2023-08-25T03:55:34,522+0000 [main] ERROR org.apache.bookkeeper.bookie.LegacyCookieValidation - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/pulsar/data/bookkeeper/journal/current, /pulsar/data/bookkeeper/ledgers/current] 2023-08-25T03:55:34,523+0000 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:116) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:422) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.doMain(Main.java:272) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4] at org.apache.bookkeeper.server.Main.main(Main.java:255) ~[org.apache.bookkeeper-bookkeeper-server-4.15.4.jar:4.15.4]

k get pod NAME READY STATUS RESTARTS AGE pulsar-bookies-0 2/2 Running 0 17m pulsar-bookies-1 1/2 CrashLoopBackOff 4 (78s ago) 4m3s pulsar-broker-0 3/3 Running 0 20m pulsar-broker-1 3/3 Running 0 6m44s pulsar-broker-2 3/3 Running 0 5m56s pulsar-zookeeper-0 2/2 Running 0 20m pulsar-zookeeper-1 2/2 Running 0 20m pulsar-zookeeper-2 2/2 Running 0 20m

[root@ip-172-31-25-255 ~]# k describe cluster pulsar Name: pulsar Namespace: default Labels: app.kubernetes.io/instance=pulsar clusterdefinition.kubeblocks.io/name=pulsar clusterversion.kubeblocks.io/name=pulsar-2.11.2 Annotations: kubeblocks.io/reconcile: 2023-08-25T03:56:03.521923563Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-08-25T03:24:32Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 13 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"bookies"}: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:componentDefRef: f:monitor: f:name: f:noCreatePDB: f:tolerations: k:{"name":"broker"}: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:componentDefRef: f:monitor: f:name: f:noCreatePDB: f:tolerations: k:{"name":"zookeeper"}: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:componentDefRef: f:monitor: f:name: f:noCreatePDB: f:tolerations: f:tenancy: f:terminationPolicy: Manager: kubectl-client-side-apply Operation: Update Time: 2023-08-25T03:24:32Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:labels: f:app.kubernetes.io/instance: Manager: kbcli Operation: Update Time: 2023-08-25T03:33:45Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: f:kubeblocks.io/reconcile: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: .: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: f:spec: f:componentSpecs: k:{"name":"bookies"}: f:replicas: f:resources: f:volumeClaimTemplates: k:{"name":"broker"}: f:replicas: f:resources: k:{"name":"zookeeper"}: f:classDefRef: .: f:class: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:requests: .: f:cpu: f:memory: f:volumeClaimTemplates: f:monitor: f:resources: .: f:cpu: f:memory: f:storage: .: f:size: Manager: manager Operation: Update Time: 2023-08-25T03:54:25Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:bookies: .: f:message: .: f:Pod/pulsar-bookies-1: f:phase: f:podsReady: f:broker: .: f:phase: f:podsReady: f:podsReadyTime: f:zookeeper: .: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-08-25T03:58:05Z Resource Version: 36149 UID: 9c98822c-8e1f-45d7-a117-d9f2daff31a8 Spec: Cluster Definition Ref: pulsar Cluster Version Ref: pulsar-2.11.2 Component Specs: Affinity: Pod Anti Affinity: Preferred Tenancy: DedicatedNode Component Def Ref: pulsar-broker Monitor: true Name: broker No Create PDB: false Replicas: 3 Resources: Tolerations: Key: pulsar-broker Operator: Equal Value: true Affinity: Pod Anti Affinity: Preferred Tenancy: DedicatedNode Component Def Ref: bookies Monitor: true Name: bookies No Create PDB: false Replicas: 2 Resources: Tolerations: Key: pulsar-bookies Operator: Equal Value: true Volume Claim Templates: Name: ledgers Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 11Gi Storage Class Name: openebs-lvmpv-pulsar-ldg Name: journal Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 11Gi Storage Class Name: openebs-lvmpv-pulsar-jnl Affinity: Pod Anti Affinity: Preferred Tenancy: DedicatedNode Class Def Ref: Class: Component Def Ref: zookeeper Monitor: true Name: zookeeper No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 200m Memory: 644245094400m Requests: Cpu: 200m Memory: 644245094400m Tolerations: Key: pulsar-zookeeper Operator: Equal Value: true Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Storage Class Name: openebs-lvmpv-pulsar-ldg Monitor: Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Tenancy: DedicatedNode Termination Policy: Delete Status: Cluster Def Generation: 2 Components: Bookies: Message: Pod/pulsar-bookies-1: back-off 2m40s restarting failed container=bookies pod=pulsar-bookies-1_default(23b69191-b64d-4d3a-890a-d2f0c3dc61c4) Phase: Abnormal Pods Ready: false Broker: Phase: Running Pods Ready: true Pods Ready Time: 2023-08-25T03:52:34Z Zookeeper: Phase: Running Pods Ready: true Pods Ready Time: 2023-08-25T03:37:38Z Conditions: Last Transition Time: 2023-08-25T03:54:50Z Message: HorizontalScaling opsRequest: pulsar-horizontalscaling-l2w88 has been processed Reason: Processed Status: True Type: LatestOpsRequestProcessed Last Transition Time: 2023-08-25T03:24:32Z Message: The operator has started the provisioning of Cluster: pulsar Observed Generation: 13 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-08-25T03:53:05Z Message: Successfully applied for resources Observed Generation: 13 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2023-08-25T03:53:46Z Message: pods are not ready in Components: [bookies], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2023-08-25T03:53:46Z Message: pods are unavailable in Components: [bookies], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 13 Phase: Abnormal Events: Type Reason Age From Message

Normal ComponentPhaseTransition 33m (x3 over 33m) cluster-controller Create a new component Normal PreCheckSucceed 33m cluster-controller The operator has started the provisioning of Cluster: pulsar Warning ReplicasNotReady 33m cluster-controller pods are not ready in Components: [bookies broker], refer to related component message in Cluster.status.components Warning ComponentsNotReady 33m cluster-controller pods are unavailable in Components: [bookies broker], refer to related component message in Cluster.status.components Warning ApplyResourcesFailed 32m cluster-controller Operation cannot be fulfilled on statefulsets.apps "pulsar-bookies": the object has been modified; please apply your changes to the latest version and try again Normal ApplyResourcesSucceed 32m (x2 over 33m) cluster-controller Successfully applied for resources Warning ComponentsNotReady 32m cluster-controller pods are unavailable in Components: [broker], refer to related component message in Cluster.status.components Warning ReplicasNotReady 32m cluster-controller pods are not ready in Components: [broker], refer to related component message in Cluster.status.components Normal AllReplicasReady 31m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 31m cluster-controller Cluster: pulsar is ready, current phase is Running Normal Running 31m cluster-controller Cluster: pulsar is ready, current phase is Running Normal Reconfiguring 24m ops-request-controller Start to process the Reconfiguring opsRequest "pulsar-reconfiguring-pqcv9" in Cluster: pulsar Warning ApplyResourcesFailed 24m cluster-controller Operation cannot be fulfilled on statefulsets.apps "pulsar-zookeeper": the object has been modified; please apply your changes to the latest version and try again Normal ComponentPhaseTransition 24m (x5 over 33m) cluster-controller Running: true, PodsReady: true, PodsTimedout: false Normal Restart 23m ops-request-controller Start to process the Restart opsRequest "pulsar-restart-tpbgj" in Cluster: pulsar Normal VerticalScaling 21m ops-request-controller Start to process the VerticalScaling opsRequest "pulsar-verticalscaling-92flz" in Cluster: pulsar Normal Stop 21m ops-request-controller Start to process the Stop opsRequest "pulsar-stop-s6hnp" in Cluster: pulsar Normal Start 20m ops-request-controller Start to process the Start opsRequest "pulsar-start-ktztt" in Cluster: pulsar Normal Reconfiguring 18m ops-request-controller Start to process the Reconfiguring opsRequest "pulsar-reconfiguring-v9h57" in Cluster: pulsar Normal WaitingForProbeSuccess 18m (x15 over 24m) cluster-controller Waiting for probe success Normal VolumeExpansion 16m ops-request-controller Start to process the VolumeExpansion opsRequest "pulsar-volumeexpansion-jqqd6" in Cluster: pulsar Normal VolumeExpansion 15m ops-request-controller Start to process the VolumeExpansion opsRequest "pulsar-volumeexpansion-cg8wb" in Cluster: pulsar Normal ComponentPhaseTransition 13m (x22 over 24m) cluster-controller Component workload updated Normal VolumeExpansion 13m ops-request-controller Start to process the VolumeExpansion opsRequest "pulsar-volumeexpansion-wqlfk" in Cluster: pulsar Warning HorizontalScaling 7m56s cluster-controller HorizontalScaling opsRequest: pulsar-horizontalscaling-tvjj8 is processing Normal HorizontalScaling 7m56s ops-request-controller Start to process the HorizontalScaling opsRequest "pulsar-horizontalscaling-tvjj8" in Cluster: pulsar Normal HorizontalScaling 7m2s ops-request-controller Start to process the HorizontalScaling opsRequest "pulsar-horizontalscaling-gz8rz" in Cluster: pulsar Normal HorizontalScaling 6m14s ops-request-controller Start to process the HorizontalScaling opsRequest "pulsar-horizontalscaling-8f9k7" in Cluster: pulsar Normal HorizontalScaling 5m3s ops-request-controller Start to process the HorizontalScaling opsRequest "pulsar-horizontalscaling-gg8tm" in Cluster: pulsar Normal HorizontalScaling 4m22s ops-request-controller Start to process the HorizontalScaling opsRequest "pulsar-horizontalscaling-l2w88" in Cluster: pulsar Warning FailedScheduling 3m40s event-controller Pod pulsar-bookies-1: 0/9 nodes are available: 3 node(s) did not have enough free storage, 3 node(s) had taint {pulsar-broker: true}, that the pod didn't tolerate, 3 node(s) had taint {pulsar-zookeeper: true}, that the pod didn't tolerate. Normal Processed 3m15s (x2 over 4m54s) cluster-controller (combined from similar events): HorizontalScaling opsRequest: pulsar-horizontalscaling-l2w88 has been processed Warning BackOff 2m2s (x5 over 2m45s) event-controller Pod pulsar-bookies-1: Back-off restarting failed container

[root@ip-172-31-25-255 ~]# k get cluster pulsar -o yaml apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2023-08-25T03:56:03.521923563Z" kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":{"annotations":{},"name":"pulsar","namespace":"default"},"spec":{"clusterDefinitionRef":"pulsar","clusterVersionRef":"pulsar-2.11.2","componentSpecs":[{"affinity":{"tenancy":"DedicatedNode"},"componentDefRef":"pulsar-broker","monitor":true,"name":"broker","replicas":3,"tolerations":[{"key":"pulsar-broker","operator":"Equal","value":"true"}]},{"affinity":{"tenancy":"DedicatedNode"},"componentDefRef":"bookies","monitor":true,"name":"bookies","replicas":3,"tolerations":[{"key":"pulsar-bookies","operator":"Equal","value":"true"}],"volumeClaimTemplates":[{"name":"ledgers","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"openebs-lvmpv-pulsar-ldg"}},{"name":"journal","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"openebs-lvmpv-pulsar-jnl"}}]},{"affinity":{"tenancy":"DedicatedNode"},"componentDefRef":"zookeeper","monitor":true,"name":"zookeeper","replicas":3,"tolerations":[{"key":"pulsar-zookeeper","operator":"Equal","value":"true"}],"volumeClaimTemplates":[{"name":"data","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"openebs-lvmpv-pulsar-ldg"}}]}],"tenancy":"DedicatedNode","terminationPolicy":"Delete"}} creationTimestamp: "2023-08-25T03:24:32Z" finalizers:

wangyelei commented 1 year ago

bookies at least 3 nodes, otherwise it can not write. Ensemble Size is 3 by default