search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.15k stars 3.57k forks source link

Error when trying to run Producer-Consumer commands in a 2 node cluster #5630

Closed vishalananth07 closed 4 years ago

vishalananth07 commented 4 years ago

Describe the bug Producer command

$ bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"

and Consumer command

bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"

are giving the following error messages:

Producer error message:

bin/pulsar-client produce persistent://public/default/test -n 1 -m "hello pulsar"
14:28:32.171 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x2c91e691, L:/192.168.1.13:35714 - R:192.168.1.5/192.168.1.5:6650]] Connected to server
14:28:32.707 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
  "topicName" : "persistent://public/default/test",
  "producerName" : null,
  "sendTimeoutMs" : 30000,
  "blockIfQueueFull" : false,
  "maxPendingMessages" : 1000,
  "maxPendingMessagesAcrossPartitions" : 50000,
  "messageRoutingMode" : "RoundRobinPartition",
  "hashingScheme" : "JavaStringHash",
  "cryptoFailureAction" : "FAIL",
  "batchingMaxPublishDelayMicros" : 1000,
  "batchingMaxMessages" : 1000,
  "batchingEnabled" : true,
  "compressionType" : "NONE",
  "initialSequenceId" : null,
  "autoUpdatePartitions" : true,
  "properties" : { }
}
14:28:32.710 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
  "serviceUrl" : "pulsar://192.168.1.13:6650,192.168.1.5:6650",
  "authPluginClassName" : null,
  "authParams" : null,
  "operationTimeoutMs" : 30000,
  "statsIntervalSeconds" : 60,
  "numIoThreads" : 1,
  "numListenerThreads" : 1,
  "connectionsPerBroker" : 1,
  "useTcpNoDelay" : true,
  "useTls" : false,
  "tlsTrustCertsFilePath" : "",
  "tlsAllowInsecureConnection" : false,
  "tlsHostnameVerificationEnable" : false,
  "concurrentLookupRequest" : 5000,
  "maxLookupRequest" : 50000,
  "maxNumberOfRejectedRequestPerConnection" : 50,
  "keepAliveIntervalSeconds" : 30,
  "connectionTimeoutMs" : 10000,
  "requestTimeoutMs" : 60000,
  "defaultBackoffIntervalNanos" : 100000000,
  "maxBackoffIntervalNanos" : 30000000000
}
14:28:32.714 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x6e57db24, L:/192.168.1.13:43896 - R:192.168.1.13/192.168.1.13:6650]] Connected to server
14:28:32.734 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]] Connected to server
14:28:32.741 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Creating producer on cnx [id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]
14:28:32.864 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [test-cluster-1-0-3] Created producer on cnx [id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]
14:28:32.951 [main] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
14:29:02.954 [pulsar-timer-4-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [test-cluster-1-0-3] Message send timed out. Failing 1 messages
14:29:02.964 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
14:29:02.965 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Could not send message to broker within given timeout
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Could not send message to broker within given timeout
    at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:247) ~[org.apache.pulsar-pulsar-client-api-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:86) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:61) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:51) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:189) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:169) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:131) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
    at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:165) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
14:29:02.978 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced

Consumer error message:

bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"
14:29:02.442 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xc40711ef, L:/192.168.1.5:46150 - R:192.168.1.13/192.168.1.13:6650]] Connected to server
14:29:02.653 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer perf with config: {
  "topicNames" : [ "persistent://public/default/test" ],
  "topicsPattern" : null,
  "subscriptionName" : "consumer-test",
  "subscriptionType" : "Exclusive",
  "receiverQueueSize" : 1000,
  "acknowledgementsGroupTimeMicros" : 100000,
  "negativeAckRedeliveryDelayMicros" : 60000000,
  "maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
  "consumerName" : null,
  "ackTimeoutMillis" : 0,
  "tickDurationMillis" : 1000,
  "priorityLevel" : 0,
  "cryptoFailureAction" : "FAIL",
  "properties" : { },
  "readCompacted" : false,
  "subscriptionInitialPosition" : "Latest",
  "patternAutoDiscoveryPeriod" : 1,
  "regexSubscriptionMode" : "PersistentOnly",
  "deadLetterPolicy" : null,
  "autoUpdatePartitions" : true,
  "replicateSubscriptionState" : false,
  "resetIncludeHead" : false
}
14:29:02.660 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {
  "serviceUrl" : "pulsar://192.168.1.13:6650,192.168.1.5:6650",
  "authPluginClassName" : null,
  "authParams" : null,
  "operationTimeoutMs" : 30000,
  "statsIntervalSeconds" : 60,
  "numIoThreads" : 1,
  "numListenerThreads" : 1,
  "connectionsPerBroker" : 1,
  "useTcpNoDelay" : true,
  "useTls" : false,
  "tlsTrustCertsFilePath" : "",
  "tlsAllowInsecureConnection" : false,
  "tlsHostnameVerificationEnable" : false,
  "concurrentLookupRequest" : 5000,
  "maxLookupRequest" : 50000,
  "maxNumberOfRejectedRequestPerConnection" : 50,
  "keepAliveIntervalSeconds" : 30,
  "connectionTimeoutMs" : 10000,
  "requestTimeoutMs" : 60000,
  "defaultBackoffIntervalNanos" : 100000000,
  "maxBackoffIntervalNanos" : 30000000000
}
14:29:02.679 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xa7cf0295, L:/192.168.1.5:42154 - R:192.168.1.5/192.168.1.5:6650]] Connected to server
14:29:02.911 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to sundar-UX303UB:6650 : java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries 
14:29:02.913 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [consumer-test] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries 
14:29:02.913 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [consumer-test] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries  -- Will try again in 0.1 s

To Reproduce Steps to reproduce the behavior:

  1. Start zookeeper, set cluster meta data, start bookie, start broker after modifying necessary configurations.
  2. Bookiesanity is succeeding but simpletest fails with the following error, we are using 2 systems which run zookeeper,bookkeeper and broker 
    bin/bookkeeper shell simpletest --ensemble 2 --writeQuorum 2 --ackQuorum 2 --numEntries 2
JMX enabled by default
14:43:53.364 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
14:43:53.366 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
14:43:53.366 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
14:43:53.373 [main] INFO  org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://192.168.1.13:2181;192.168.1.5:2181/ledgers : zkServers = 192.168.1.13:2181,192.168.1.5:2181, ledgersRootPath = /ledgers.
14:43:53.381 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
14:43:53.381 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=sundar-UX303UB
14:43:53.382 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_222
14:43:53.382 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Private Build
14:43:53.382 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
14:43:53.382 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/sundar/apache-pulsar-2.4.1/conf:::/home/sundar/apache-pulsar-2.4.1/lib/com.beust-jcommander-1.48.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.carrotsearch-hppc-0.7.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-annotations-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-core-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-databind-2.9.9.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.ben-manes.caffeine-caffeine-2.6.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.luben-zstd-jni-1.3.7-3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.api.grpc-proto-google-common-protos-1.12.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.code.gson-gson-2.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.errorprone-error_prone_annotations-2.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.guava-guava-21.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-cli-commons-cli-1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-codec-commons-codec-1.10.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-collections-commons-collections-3.2.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-configuration-commons-configuration-1.10.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-io-commons-io-2.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-lang-commons-lang-2.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-logging-commons-logging-1.1.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-logging-interceptor-2.7.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-ws-2.7.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okio-okio-1.13.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.sun.activation-javax.activation-1.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.thoughtworks.paranamer-paranamer-2.7.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-memory-0.8.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-core-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-graphite-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-jvm-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-all-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-auth-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-context-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-core-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-netty-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-okhttp-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-lite-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-nano-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-stub-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-testing-1.12.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-api-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-impl-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-jackson-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-api-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-proto-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-3.10.1.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-all-4.1.32.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-tcnative-boringssl-static-2.0.20.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-api-0.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_common-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-annotations-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-core-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-models-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-auth-common-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-core-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-web-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.annotation-javax.annotation-api-1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.inject-javax.inject-1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.validation-validation-api-1.1.0.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/jline-jline-0.9.94.jar:/home/sundar/apache-pulsar-2.4.1/lib/joda-time-joda-time-2.10.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/log4j-log4j-1.2.17.jar:/home/sundar/apache-pulsar-2.4.1/lib/net.java.dev.jna-jna-4.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/net.jodah-typetools-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.avro-avro-1.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.avro-avro-protobuf-1.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-proto-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-circe-checksum-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-cpu-affinity-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-http-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-vertx-http-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-statelib-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-proto-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-impl-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-collections4-4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-compress-1.15.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-lang3-3.4.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-client-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-framework-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-recipes-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2-tests.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-protocol-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-managed-ledger-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-protobuf-shaded-2.1.0-incubating.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-api-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-tools-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-discovery-service-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-examples-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-proto-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-secrets-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-utils-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-worker-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-io-core-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-proxy-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-testclient-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-transaction-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-websocket-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.aspectj-aspectjrt-1.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.aspectj-aspectjweaver-1.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-2.7.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-netty-utils-2.7.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcpkix-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-ext-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.checkerframework-checker-compat-qual-2.5.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-core-asl-1.9.13.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-mapper-asl-1.9.13.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.mojo-animal-sniffer-annotations-1.17.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-client-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-continuation-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-http-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-proxy-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-security-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlets-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-xml-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-api-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-client-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-common-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-server-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-api-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-locator-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-utils-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-client-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-common-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-server-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.inject-jersey-hk2-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-multipart-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.inferred-freebuilder-1.14.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.javassist-javassist-3.25.0-GA.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.jctools-jctools-core-2.1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.lz4-lz4-java-1.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.objenesis-objenesis-2.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.reactivestreams-reactive-streams-1.0.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.rocksdb-rocksdbjni-5.13.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-jcl-over-slf4j-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-slf4j-api-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.tukaani-xz-1.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.xerial.snappy-snappy-java-1.1.1.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.yaml-snakeyaml-1.23.jar:
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/home/sundar/ns-allinone-2.35/otcl-1.14/:/home/sundar/ns-allinone-2.35/lib/:/usr/local/lib/:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-66-generic
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=sundar
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/sundar
14:43:53.384 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/sundar/apache-pulsar-2.4.1
14:43:53.389 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.13:2181,192.168.1.5:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@50468873
14:43:53.510 [main-SendThread(192.168.1.5:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.5/192.168.1.5:2181. Will not attempt to authenticate using SASL (unknown error)
14:43:53.517 [main-SendThread(192.168.1.5:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.5/192.168.1.5:2181, initiating session
14:43:53.536 [main-SendThread(192.168.1.5:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.1.5/192.168.1.5:2181, sessionid = 0x2000535ffc2000e, negotiated timeout = 30000
14:43:53.541 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
14:43:54.506 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver.
14:43:54.539 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:127.0.1.1:0> @ /default-rack : org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl$DefaultResolver.
14:43:54.540 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
14:43:54.548 [main] INFO  org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
14:43:54.592 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/sundar-UX303UB:3181
14:43:54.593 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/vishal-Satellite-C55-B:3181
14:43:54.641 [main] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [vishal-Satellite-C55-B:3181, sundar-UX303UB:3181] is not adhering to Placement Policy. quarantinedBookies: []
14:43:54.808 [main-EventThread] ERROR org.apache.bookkeeper.proto.checksum.CRC32CDigestManager - Sse42Crc32C is not supported, will use a slower CRC32C implementation.
14:43:54.818 [main-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [vishal-Satellite-C55-B:3181, sundar-UX303UB:3181] for ledger: 1371
Ledger ID: 1371
14:43:54.888 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
14:43:55.006 [bookkeeper-io-3-1] WARN  org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x6515e025]/vishal-Satellite-C55-B:3181, current state CONNECTING : vishal-Satellite-C55-B
14:43:55.008 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: [id: 0xdb053f5e, L:/127.0.0.1:41224 - R:sundar-UX303UB/127.0.1.1:3181]
14:43:55.019 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: sundar-UX303UB:3181
14:43:55.020 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (1371, 0): Bookie handle is not available
14:43:55.037 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>].
14:43:55.039 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], fallback to choose bookie randomly from the cluster.
14:43:55.039 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:vishal-Satellite-C55-B:3181>, <Bookie:sundar-UX303UB:3181>].
14:43:55.040 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>].
14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], fallback to choose bookie randomly from the cluster.
14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:vishal-Satellite-C55-B:3181>, <Bookie:sundar-UX303UB:3181>].
14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=1371,loopId=2d2b0a86) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
14:43:55.052 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xdb053f5e, L:/127.0.0.1:41224 - R:sundar-UX303UB/127.0.1.1:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
14:43:55.056 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN  org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:1371, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
14:43:55.058 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 1371 due to NotEnoughBookiesException: Not enough non-faulty bookies available
14:43:55.061 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L1371 E0
14:43:55.186 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for sundar-UX303UB:3181
14:43:55.190 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for vishal-Satellite-C55-B:3181
14:43:55.190 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xdb053f5e, L:/127.0.0.1:41224 ! R:sundar-UX303UB/127.0.1.1:3181]
14:43:55.210 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x2000535ffc2000e closed
14:43:55.210 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x2000535ffc2000e
14:43:55.211 [main] ERROR org.apache.bookkeeper.tools.cli.helpers.ClientCommand - Failed to process command 'simpletest'
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:390) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1772) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:572) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
  3. Try running the producer and consumer commands on different broker systems after modifying the service urls in client.conf(we are using the following url webServiceUrl=http://192.168.1.13:8080,192.168.1.5:8080, brokerServiceUrl=pulsar://192.168.1.13:6650,192.168.1.5:6650)

Expected behavior Producer should produce message and Consumer should consume it.

Desktop (please complete the following information):

Additional context All of Zookeeper, Bookkeeper and Broker are starting, we are able to see all the processes running when we give jps command. But when we try to run producer or consumer command consumer broker crashes and error message is displayed.

vishalananth07 commented 4 years ago

I noticed that the Broker process stops automatically after 2 minutes on one node, but on the other node its running fine. Also the node in which the broker stops is not able to produce and consume, but the other node is able to run as consumer alone but fails when it tries to run the producer.

sijie commented 4 years ago

org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available

It indicates that you don't have enough bookies. use bookkeeper shell listbookies to list the bookies. make sure you have enough bookies, because by default you need at least 2 bookies.

vishalananth07 commented 4 years ago

I have run bookkeeper on two nodes

jiazhai commented 4 years ago

@vishalananth07 Please check the log of each bookie node, there should be some error caused bookie not start success.

vishalananth07 commented 4 years ago

Sorry for the late reply, one of our machines crashed. We tried using another machine and tried setting up Pulsar, but we could not even start Bookie because of less RAM availability. We used two machines one with 12Gb RAM + SSD and another with 8Gb RAM + HDD, Bookie ran perfectly on the 12GB system, but in the 8Gb system, it crashed exactly after 2 minutes. Is lesser RAM causing the issue? What are the recommended specs for running Pulsar?

jiazhai commented 4 years ago

@vishalananth07 Have you figure out the issue. 8G RAM should be enough. you could check the settings in conf/pulsar_env.sh and conf/bkenv.sh

vishalananth07 commented 4 years ago

Yes, I have fixed this issue, there were other processes taking more RAM in my machine. Thanks for your help :)