search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.25k stars 3.58k forks source link

Bookie does not start because of UnknownHostException #5749

Closed vishalananth07 closed 4 years ago

vishalananth07 commented 4 years ago

We are trying to set up Pulsar using 3 systems each with 8GB RAM and AMD A8 processors. We are planning on running bookkeeper, zookeeper, broker on the same machines.

Describe the bug [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server java.net.UnknownHostException: Trying to listen on loopback address, 127.0.1.1:3181 but this is forbidden by default (see ServerConfiguration#getAllowLoopback()).

To Reproduce Steps to reproduce the behavior:

  1. Follow the setup steps given at https://pulsar.apache.org/docs/en/deploy-bare-metal/

  2. We added the servers to the zookeeper.conf file and create data directory and id. We are using the following Zookeper configuration

    server.1=10.6.8.14:2888:3888
server.2=10.6.8.13:2888:3888
server.3=10.6.8.15:2888:3888

  3. We initialized cluster metadata as follows in the server.1 machine

    bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster-1 --zookeeper 10.6.8.14:2181 --web-service-url http://10.6.8.14:8080,10.6.8.13:8080,10.6.8.15:8080 --broker-service-url pulsar://10.6.8.14:6650,10.6.8.13:6650,10.6.8.15:6650 --configuration-store 10.6.8.14:2181

  4. We changed bookkeeper.conf as follows

    zkServers=10.6.8.14:2181,10.6.8.13:2181,10.6.8.15:2181

  5. We started Bookie and got the following error

    bin/bookkeeper bookie
JMX enabled by default
JAVA_HOME not set, using java from PATH. (/usr/bin/java)
15:02:38.366 [main] INFO  org.apache.bookkeeper.server.Main - Using configuration file /home/slave1fyp/apache-pulsar-2.4.1/conf/bookkeeper.conf
15:02:38.371 [main] INFO  org.apache.bookkeeper.server.Main - Hello, I'm your bookie, listening on port 3181. Metadata service uri is zk+null://10.6.8.14:2181;10.6.8.13:2181;10.6.8.15:2181/ledgers. Journals are in [data/bookkeeper/journal]. Ledgers are stored in data/bookkeeper/ledgers.
15:02:38.394 [main] INFO  org.apache.bookkeeper.server.Main - Load lifecycle component : org.apache.bookkeeper.server.service.StatsProviderService
15:02:38.741 [main] INFO  org.apache.bookkeeper.proto.BookieServer - {
"readBufferSizeBytes" : "4096",
"statsProviderClass" : "org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider",
"majorCompactionThreshold" : "0.5",
"numJournalCallbackThreads" : "8",
"httpServerPort" : "8000",
"lostBookieRecoveryDelay" : "0",
"journalAlignmentSize" : "4096",
"compactionRateByBytes" : "1000000",
"httpServerClass" : "org.apache.bookkeeper.http.vertx.VertxHttpServer",
"dbStorage_rocksDB_numFilesInLevel0" : "4",
"minUsableSizeForIndexFileCreation" : "1073741824",
"gcOverreplicatedLedgerWaitTime" : "86400000",
"journalMaxGroupWaitMSec" : "1",
"dbStorage_rocksDB_numLevels" : "-1",
"dbStorage_rocksDB_bloomFilterBitsPerKey" : "10",
"ledgerStorageClass" : "org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage",
"auditorPeriodicBookieCheckInterval" : "86400",
"gcWaitTime" : "900000",
"compactionRate" : "1000",
"fileInfoFormatVersionToWrite" : "0",
"entryLogFilePreallocationEnabled" : "true",
"journalSyncData" : "true",
"zkServers" : "[10.6.8.14:2181, 10.6.8.13:2181, 10.6.8.15:2181]",
"compactionRateByEntries" : "1000",
"dbStorage_rocksDB_maxSizeInLevel1MB" : "256",
"diskCheckInterval" : "10000",
"auditorPeriodicCheckInterval" : "604800",
"dbStorage_rocksDB_writeBufferSizeMB" : "64",
"autoRecoveryDaemonEnabled" : "true",
"maxPendingAddRequestsPerThread" : "10000",
"majorCompactionInterval" : "86400",
"httpServerEnabled" : "false",
"flushInterval" : "60000",
"journalFlushWhenQueueEmpty" : "false",
"minorCompactionInterval" : "3600",
"dbStorage_rocksDB_blockCacheSize" : "",
"isThrottleByBytes" : "false",
"numAddWorkerThreads" : "0",
"dbStorage_rocksDB_sstSizeInMB" : "64",
"journalDirectory" : "data/bookkeeper/journal",
"journalWriteBufferSizeKB" : "64",
"diskUsageThreshold" : "0.95",
"openFileLimit" : "0",
"prometheusStatsHttpPort" : "8000",
"zkLedgersRootPath" : "/ledgers",
"journalMaxSizeMB" : "2048",
"journalAdaptiveGroupWrites" : "true",
"openLedgerRereplicationGracePeriod" : "30",
"ledgerDirectories" : "data/bookkeeper/ledgers",
"zkTimeout" : "30000",
"dbStorage_rocksDB_blockSize" : "65536",
"journalMaxBackups" : "5",
"maxPendingReadRequestsPerThread" : "2500",
"useHostNameAsBookieID" : "false",
"rereplicationEntryBatchSize" : "100",
"allowLoopback" : "false",
"readOnlyModeEnabled" : "true",
"journalRemoveFromPageCache" : "true",
"dbStorage_readAheadCacheMaxSizeMb" : "",
"zkEnableSecurity" : "false",
"numHighPriorityWorkerThreads" : "8",
"dbStorage_readAheadCacheBatchSize" : "1000",
"journalFormatVersionToWrite" : "5",
"writeBufferSizeBytes" : "65536",
"bookiePort" : "3181",
"dbStorage_writeCacheMaxSizeMb" : "",
"pageLimit" : "0",
"logSizeLimit" : "1073741824",
"advertisedAddress" : "",
"bookieDeathWatchInterval" : "1000",
"numReadWorkerThreads" : "8",
"minorCompactionThreshold" : "0.2",
"serverTcpNoDelay" : "true",
"journalBufferedWritesThreshold" : "524288",
"compactionMaxOutstandingRequests" : "100000",
"journalPreAllocSizeMB" : "16"
}
15:02:39.490 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
java.net.UnknownHostException: Trying to listen on loopback address, 127.0.1.1:3181 but this is forbidden by default (see ServerConfiguration#getAllowLoopback()).
If this happen, you can consider specifying the network interface to listen on (e.g. listeningInterface=eth0) or specifying the advertised address (e.g. advertisedAddress=172.x.y.z)
at org.apache.bookkeeper.bookie.Bookie.getBookieAddress(Bookie.java:570) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieNettyServer.<init>(BookieNettyServer.java:164) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:104) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:314) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]

Expected behavior Bookkeeper starts

Desktop (please complete the following information):

jiazhai commented 4 years ago

@vishalananth07 regarding this issue:

[main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
java.net.UnknownHostException: Trying to listen on loopback address, 127.0.1.1:3181 but this is forbidden by default (see ServerConfiguration#getAllowLoopback()).

this is because listen on 127.0.0.1 while you are setting "allowLoopback=false" in bookkeeper.conf. you could try to set useHostNameAsBookieID=true in bookkeeper.conf

vishalananth07 commented 4 years ago

@jiazhai sorry for the delayed response, I tried changing useHostNameAsBookieID=true, still Bookie failed with same error. I tried changing allowLoopback=true and the Bookie started and Bookie sanity passed. But when we tried (we have 3 bookies and 3 zookeepers on the same nodes)

bin/bookkeeper shell simpletest --ensemble 3 --writeQuorum 3 --ackQuorum 3 --numEntries 3

It failed with the following error

JMX enabled by default
JAVA_HOME not set, using java from PATH. (/usr/bin/java)
08:26:43.490 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
08:26:43.493 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
08:26:43.494 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
08:26:43.504 [main] INFO  org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://10.6.8.14:2181;10.6.8.13:2181;10.6.8.15:2181/ledgers : zkServers = 10.6.8.14:2181,10.6.8.13:2181,10.6.8.15:2181, ledgersRootPath = /ledgers.
08:26:43.515 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
08:26:43.515 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=slave1fyp-HP-Compaq-Pro-6305-SFF
08:26:43.515 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_222
08:26:43.515 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Private Build
08:26:43.515 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
08:26:43.515 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/slave1fyp/apache-pulsar-2.4.1/conf:::/home/slave1fyp/apache-pulsar-2.4.1/lib/com.beust-jcommander-1.48.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.carrotsearch-hppc-0.7.3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-annotations-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-core-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-databind-2.9.9.3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.github.ben-manes.caffeine-caffeine-2.6.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.github.luben-zstd-jni-1.3.7-3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.api.grpc-proto-google-common-protos-1.12.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.code.gson-gson-2.8.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.errorprone-error_prone_annotations-2.2.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.guava-guava-21.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-cli-commons-cli-1.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-codec-commons-codec-1.10.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-collections-commons-collections-3.2.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-configuration-commons-configuration-1.10.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-io-commons-io-2.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-lang-commons-lang-2.6.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/commons-logging-commons-logging-1.1.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.squareup.okhttp-logging-interceptor-2.7.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-ws-2.7.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.squareup.okio-okio-1.13.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.sun.activation-javax.activation-1.2.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.thoughtworks.paranamer-paranamer-2.7.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-memory-0.8.3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-core-3.1.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-graphite-3.1.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-jvm-3.1.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-all-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-auth-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-context-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-core-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-netty-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-okhttp-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-lite-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-nano-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-stub-1.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.grpc-grpc-testing-1.12.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-api-0.10.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-impl-0.10.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-jackson-0.10.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-2.0.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-api-2.0.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-proto-2.0.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.netty-netty-3.10.1.Final.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.netty-netty-all-4.1.32.Final.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.netty-netty-tcnative-boringssl-static-2.0.20.Final.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-api-0.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_common-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.swagger-swagger-annotations-1.5.21.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.swagger-swagger-core-1.5.21.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.swagger-swagger-models-1.5.21.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.vertx-vertx-auth-common-3.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.vertx-vertx-core-3.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/io.vertx-vertx-web-3.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.annotation-javax.annotation-api-1.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.inject-javax.inject-1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.validation-validation-api-1.1.0.Final.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/jline-jline-0.9.94.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/joda-time-joda-time-2.10.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/log4j-log4j-1.2.17.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/net.java.dev.jna-jna-4.2.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/net.jodah-typetools-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.avro-avro-1.8.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.avro-avro-protobuf-1.8.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-proto-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-circe-checksum-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-cpu-affinity-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-http-server-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-vertx-http-server-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-statelib-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-api-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-common-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-proto-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-server-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-api-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-impl-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.commons-commons-collections4-4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.commons-commons-compress-1.15.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.commons-commons-lang3-3.4.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.curator-curator-client-4.0.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.curator-curator-framework-4.0.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.curator-curator-recipes-4.0.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-common-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2-tests.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-protocol-4.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-managed-ledger-original-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-protobuf-shaded-2.1.0-incubating.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-common-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-api-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-original-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-tools-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-common-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-discovery-service-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-examples-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-proto-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-secrets-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-utils-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-worker-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-io-core-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-proxy-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-testclient-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-transaction-common-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-websocket-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.aspectj-aspectjrt-1.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.aspectj-aspectjweaver-1.9.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-2.7.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-netty-utils-2.7.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.bouncycastle-bcpkix-jdk15on-1.60.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-ext-jdk15on-1.60.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-jdk15on-1.60.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.checkerframework-checker-compat-qual-2.5.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-core-asl-1.9.13.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-mapper-asl-1.9.13.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.codehaus.mojo-animal-sniffer-annotations-1.17.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-client-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-continuation-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-http-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-proxy-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-security-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlets-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-xml-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-api-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-client-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-common-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-server-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.12.v20180830.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b42.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b42.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-api-2.5.0-b42.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-locator-2.5.0-b42.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-utils-2.5.0-b42.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-client-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-common-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-server-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.inject-jersey-hk2-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-multipart-2.27.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.inferred-freebuilder-1.14.9.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.javassist-javassist-3.25.0-GA.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.jctools-jctools-core-2.1.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.lz4-lz4-java-1.5.0.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.objenesis-objenesis-2.6.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.reactivestreams-reactive-streams-1.0.2.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.rocksdb-rocksdbjni-5.13.3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.slf4j-jcl-over-slf4j-1.7.25.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.slf4j-slf4j-api-1.7.25.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.tukaani-xz-1.5.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.xerial.snappy-snappy-java-1.1.1.3.jar:/home/slave1fyp/apache-pulsar-2.4.1/lib/org.yaml-snakeyaml-1.23.jar:
08:26:43.519 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
08:26:43.519 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
08:26:43.519 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
08:26:43.519 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
08:26:43.520 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
08:26:43.520 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.10.0-28-generic
08:26:43.520 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=slave1fyp
08:26:43.520 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/slave1fyp
08:26:43.520 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/slave1fyp/apache-pulsar-2.4.1
08:26:43.526 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.6.8.14:2181,10.6.8.13:2181,10.6.8.15:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@2b4c1d96
08:26:43.562 [main-SendThread(10.6.8.15:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.6.8.15/10.6.8.15:2181. Will not attempt to authenticate using SASL (unknown error)
08:26:43.570 [main-SendThread(10.6.8.15:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 10.6.8.15/10.6.8.15:2181, initiating session
08:26:43.602 [main-SendThread(10.6.8.15:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 10.6.8.15/10.6.8.15:2181, sessionid = 0x3000010c7d50002, negotiated timeout = 30000
08:26:43.606 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
08:26:44.755 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver.
08:26:44.791 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:127.0.1.1:0> @ /default-rack : org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl$DefaultResolver.
08:26:44.791 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
08:26:44.796 [main] INFO  org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
08:26:44.840 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/slave2fyp-HP-Compaq-Pro-6305-SFF:3181
08:26:44.841 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/slave1fyp-HP-Compaq-Pro-6305-SFF:3181
08:26:44.843 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/masterfyp-HP-Compaq-Pro-6305-SFF:3181
08:26:44.892 [main] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [slave2fyp-HP-Compaq-Pro-6305-SFF:3181, slave1fyp-HP-Compaq-Pro-6305-SFF:3181, masterfyp-HP-Compaq-Pro-6305-SFF:3181] is not adhering to Placement Policy. quarantinedBookies: []
08:26:45.085 [main-EventThread] ERROR org.apache.bookkeeper.proto.checksum.CRC32CDigestManager - Sse42Crc32C is not supported, will use a slower CRC32C implementation.
08:26:45.090 [main-EventThread] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [slave2fyp-HP-Compaq-Pro-6305-SFF:3181, slave1fyp-HP-Compaq-Pro-6305-SFF:3181, masterfyp-HP-Compaq-Pro-6305-SFF:3181] for ledger: 2
Ledger ID: 2
08:26:45.128 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
08:26:45.260 [bookkeeper-io-3-1] WARN  org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0xb79dd2a6]/slave2fyp-HP-Compaq-Pro-6305-SFF:3181, current state CONNECTING : slave2fyp-HP-Compaq-Pro-6305-SFF
08:26:45.264 [bookkeeper-io-3-3] WARN  org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x59f4a00d]/masterfyp-HP-Compaq-Pro-6305-SFF:3181, current state CONNECTING : masterfyp-HP-Compaq-Pro-6305-SFF
08:26:45.264 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: [id: 0x5be616e6, L:/127.0.0.1:48760 - R:slave1fyp-HP-Compaq-Pro-6305-SFF/127.0.1.1:3181]
08:26:45.283 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (2, 0): Bookie handle is not available
08:26:45.282 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: slave1fyp-HP-Compaq-Pro-6305-SFF:3181
08:26:45.309 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>], allBookies [<Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>].
08:26:45.310 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>], fallback to choose bookie randomly from the cluster.
08:26:45.310 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>], allBookies [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>].
08:26:45.311 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>], allBookies [<Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>].
08:26:45.312 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>], fallback to choose bookie randomly from the cluster.
08:26:45.312 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>], allBookies [<Bookie:slave1fyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:masterfyp-HP-Compaq-Pro-6305-SFF:3181>, <Bookie:slave2fyp-HP-Compaq-Pro-6305-SFF:3181>].
08:26:45.313 [BookKeeperClientWorker-OrderedExecutor-2-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=2,loopId=0d2fbe2f) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
08:26:45.322 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (2, 0): Bookie handle is not available
08:26:45.323 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x5be616e6, L:/127.0.0.1:48760 - R:slave1fyp-HP-Compaq-Pro-6305-SFF/127.0.1.1:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
08:26:45.323 [BookKeeperClientWorker-OrderedExecutor-2-0] WARN  org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:2, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
08:26:45.324 [BookKeeperClientWorker-OrderedExecutor-2-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 2 due to NotEnoughBookiesException: Not enough non-faulty bookies available
08:26:45.332 [BookKeeperClientWorker-OrderedExecutor-2-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L2 E0
08:26:45.358 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for slave2fyp-HP-Compaq-Pro-6305-SFF:3181
08:26:45.364 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for slave1fyp-HP-Compaq-Pro-6305-SFF:3181
08:26:45.373 [bookkeeper-io-3-2] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x5be616e6, L:/127.0.0.1:48760 ! R:slave1fyp-HP-Compaq-Pro-6305-SFF/127.0.1.1:3181]
08:26:45.374 [main] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for masterfyp-HP-Compaq-Pro-6305-SFF:3181
08:26:45.396 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x3000010c7d50002 closed
08:26:45.396 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3000010c7d50002
08:26:45.396 [main] ERROR org.apache.bookkeeper.tools.cli.helpers.ClientCommand - Failed to process command 'simpletest'
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
    at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:390) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1772) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:572) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]

I tried doing ./bin/bookkeeper shell listbookies -rw, but it is failing with this error

08:32:11.981 [main] ERROR org.apache.bookkeeper.tools.cli.helpers.DiscoveryCommand - Fail to process command 'list'
java.lang.NullPointerException: null
    at org.apache.bookkeeper.tools.cli.helpers.CommandHelpers.getBookieSocketAddrStringRepresentation(CommandHelpers.java:46) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand.printBookies(ListBookiesCommand.java:106) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand.run(ListBookiesCommand.java:85) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand.run(ListBookiesCommand.java:39) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.tools.cli.helpers.DiscoveryCommand.apply(DiscoveryCommand.java:57) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.tools.cli.helpers.ClientCommand.apply(ClientCommand.java:60) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.BookieShell$ListBookiesCmd.runCmd(BookieShell.java:1711) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
    at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]

Can you please help us with this, Thank you :)

jiazhai commented 4 years ago

@vishalananth07 sorry, not notice that you are using 3 nodes. For 3 nodes, you should not change allowLoopback to true. How about a try on this? 

allowLoopback=false
advertisedAddress=you_bookie_address
useHostNameAsBookieID=false

Then in a clean deploy, all bookie nodess should be identified by their IP:3181

vishalananth07 commented 4 years ago

@jiazhai Thanks a lot, all the components are starting up and the bookies are listed. I just need one more help, How do you connect to this cluster from external computers which will act as producer and consumer? Should the computers be on the same network? My, web service url: http://10.6.8.14:8080,10.6.8.13:8080,10.6.8.15:8080 broker service url: pulsar://10.6.8.14:6650,10.6.8.13:6650,10.6.8.15:6650

jiazhai commented 4 years ago

@vishalananth07 It should be working on any computer that could telnet these x.x.x.x:6650/8080

vishalananth07 commented 4 years ago

@jiazhai I tried to connect to this cluster from an external system using the pulsar-client tool, I modified the conf/client.conf as follows

webServiceUrl=http://10.6.8.14:8080,10.6.8.13:8080,10.6.8.15:8080
brokerServiceUrl=pulsar://10.6.8.14:6650,10.6.8.13:6650,10.6.8.15:6650

But I am not able to produce or consumer and getting this error

bin/pulsar-client produce   persistent://public/default/test   -n 1   -m "Hello Pulsar"
12:53:28.369 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xc48e6140, L:/10.6.8.16:57588 - R:10.6.8.15/10.6.8.15:6650]] Connected to server
12:53:28.553 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
  "topicName" : "persistent://public/default/test",
  "producerName" : null,
  "sendTimeoutMs" : 30000,
  "blockIfQueueFull" : false,
  "maxPendingMessages" : 1000,
  "maxPendingMessagesAcrossPartitions" : 50000,
  "messageRoutingMode" : "RoundRobinPartition",
  "hashingScheme" : "JavaStringHash",
  "cryptoFailureAction" : "FAIL",
  "batchingMaxPublishDelayMicros" : 1000,
  "batchingMaxMessages" : 1000,
  "batchingEnabled" : true,
  "compressionType" : "NONE",
  "initialSequenceId" : null,
  "autoUpdatePartitions" : true,
  "properties" : { }
}
12:53:28.562 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
  "serviceUrl" : "pulsar://10.6.8.14:6650,10.6.8.13:6650,10.6.8.15:6650",
  "authPluginClassName" : null,
  "authParams" : null,
  "operationTimeoutMs" : 30000,
  "statsIntervalSeconds" : 60,
  "numIoThreads" : 1,
  "numListenerThreads" : 1,
  "connectionsPerBroker" : 1,
  "useTcpNoDelay" : true,
  "useTls" : false,
  "tlsTrustCertsFilePath" : "",
  "tlsAllowInsecureConnection" : false,
  "tlsHostnameVerificationEnable" : false,
  "concurrentLookupRequest" : 5000,
  "maxLookupRequest" : 50000,
  "maxNumberOfRejectedRequestPerConnection" : 50,
  "keepAliveIntervalSeconds" : 30,
  "connectionTimeoutMs" : 10000,
  "requestTimeoutMs" : 60000,
  "defaultBackoffIntervalNanos" : 100000000,
  "maxBackoffIntervalNanos" : 30000000000
}
12:53:28.570 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x1d9c4ae7, L:/10.6.8.16:37992 - R:10.6.8.14/10.6.8.14:6650]] Connected to server
12:53:28.737 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to masterfyp-HP-Compaq-Pro-6305-SFF:6650 : java.net.UnknownHostException: failed to resolve 'masterfyp-HP-Compaq-Pro-6305-SFF' after 2 queries 
12:53:28.738 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [null] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'masterfyp-HP-Compaq-Pro-6305-SFF' after 2 queries 
12:53:28.739 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [null] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'masterfyp-HP-Compaq-Pro-6305-SFF' after 2 queries  -- Will try again in 0.1 s
12:53:28.841 [pulsar-timer-4-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [null] Reconnecting after connection was closed
12:53:28.843 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xa4498898, L:/10.6.8.16:57586 - R:10.6.8.13/10.6.8.13:6650]] Connected to server
12:53:28.873 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to masterfyp-HP-Compaq-Pro-6305-SFF:6650 : java.net.UnknownHostException: failed to resolve 'masterfyp-HP-Compaq-Pro-6305-SFF' after 2 queries 
12:53:28.873 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [null] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'masterfyp-HP-Compaq-Pro-6305-SFF' after 2 queries 
12:53:28.873 [pulsar-client-io-1-1] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [null] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'masterfyp-HP-Compaq-Pro-6305-SFF' after 2 queries  -- Will try again in 0.194 s
12:53:29.068 [pulsar-timer-4-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [null] Reconnecting after connection was closed
jiazhai commented 4 years ago

@vishalananth07 there was also a setting in pulsar broker.conf called advertisedAddress would you please try to set it as your _ip_address? and then reboot the broker. From the error log. seems it is trying to use hostname "masterfyp-HP-Compaq-Pro-6305-SFF".

vishalananth07 commented 4 years ago

@jiazhai Everything is working perfectly now :), Thanks a lot for your patient help and support