Bookie sanity fails - Unable to connect to Bookie after writing to Ledger

[dsambandam]

Expected behavior

Successful Bookie sanity after starting bookies in IPV6

Actual behavior

Bookie sanity fails - Unable to connect to Bookie after writing to Ledger

Steps to reproduce

1. Update the configuration file for Bookies with Zk Server entries
2. Start the bookie ( No major errors reported)
3. Run Bookie sanity
4. Issue : Connection lost to bookie after writing to ledgers

12:47:57.979 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: [id: 0x426035ec, L:/2001:ssss:a06:xxxx:f1:fef:0:cddd%0:4760 - R:2001:ssss:a06:xxxx:f1:fef:0:cddd/2001:ssss:a06:xxxx:f1:fef:0:cddd:3181] 12:47:57.989 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 2001:ssss:a06:xxxx:f1:fef:0:cddd:3181 12:47:58.009 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x426035ec, L:/2001:ssss:a06:xxxx:f1:fef:0:cddd%0:4760 - R:2001:ssss:a06:xxxx:f1:fef:0:cddd/2001:ssss:a06:xxxx:f1:fef:0:cddd:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 12:47:58.078 [main] INFO org.apache.bookkeeper.bookie.BookieShell - Written 10 entries in ledger 7 12:47:58.099 [bookkeeper-io-3-2] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0xf86d5071]/2001:ssss, current state CONNECTING : io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: /0.0.7.209:4888 at io.netty.channel.unix.Errors.throwConnectException(Errors.java:104) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.unix.Socket.connect(Socket.java:257) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:732) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:717) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:559) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1346) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1004) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:259) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.NoRouteToHostException ... 30 more 12:47:58.105 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.ReadLastConfirmedOp - While readLastConfirmed ledger: 7 did not hear success responses from all quorums 12:47:58.112 [main] WARN org.apache.bookkeeper.bookie.BookieShell - Error in bookie sanity test org.apache.bookkeeper.client.BKException$BKLedgerRecoveryException: Error while recovering ledger at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncOpenCallback.openComplete(SyncCallbackUtils.java:143) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:225) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.LedgerOpenOp$1.safeOperationComplete(LedgerOpenOp.java:194) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.LedgerOpenOp$1.safeOperationComplete(LedgerOpenOp.java:1) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.util.OrderedGenericCallback.operationComplete(OrderedGenericCallback.java:55) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.proto.BookkeeperInternalCallbacks$TimedGenericCallback.operationComplete(BookkeeperInternalCallbacks.java:131) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.LedgerRecoveryOp.submitCallback(LedgerRecoveryOp.java:145) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.LedgerRecoveryOp.access$1(LedgerRecoveryOp.java:137) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.LedgerRecoveryOp$1.readLastConfirmedDataComplete(LedgerRecoveryOp.java:124) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:134) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.proto.BookieClient$2.safeRun(BookieClient.java:272) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131] 12:47:58.124 [main] INFO org.apache.bookkeeper.bookie.BookieShell - Deleted ledger 7

System configuration

Pulsar version: 2.0

[sijie]

@dsambandam I think there is a problem on bookkeeper handling ipv6 ip address. I will file an issue on bookkeeper to address that. for getting around on this issue, I would suggest setting useHostNameAsBookieID to true.

https://github.com/apache/incubator-pulsar/blob/master/conf/bookkeeper.conf#L97

[sijie]

@dsambandam same thing might happen to brokers as well. you might consider using advertisedAddress for broker to advertise hostname. https://github.com/apache/incubator-pulsar/blob/master/conf/broker.conf#L44

[dsambandam]

@sijie useHostNameAsBookieID to true (worked fine) but having issues with brokers

14:34:08.223 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x9faf4b3a, L:/2001:dddd:a06:ssss:f1:fef:0:dgdg%0:5317 - R:host1/2001:ssss:a06:xxxx:f1:fef:0:ffff:6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available 14:34:08.224 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available 14:34:08.225 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages 14:34:08.226 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available

[sijie]

@dsambandam "Not enough non-faulty bookie" means there are not enough bookies.

a couple of questions:

• how many bookie are you running?
• can you run bin/bookkeeper listbookies -rw to show me the list of bookies advertised in the cluster?

[dsambandam]

I am running 3 bookies ...how should I run the below command [root@carocnxodapa1v apache-pulsar-2.0.0-rc1-incubating]# ./bin/bookkeeper listbookies -rw JMX enabled by default JAVA_HOME not set, using java from PATH. (/usr/bin/java) Error: Could not find or load main class listbookies

[dsambandam]

Tried ./bin/bookkeeper shell listbookies -rw and it shows up lot of logs ..do you need them ?

[dsambandam]

All the bookies are up . Below is the broker conf update

Zookeeper quorum connection string

zookeeperServers=host8:2181,host9:2181,host10:2181

Configuration Store connection string

configurationStoreServers=host8v:2181,host9:2181,host10:2181 advertisedAddress=host1 (Broker hostname)

[sijie]

Tried ./bin/bookkeeper shell listbookies -rw and it shows up lot of logs ..do you need them ?

not all of them, can you paste the lines that have `3181?

[dsambandam]

Unfortunately nothing on port 3181..Is my configuration above for starting broker correct? I have 3 zookeepers and 3 bookies running ..For some strange reason brokers aren't able to see them

[dsambandam]

Also when useHostNameAsBookieID is set to true and bookie restarted , process fails with the below error.

https://github.com/apache/incubator-pulsar/blob/master/conf/bookkeeper.conf#L97

The only way I could get the sanity passed is when the above setting is false , bookie started and sanity run with bookie conf file updated

18:22:49.968 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 18:22:50.095 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 18:22:50.106 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "host5:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" ] is not matching with [4 bookieHost: "2001:ffff:a06:ssss:f1:fef:0:ssss:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"

[sijie]

okay. so the problem is the bookies didn't start correctly. there is a mechanism in bookkeeper to prevent bookies startup with malformed state which is called cookie. this ensures disk layout and bookie identifier unchanged between different bookie restart cycles.

in this case, since we changed the bookie to use hostname, it changes the bookie identifier. bookkeeper prevents its startup. what I would suggest to do:

on each bookie node, run "bin/bookkeeper shell bookieformat". this would reinitialize your bookie nodes.

[dsambandam]

Should I restart bookie after bookieformat?

[sijie]

@dsambandam yes.

[dsambandam]

19:00:27.594 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 19:00:27.702 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer Fails with the same error

19:00:27.722 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "carocnxodapa4v:3181"

[dsambandam]

Fails with the same error after "bookieformat"

[sijie]

@dsambandam sorry you might need to remove cookie as well.

try bin/bookkeeper shell bookieformat -deleteCookie

[dsambandam]

Bookie doesn't start ..

19:22:42.675 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 19:22:42.702 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "carocnxodapa4v:3181"

[sijie]

@dsambandam do you have more logging around this? also can you share your bookkeeper.conf?

[dsambandam]

Attached logs for your reference

bookkeeper.conf.txt pulsar-bookie-host4.log

[sijie]

@dsambandam I see still the same error. actually can you do following:

1) on the bookie node, stop the bookie process 2) run bin/bookkeeper shell bookieformat -deleteCookie -f 3) check the file by cat data/bookkeeper/journal/current/VERSION and cat data/bookkeeper/storage/current/VERSION. I would like to see what is recorded in the VERSION file. 4) then restart the bookie again.

[dsambandam]

When bookie format is attempted, I can't start back the bookie process again since it fails with the same error report above and data directories are empty.

Working Bookie Node version with useHostNameAsBookieID set to False.

bookieHost: "2001:dddd:a06:dddd:f1:fef:0:ffff:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"

And by default only below directories are created and no storage nodes

[root@host5 bookkeeper]# ls -lrt total 8 drwx------ 3 root root 4096 Jun 12 11:50 journal drwx------ 3 root root 4096 Jun 12 11:50 ledgers

[dsambandam]

Ledger version:

[root@host5 current]# cat VERSION 4 bookieHost: "2001:dddd:a06:dddd:f1:fef:0:ffff::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116"

[sijie]

interesting, so it seems that VERSION file is still using ipv6 ip as the bookie identifier. what is the output when you using bin/bookkeeper shell bookieformat -deleteCookie -f?

[dsambandam]

bin/bookkeeper shell bookieformat -deleteCookie -f JMX enabled by default JAVA_HOME not set, using java from PATH. (/usr/bin/java) Are you sure to format Bookie data..? (Y or N) Y 08:03:42.304 [main] INFO org.apache.bookkeeper.bookie.Bookie - Bookie format completed successfully 08:03:42.328 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 08:03:42.328 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 08:03:42.328 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 08:03:42.347 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+hierarchical://host8:2181;host9:2181;host10:2181/ledgers : zkServers = host8:2181,host9:2181,host10:2181, ledgersRootPath = /ledgers. 08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=Bookiehost 08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_131 08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/java/jdk1.8.0_131/jre 08:03:42.353 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/conf:::/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.beust-jcommander-1.48.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.carrotsearch-hppc-0.7.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.ea.agentloader-ea-agent-loader-1.0.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.core-jackson-annotations-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.core-jackson-core-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.core-jackson-databind-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.datatype-jackson-datatype-joda-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.8.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.github.ben-manes.caffeine-caffeine-2.3.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.api.grpc-proto-google-common-protos-0.1.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.auth-google-auth-library-credentials-0.4.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.code.findbugs-jsr305-3.0.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.code.gson-gson-2.8.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.errorprone-error_prone_annotations-2.0.19.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.guava-guava-20.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.instrumentation-instrumentation-api-0.4.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.protobuf-protobuf-java-2.4.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.google.protobuf-protobuf-java-util-3.3.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-beanutils-commons-beanutils-1.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-beanutils-commons-beanutils-core-1.8.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-cli-commons-cli-1.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-codec-commons-codec-1.10.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-collections-commons-collections-3.2.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-configuration-commons-configuration-1.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-digester-commons-digester-1.8.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-io-commons-io-2.5.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-lang-commons-lang-2.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/commons-logging-commons-logging-1.1.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.squareup.okio-okio-1.6.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.wordnik-swagger-annotations-1.5.3-M1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.yahoo.datasketches-memory-0.8.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-3.10.1.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-all-4.1.21.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-codec-http2-4.1.12.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-codec-socks-4.1.12.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-handler-proxy-4.1.12.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.netty-netty-tcnative-boringssl-static-2.0.7.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient_common-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient_hotspot-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.prometheus-simpleclient_servlet-0.0.23.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.swagger-swagger-annotations-1.5.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.swagger-swagger-core-1.5.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/io.swagger-swagger-models-1.5.3.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.annotation-javax.annotation-api-1.2.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.validation-validation-api-1.1.0.Final.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.websocket-javax.websocket-api-1.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/jline-jline-0.9.94.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/joda-time-joda-time-2.7.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/log4j-log4j-1.2.17.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/net.java.dev.jna-jna-4.2.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/net.jpountz.lz4-lz4-1.3.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper-circe-checksum-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper.http-http-server-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.commons-commons-collections4-4.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.commons-commons-lang3-3.4.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.distributedlog-distributedlog-core-shaded-4.7.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-managed-ledger-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-broker-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-broker-common-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-client-admin-original-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-client-original-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-client-tools-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-common-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-connect-core-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-discovery-service-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-api-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-api-examples-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-proto-shaded-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-runtime-all-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-runtime-shaded-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-functions-worker-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-proxy-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-testclient-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-websocket-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-zookeeper-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.0.0-rc1-incubating.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.aspectj-aspectjrt-1.8.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.aspectj-aspectjweaver-1.8.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.asynchttpclient-async-http-client-2.1.0-alpha26.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.asynchttpclient-async-http-client-netty-utils-2.1.0-alpha26.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.bouncycastle-bcpkix-jdk15on-1.55.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.bouncycastle-bcprov-jdk15on-1.55.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-client-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-continuation-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-http-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-proxy-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-security-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-servlets-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty-jetty-util-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-api-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-client-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-common-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-server-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.eclipse.jetty.websocket-websocket-servlet-9.3.11.v20160721.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-hk2-api-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-hk2-locator-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-hk2-utils-2.5.0-b30.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.bundles.repackaged-jersey-guava-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.core-jersey-client-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.core-jersey-common-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.core-jersey-server-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.glassfish.jersey.media-jersey-media-multipart-2.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.javassist-javassist-3.21.0-GA.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.projectlombok-lombok-1.16.20.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.reactivestreams-reactive-streams-1.0.0.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.rocksdb-rocksdbjni-5.8.6.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.slf4j-slf4j-api-1.7.25.jar:/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating/lib/org.yaml-snakeyaml-1.15.jar: 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-693.21.1.el7.x86_64 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 08:03:42.356 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/VMB/vmb_install/apache-pulsar-2.0.0-rc1-incubating 08:03:42.360 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host8:2181,host9:2181,host10:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@4f5991f6 08:03:42.393 [main-SendThread(host8:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server host8/2001:dddd:a06:ssss:f1:fef:0:xxxx:2181. Will not attempt to authenticate using SASL (unknown error) 08:03:42.400 [main-SendThread(host8:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to host8/2001:dddd:a06:ssss:f1:fef:0:xxxx:2181, initiating session 08:03:42.409 [main-SendThread(host8:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server host8/2001:dddd:a06:ssss:f1:fef:0:xxxx:2181, sessionid = 0x163f5d957ec001d, negotiated timeout = 30000 08:03:42.423 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 08:03:42.447 [main] WARN org.apache.bookkeeper.bookie.BookieShell - No cookie to remove : org.apache.bookkeeper.bookie.BookieException$CookieNotFoundException: Bookiehost:3181 at org.apache.bookkeeper.discover.ZKRegistrationManager.readCookie(ZKRegistrationManager.java:339) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.Cookie.readFromRegistrationManager(Cookie.java:332) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.Cookie.readFromRegistrationManager(Cookie.java:316) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.BookieShell$BookieFormatCmd.lambda$0(BookieShell.java:423) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.meta.MetadataDrivers.lambda$1(MetadataDrivers.java:363) ~[org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:338) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:363) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.BookieShell$BookieFormatCmd.runCmd(BookieShell.java:421) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:232) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2816) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2907) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0.jar:4.7.0] 08:03:42.453 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x163f5d957ec001d closed 08:03:42.453 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x163f5d957ec001d

[dsambandam]

Version file still uses IPV6 since bookie never started successfully after modifying conf file

[sijie]

No cookie to remove : org.apache.bookkeeper.bookie.BookieException$CookieNotFoundException: Bookiehost:3181 at

so the bookieformat didn't succeed. I think the problem might be failing on handling the ipv6 ip address.

so I would suggest doing following:

1) stop all your bookies. 2) on one bookie node, run 'bin/bookkeeper shell metaformat -f', this will clean up the bad state in metadata. run this only once. 3) on each bookie, update bookkeeper.conf to set advertisedAddress to the hostname and then run bin/bookkeeper shell bookieformat -f, then start the bookie.

[dsambandam]

Bookies came up fine after above changes and I see hostname configured for bookie host.

Do you recommend any changes on Brokers conf other than advertisedAddress before I restart the same ?

[dsambandam]

Brokers seems to be working now and able to discover the bookies configured

13:01:31.329 [bookkeeper-io-12-11] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: host5:3181 13:01:31.332 [bookkeeper-io-12-17] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: host4:3181

[sijie]

@dsambandam yeah broker just needs to set advertisedAddress. bookies are tricky since it has a lot of defense mechanisms to prevent things go wrong, since it is a storage system.

Glad to hear that it is working for you now!

[dsambandam]

Thanks for your help..I shall keep you posted for any further questions.

[sijie]

great! feel free to ping us via github issues, mailing list or slack channels. Glad we helped you.

[vishnuvardhanp4]

after replacing lib and started apache pulsar facing below error ERROR org.apache.bookkeeper.bookie.Bookie - Cookie for this bookie is not stored in metadata store. Bookie failing to come up INSTANCEID not exists in zookeeper. Not considering it for data verification ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.

Using it in production.data should not be loose .is there any other scenario to solve this issue .please help me out of this Why the error is showing