Geo-replication setup on K8s cluster

I am trying to setup geo-replication using bookkeeper with global zk established, test and my first create zk node is getting replication on the zk servers on the 2 different site clusters.

Now i am setting up bookkeeper (statefulset) using the below config paramenters , but my bookie servers are crashlooping, see below my parameters:

BK_zkServers: zookeeper-1.abcdefg.com,zookeeper-2.abcdefg.com,zookeeper-3.abcdefg.com,zookeeper-4.abcdefg.com,zookeeper-5.abcdefg.com,zookeeper-6.abcdefg.com

where the zk are all my DNS records.

Error on the bookie server: 2019-05-02 12:35:29,117 - ERROR - [main:Main@223] - Failed to build bookie server java.lang.IllegalArgumentException: Path length must be > 0 at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1090) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2301(ZooKeeperClient.java:70) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:830) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:824) at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:824) at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:378) at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:408) at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:691) at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:106) at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:43) at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) at org.apache.bookkeeper.server.Main.doMain(Main.java:221) at org.apache.bookkeeper.server.Main.main(Main.java:203)

Any ideas how should the format to specify the "BK_zkServers" parameter be ?

Thanks for the support , appreciate if someone has solved this before.

@maurya-m Can you try using ";" instead of ","? I think when "," is used in the bookie service uri, java URI ignores the string after ",", hence the ledger path becomes empty.

BK_zkServers: zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com

Btw, if you are using kubernetes and have setup a service for zookeeper, I think you can just the zookeeper service name without specifying all the zookeeper instances.

That's the same solution I was writing :)

@sijie thanks so much for the quick response:)

I corrected the ',' with ';' now the bookie servers are ready and running, but i am getting this warning message , any hints to resolve this too :

2019-05-03 07:47:55,713 - WARN - [main-SendThread(zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181):ClientCnxn$SendThread@1168] - Session 0x0 for server zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181, unexpected error, closing socket connection and attempting reconnect java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddress(Net.java:101) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064) Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for / at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1569) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:732) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:600) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:363) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:291)

@sijie @eolivelli , thanks for the 'zookeeper' service option to mention, i was playing with it earlier , was working for me , but now trying to test replication across clusters , and pointer on quick test for this usecase pls?

update: the recovery bookie pods are failing now on this , any ideas?

K8s Deployment spec:

apiVersion: apps/v1beta1 kind: Deployment metadata: name: bookie-autorecovery spec: replicas: 2 template: metadata: labels: app: bookkeeper component: bookkeeper-replication spec: containers:

name: replication-worker image: apache/bookkeeper:latest command: [ "/bin/bash", "/opt/bookkeeper/scripts/entrypoint.sh" ] args: ["/opt/bookkeeper/bin/bookkeeper", "autorecovery"] envFrom:
- configMapRef: name: bookie-config

2019-05-03 07:57:22,107 - ERROR - [main:BookieWatcherImpl@70] - Failed to get bookie list : org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$0(ZKRegistrationClient.java:212) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1174) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:604) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) 2019-05-03 07:57:22,109 - ERROR - [main:AutoRecoveryMain@327] - Failed to build AutoRecovery Server java.io.IOException: Failed to create bookkeeper client at org.apache.bookkeeper.replication.Auditor.createBookKeeperClient(Auditor.java:176) at org.apache.bookkeeper.replication.Auditor.createBookKeeperClient(Auditor.java:166) at org.apache.bookkeeper.replication.AutoRecoveryMain.(AutoRecoveryMain.java:94) at org.apache.bookkeeper.server.service.AutoRecoveryService.(AutoRecoveryService.java:42) at org.apache.bookkeeper.replication.AutoRecoveryMain.buildAutoRecoveryServer(AutoRecoveryMain.java:357) at org.apache.bookkeeper.replication.AutoRecoveryMain.doMain(AutoRecoveryMain.java:325) at org.apache.bookkeeper.replication.AutoRecoveryMain.main(AutoRecoveryMain.java:307) Caused by: org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$0(ZKRegistrationClient.java:212) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1174) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:604) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)

I'm working with Maurya on this. If we use zookeper k8s service as value for BK_zkServers, it will reference only zookeeper servers on local cluster. Based on the uncomplete article (and the only available I've found) https://bookkeeper.apache.org/docs/latest/admin/geo-replication/ it's not clear how to set it up. We have global zookeeper which consist of servers distributed across multiple data centers. Zookeeper itself works across multiple data centers. The challenge is how to achieve bookkeper replication across those multiple data centers. Any clues?

Bookie startup log:

Environment Vars for bookie:

  [metadata service]
  BK_zkServers is zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com
  BK_CLUSTER_ROOT_PATH is 
  BK_metadataServiceUri is zk://zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com/ledgers

  [bookie]
  BK_bookiePort bookie service port is 3181
  BK_DATA_DIR is /data/bookkeeper
  BK_journalDirectory is /bookkeeper/data/journal
  BK_ledgerDirectories are /bookkeeper/data/ledgers
  BK_indexDirectories are /bookkeeper/data/ledgers

  [bookie http]
  BK_httpServerEnabled is true
  BK_httpServerPort is 

  [dlog]
  BK_dlogRootPath is /distributedlog

  [stream storage]
  BK_STREAM_STORAGE_ROOT_PATH is /stream
  BK_NUM_STORAGE_CONTAINERS is 32
  BOOKIE_GRPC_PORT is 
conf files: 
['/opt/bookkeeper/conf/zookeeper.conf', '/opt/bookkeeper/conf/bkenv.sh', '/opt/bookkeeper/conf/zookeeper.conf.dynamic', '/opt/bookkeeper/conf/bk_server.conf', '/opt/bookkeeper/conf/nettyenv.sh', '/opt/bookkeeper/conf/log4j.properties', '/opt/bookkeeper/conf/log4j.shell.properties', '/opt/bookkeeper/conf/standalone.conf', '/opt/bookkeeper/conf/log4j.cli.properties', '/opt/bookkeeper/conf/bk_cli_env.sh', '/opt/bookkeeper/conf/jaas_example.conf']
[/opt/bookkeeper/conf/bkenv.sh] Applying config BOOKIE_EXTRA_OPTS = "-Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -XX:+UseG1GC  -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB"
[/opt/bookkeeper/conf/bk_server.conf] Applying config bookiePort = 3181
[/opt/bookkeeper/conf/bk_server.conf] Applying config httpServerEnabled = true
[/opt/bookkeeper/conf/bk_server.conf] Applying config httpServerPort = 
[/opt/bookkeeper/conf/bk_server.conf] Applying config indexDirectories = /bookkeeper/data/ledgers
[/opt/bookkeeper/conf/bk_server.conf] Applying config journalDirectory = /bookkeeper/data/journal
[/opt/bookkeeper/conf/bk_server.conf] Applying config ledgerDirectories = /bookkeeper/data/ledgers
[/opt/bookkeeper/conf/bk_server.conf] Applying config metadataServiceUri = zk://zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com/ledgers
[/opt/bookkeeper/conf/bk_server.conf] Applying config useHostNameAsBookieID = true
[/opt/bookkeeper/conf/bk_server.conf] Applying config zkLedgersRootPath = /ledgers
[/opt/bookkeeper/conf/bk_server.conf] Applying config zkServers = zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com
Created bookie dirs : 
  journal = /bookkeeper/data/journal
  ledger = /bookkeeper/data/ledgers
  index = /bookkeeper/data/ledgers
wait for zookeeper
Connecting to zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com
2019-05-03 07:47:33,088 - INFO  - [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
2019-05-03 07:47:33,091 - INFO  - [main:Environment@100] - Client environment:host.name=k8s-agentpool1-24779565-0
2019-05-03 07:47:33,091 - INFO  - [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-05-03 07:47:33,093 - INFO  - [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-05-03 07:47:33,093 - INFO  - [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:java.class.path=/opt/bookkeeper/conf:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-server-4.9.0.jar::/opt/bookkeeper/lib/com.beust-jcommander-1.48.jar:/opt/bookkeeper/lib/com.fasterxml.jackson.core-jackson-annotations-2.9.7.jar:/opt/bookkeeper/lib/com.fasterxml.jackson.core-jackson-core-2.9.7.jar:/opt/bookkeeper/lib/com.fasterxml.jackson.core-jackson-databind-2.9.7.jar:/opt/bookkeeper/lib/com.google.api.grpc-proto-google-common-protos-1.0.0.jar:/opt/bookkeeper/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/opt/bookkeeper/lib/com.google.code.gson-gson-2.7.jar:/opt/bookkeeper/lib/com.google.errorprone-error_prone_annotations-2.1.2.jar:/opt/bookkeeper/lib/com.google.guava-guava-21.0.jar:/opt/bookkeeper/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/opt/bookkeeper/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/opt/bookkeeper/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/opt/bookkeeper/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/opt/bookkeeper/lib/com.squareup.okio-okio-1.13.0.jar:/opt/bookkeeper/lib/com.yahoo.datasketches-memory-0.8.3.jar:/opt/bookkeeper/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/opt/bookkeeper/lib/commons-cli-commons-cli-1.2.jar:/opt/bookkeeper/lib/commons-codec-commons-codec-1.6.jar:/opt/bookkeeper/lib/commons-configuration-commons-configuration-1.10.jar:/opt/bookkeeper/lib/commons-io-commons-io-2.4.jar:/opt/bookkeeper/lib/commons-lang-commons-lang-2.6.jar:/opt/bookkeeper/lib/commons-logging-commons-logging-1.1.1.jar:/opt/bookkeeper/lib/io.grpc-grpc-all-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-auth-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-context-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-core-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-netty-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-okhttp-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-protobuf-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-protobuf-lite-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-protobuf-nano-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-stub-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-testing-1.12.0.jar:/opt/bookkeeper/lib/io.netty-netty-buffer-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-dns-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-http-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-http2-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-socks-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-common-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-handler-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-handler-proxy-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-resolver-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-resolver-dns-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-tcnative-boringssl-static-2.0.19.Final.jar:/opt/bookkeeper/lib/io.netty-netty-transport-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-transport-native-epoll-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-transport-native-unix-common-4.1.31.Final.jar:/opt/bookkeeper/lib/io.opencensus-opencensus-api-0.11.0.jar:/opt/bookkeeper/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.11.0.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient-0.0.21.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient_common-0.0.21.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient_hotspot-0.0.21.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient_servlet-0.0.21.jar:/opt/bookkeeper/lib/io.vertx-vertx-auth-common-3.4.1.jar:/opt/bookkeeper/lib/io.vertx-vertx-core-3.4.1.jar:/opt/bookkeeper/lib/io.vertx-vertx-web-3.4.1.jar:/opt/bookkeeper/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/opt/bookkeeper/lib/jline-jline-2.11.jar:/opt/bookkeeper/lib/log4j-log4j-1.2.17.jar:/opt/bookkeeper/lib/net.java.dev.jna-jna-3.2.7.jar:/opt/bookkeeper/lib/net.jpountz.lz4-lz4-1.3.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-common-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-proto-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-tools-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-tools-ledger-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-circe-checksum-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-cpu-affinity-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-statelib-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-cli-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-java-client-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-service-api-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-service-impl-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.http-http-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.http-vertx-http-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.9.0.jar:/opt/bookkeeper/lib/org.apache.commons-commons-collections4-4.1.jar:/opt/bookkeeper/lib/org.apache.commons-commons-lang3-3.6.jar:/opt/bookkeeper/lib/org.apache.curator-curator-client-4.0.1.jar:/opt/bookkeeper/lib/org.apache.curator-curator-framework-4.0.1.jar:/opt/bookkeeper/lib/org.apache.curator-curator-recipes-4.0.1.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-common-4.9.0.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-core-4.9.0-tests.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-core-4.9.0.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-protocol-4.9.0.jar:/opt/bookkeeper/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/opt/bookkeeper/lib/org.apache.zookeeper-zookeeper-3.4.13.jar:/opt/bookkeeper/lib/org.codehaus.jackson-jackson-core-asl-1.9.11.jar:/opt/bookkeeper/lib/org.codehaus.jackson-jackson-mapper-asl-1.9.11.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-http-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-io-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-security-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-server-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-servlet-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-util-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.inferred-freebuilder-1.14.9.jar:/opt/bookkeeper/lib/org.jctools-jctools-core-2.1.2.jar:/opt/bookkeeper/lib/org.rocksdb-rocksdbjni-5.13.1.jar:/opt/bookkeeper/lib/org.slf4j-slf4j-api-1.7.25.jar:/opt/bookkeeper/lib/org.slf4j-slf4j-log4j12-1.7.25.jar:
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:java.compiler=<NA>
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:os.name=Linux
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:os.arch=amd64
2019-05-03 07:47:33,094 - INFO  - [main:Environment@100] - Client environment:os.version=4.15.0-1041-azure
2019-05-03 07:47:33,095 - INFO  - [main:Environment@100] - Client environment:user.name=root
2019-05-03 07:47:33,095 - INFO  - [main:Environment@100] - Client environment:user.home=/root
2019-05-03 07:47:33,095 - INFO  - [main:Environment@100] - Client environment:user.dir=/opt/bookkeeper
2019-05-03 07:47:33,096 - INFO  - [main:ZooKeeper@442] - Initiating client connection, connectString=zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@782830e
2019-05-03 07:47:33,171 - INFO  - [main-SendThread(zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181. Will not attempt to authenticate using SASL (unknown error)
2019-05-03 07:47:33,234 - WARN  - [main-SendThread(zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181):ClientCnxn$SendThread@1168] - Session 0x0 for server zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181, unexpected error, closing socket connection and attempting reconnect
java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Net.java:101)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541)
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1569)
    at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:732)
    at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:600)
    at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:363)
    at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:291)
Connecting to zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com

@maurya-m @rolish Ah I see. It seems that somehow the autorecovery doesn't parse the ; separated string correctly.

Can you try to use BK_metadataServiceUri to zk://zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com/ledgers, instead of using BK_zkServers?

@maurya-m one more thing (just out of curiosity) - when you are running on k8s, how do you expose the bookie pods to other pods in remote data center? For a geo-replicated bookkeeper cluster, you need AutoRecovery process be able to access all the bookie pods.

@sijie, thanks for this, very useful. Using BK_metadataServiceUri instead of BK_zkServers (which is completely excluded from configmap) works both in case bookies and bookie-autorecovery. BK successfully connects to all global ZK nodes distributed across two data centers. We are using statefulsets with hostNetwork: true for both ZK and BK nodes. Both Azure data centers are interconnected by VPN. Bookie hosts are identified by host IP (BK_useHostNameAsBookieID: "false"). Statefulset has defined anti affinity so no two bookies can run on single node (which would cause bookie ID conflict).

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "component"
                    operator: In
                    values:
                    - bookie
              topologyKey: "kubernetes.io/hostname"

This is however not ideal, as each pod in statefulset has it's own PVC attached and when killed it can start on another node attaching the same PVC. This will result in exception

2019-05-05 18:34:56,501 - ERROR - [main:Main@223] - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
bookieHost: "172.16.189.66:3181"
journalDir: "/bookkeeper/data/journal"
ledgerDirs: "1\t/bookkeeper/data/ledgers"
instanceId: "038629cb-93b8-46ad-a469-5b9bb070dcb8"
] is not matching with [4
bookieHost: "172.16.189.35:3181"
journalDir: "/bookkeeper/data/journal"
ledgerDirs: "1\t/bookkeeper/data/ledgers"
instanceId: "038629cb-93b8-46ad-a469-5b9bb070dcb8"
]
    at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:136)

It looks like there is no other option than to use daemonset using host attached disks for bookies.

@rolish sorry for really late response. for statefulset deployments, you should configure useHostName to advertise bookie hostname as the bookie id rather than ip. because ip is not a stable identifier for statefulset pods.

apache / bookkeeper

Geo-replication setup on K8s cluster #2080

K8s Deployment spec: