search

banzaicloud / koperator

Oh no! Yet another Apache Kafka operator for Kubernetes
Apache License 2.0
785 stars 197 forks source link

Can't Install a multinode cluster with Istio installed #219

Closed sylvainOL closed 4 years ago

sylvainOL commented 4 years ago

Hello,

Describe the bug I've deployed a kubernetes cluster with istio. When trying to deploy a 3 nodes ZooKeeper Cluster for Kafka, the second one can't start because of immediate closed connections.

As banzai cloud has made a blog post on Kafka (+ZK?) on Istio (https://banzaicloud.com/blog/kafka-on-istio-performance/), and they propose to use your operator (https://github.com/banzaicloud/kafka-operator), I assumed it's possible but don't see how :-/

I've also opened an issue on zookeeper-operator (https://github.com/pravega/zookeeper-operator/issues/102), but as it's said to use it in readme, I thought you may have the answer...

Steps to reproduce the issue:

curl -L https://istio.io/downloadIstio | sh -
cd istio-1.4.0
export PATH=$PWD/bin:$PATH
istioctl manifest apply --set profile=demo
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/
helm repo update
kubectl create ns zookeeper
kubectl label namespace zookeeper istio-injection=enabled
helm install --name zookeeper-operator --namespace=zookeeper banzaicloud-stable/zookeeper-operator
kubectl apply -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: onap
  namespace: zookeeper
spec:
  replicas: 3
  persistence:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 3G
    storageClassName: ssd
EOF

Expected behavior

Well a running ZK cluster ;)

Additional context

here's the logs:

+ source /conf/env.sh
++ DOMAIN=onap-headless.zookeeper.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=onap-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=onap-0
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ onap-0 =~ (.*)-([0-9]+)$ ]]
+ NAME=onap
+ ORD=0
+ MYID=1
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
++ cat /data/myid
+ EXISTING_ID=1
+ [[ 1 == \1 ]]
+ WRITE_CONFIGURATION=false
+ set +e
+ nslookup onap-headless.zookeeper.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'onap-headless.zookeeper.svc.cluster.local': Name does not resolve
+ [[ 1 -eq 1 ]]
+ set -e
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=false
+ [[ true == true ]]
+ echo 'Writing myid: 1 to: /data/myid.'
+ echo 1
Writing myid: 1 to: /data/myid.
+ [[ 1 -eq 1 ]]
+ [[ false == false ]]
+ ROLE=participant
+ echo Initial initialization of ordinal 0 pod, creating new config.
Initial initialization of ordinal 0 pod, creating new config.
++ zkConfig
++ echo 'onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ ZKCONFIG='onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ echo Writing bootstrap configuration with the following config:
Writing bootstrap configuration with the following config:
onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181
+ echo 'onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ echo 1
+ echo 'server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
Starting zookeeper service
+ echo 'Starting zookeeper service'
+ zkServer.sh start-foreground
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2019-11-27 15:22:48,324 [myid:] - INFO  [main:QuorumPeerConfig@130] - Reading configuration from: /conf/zoo.cfg
2019-11-27 15:22:48,329 [myid:] - INFO  [main:QuorumPeerConfig@366] - clientPort is not set
2019-11-27 15:22:48,329 [myid:] - INFO  [main:QuorumPeerConfig@380] - secureClientPort is not set
2019-11-27 15:22:48,335 [myid:] - WARN  [main:QuorumPeerConfig@641] - No server failure will be tolerated. You need at least 3 servers.
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-11-27 15:22:48,340 [myid:1] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-11-27 15:22:48,351 [myid:1] - INFO  [main:QuorumPeerMain@141] - Starting quorum peer
2019-11-27 15:22:48,361 [myid:1] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-11-27 15:22:48,364 [myid:1] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
2019-11-27 15:22:48,369 [myid:1] - INFO  [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2019-11-27 15:22:48,399 [myid:1] - INFO  [main:Log@186] - Logging initialized @527ms
2019-11-27 15:22:48,460 [myid:1] - WARN  [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@1d057a39{/,null,null} contextPath ends with /*
2019-11-27 15:22:48,460 [myid:1] - WARN  [main:ContextHandler@1350] - Empty contextPath
2019-11-27 15:22:48,481 [myid:1] - INFO  [main:QuorumPeer@1430] - Local sessions disabled
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1441] - Local session upgrading disabled
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1408] - tickTime set to 2000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1452] - minSessionTimeout set to 4000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1463] - maxSessionTimeout set to 40000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1478] - initLimit set to 10
2019-11-27 15:22:48,496 [myid:1] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-11-27 15:22:48,498 [myid:1] - INFO  [main:QuorumPeer@2036] - QuorumPeer communication is not secured!
2019-11-27 15:22:48,498 [myid:1] - INFO  [main:QuorumPeer@2065] - quorum.cnxn.threads.size set to 20
2019-11-27 15:22:48,505 [myid:1] - INFO  [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.100000111
2019-11-27 15:22:48,532 [myid:1] - INFO  [main:Server@327] - jetty-9.2.18.v20160721
2019-11-27 15:22:48,561 [myid:1] - INFO  [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@1d057a39{/,null,AVAILABLE}
2019-11-27 15:22:48,566 [myid:1] - INFO  [main:AbstractConnector@266] - Started ServerConnector@71bbf57e{HTTP/1.1}{0.0.0.0:8080}
2019-11-27 15:22:48,567 [myid:1] - INFO  [main:Server@379] - Started @698ms
2019-11-27 15:22:48,567 [myid:1] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-11-27 15:22:48,571 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888
2019-11-27 15:22:48,580 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1136] - LOOKING
2019-11-27 15:22:48,581 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@893] - New election. My id =  1, proposed zxid=0x20000001e
2019-11-27 15:22:48,584 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@687] - Notification: 2 (message format version), 1 (n.leader), 0x20000001e (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-11-27 15:22:48,787 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2019-11-27 15:22:48,789 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1224] - LEADING
2019-11-27 15:22:48,795 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@67] - TCP NoDelay set to: true
2019-11-27 15:22:48,797 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@87] - zookeeper.leader.maxConcurrentSnapshots = 10
2019-11-27 15:22:48,797 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@89] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.4-beta-7f51e5b68cf2f80176ff944a9ebd2abbc65e7327, built on 05/11/2018 16:27 GMT
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:host.name=onap-0.onap-headless.zookeeper.svc.cluster.local
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_181
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/zookeeper-3.5.4-beta/bin/../build/classes:/zookeeper-3.5.4-beta/bin/../build/lib/*.jar:/zookeeper-3.5.4-beta/bin/../lib/slf4j-log4j12-1.7.25.jar:/zookeeper-3.5.4-beta/bin/../lib/slf4j-api-1.7.25.jar:/zookeeper-3.5.4-beta/bin/../lib/netty-3.10.6.Final.jar:/zookeeper-3.5.4-beta/bin/../lib/log4j-1.2.17.jar:/zookeeper-3.5.4-beta/bin/../lib/jline-2.11.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-util-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/zookeeper-3.5.4-beta/bin/../lib/jackson-mapper-asl-1.9.11.jar:/zookeeper-3.5.4-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/zookeeper-3.5.4-beta/bin/../lib/commons-cli-1.2.jar:/zookeeper-3.5.4-beta/bin/../lib/audience-annotations-0.5.0.jar:/zookeeper-3.5.4-beta/bin/../zookeeper-3.5.4-beta.jar:/zookeeper-3.5.4-beta/bin/../src/java/lib/*.jar:/conf:
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmp
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.name=Linux
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.version=4.9.0-8-amd64
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.name=root
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/root
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/zookeeper-3.5.4-beta
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=444MB
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=966MB
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=487MB
2019-11-27 15:22:48,816 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@928] - minSessionTimeout set to 4000
2019-11-27 15:22:48,816 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@937] - maxSessionTimeout set to 40000
2019-11-27 15:22:48,817 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@160] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /data/version-2 snapdir /data/version-2
2019-11-27 15:22:48,818 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@432] - LEADING - LEADER ELECTION TOOK - 30 MS
2019-11-27 15:22:48,820 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FileTxnSnapLog@367] - Snapshotting: 0x20000001e to /data/version-2/snapshot.20000001e
2019-11-27 15:22:48,831 [myid:1] - ERROR [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-11-27 15:22:48,836 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@1264] - Have quorum of supporters, sids: [ [1],[1] ]; starting up and setting last processed zxid: 0x300000000
2019-11-27 15:22:48,836 [myid:1] - ERROR [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1596] - Error closing file:
2019-11-27 15:22:48,847 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@255] - Configuring CommitProcessor with 1 worker threads.
2019-11-27 15:22:48,856 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@99] - zookeeper.skipACL=="yes", ACL checks will be skipped
2019-11-27 15:22:48,858 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2019-11-27 15:22:59,878 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40039
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[stat, envi, conf, ruok, mntr, srvr, crst, cons]]
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40039
2019-11-27 15:22:59,899 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40039 (no session established for client)
2019-11-27 15:23:00,535 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44549
2019-11-27 15:23:00,537 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44549
2019-11-27 15:23:00,539 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44549 (no session established for client)
2019-11-27 15:23:09,895 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42815
2019-11-27 15:23:09,897 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:42815
2019-11-27 15:23:09,898 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:42815 (no session established for client)
2019-11-27 15:23:10,549 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35831
2019-11-27 15:23:10,551 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35831
2019-11-27 15:23:10,551 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35831 (no session established for client)
2019-11-27 15:23:19,893 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45287
2019-11-27 15:23:19,896 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45287
2019-11-27 15:23:19,897 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45287 (no session established for client)
2019-11-27 15:23:20,517 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43115
2019-11-27 15:23:20,519 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43115
2019-11-27 15:23:20,520 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43115 (no session established for client)
2019-11-27 15:23:23,856 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33672
2019-11-27 15:23:23,862 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33672
2019-11-27 15:23:23,864 [myid:1] - INFO  [SyncThread:1:FileTxnLog@201] - Creating new log file: log.300000001
2019-11-27 15:23:23,877 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090000 with negotiated timeout 4000 for client /127.0.0.1:33672
2019-11-27 15:23:24,224 [myid:1] - WARN  [NIOWorkerThread-1:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x1000bc750090000, likely client has closed socket
2019-11-27 15:23:24,225 [myid:1] - INFO  [NIOWorkerThread-1:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=127.0.0.1,name5=0x1000bc750090000]
2019-11-27 15:23:24,226 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:33672 which had sessionid 0x1000bc750090000
2019-11-27 15:23:24,613 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33678
2019-11-27 15:23:24,615 [myid:1] - INFO  [NIOWorkerThread-2:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33678
2019-11-27 15:23:24,620 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090001 with negotiated timeout 4000 for client /127.0.0.1:33678
2019-11-27 15:23:24,635 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@432] - skipACL is set, reconfig operation will skip ACL checks!
2019-11-27 15:23:24,635 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@474] - Incremental reconfig
2019-11-27 15:23:24,639 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::QuorumPeer$QuorumServer@173] - Failed to resolve address: onap-1.onap-headless.zookeeper.svc.cluster.local
java.net.UnknownHostException: onap-1.onap-headless.zookeeper.svc.cluster.local
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at java.net.InetAddress.getByName(InetAddress.java:1076)
    at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:171)
    at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:727)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:682)
    at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1519)
    at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1556)
    at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1074)
    at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
    at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:906)
    at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:145)
2019-11-27 15:23:24,644 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::QuorumCnxManager@660] - Cannot open channel to 2 at election address onap-1.onap-headless.zookeeper.svc.cluster.local:3888
java.net.UnknownHostException: onap-1.onap-headless.zookeeper.svc.cluster.local
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:638)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:695)
    at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1519)
    at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1556)
    at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1074)
    at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
    at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:906)
    at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:145)
2019-11-27 15:23:24,645 [myid:1] - ERROR [ProcessThread(sid:1 cport:-1)::QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-11-27 15:23:24,647 [myid:1] - ERROR [SyncThread:1:QuorumPeer@1596] - Error closing file:
2019-11-27 15:23:24,649 [myid:1] - WARN  [SyncThread:1:QuorumPeer@1527] - Restarting Leader Election
2019-11-27 15:23:24,650 [myid:1] - INFO  [onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888:QuorumCnxManager$Listener@892] - Leaving listener
2019-11-27 15:23:24,651 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888
2019-11-27 15:23:25,005 [myid:1] - WARN  [NIOWorkerThread-2:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x1000bc750090001, likely client has closed socket
2019-11-27 15:23:25,006 [myid:1] - INFO  [NIOWorkerThread-2:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=127.0.0.1,name5=0x1000bc750090001]
2019-11-27 15:23:25,007 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:33678 which had sessionid 0x1000bc750090001
2019-11-27 15:23:27,586 [myid:1] - INFO  [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@469] - WorkerSender is down
2019-11-27 15:23:27,588 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@439] - WorkerReceiver is down
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:ZooKeeperServer@392] - Expiring session 0x1000bc750090000, timeout of 4000ms exceeded
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1000bc750090000
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:ZooKeeperServer@392] - Expiring session 0x1000bc750090001, timeout of 4000ms exceeded
2019-11-27 15:23:29,312 [myid:1] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1000bc750090001
2019-11-27 15:23:29,312 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x1000bc750090000
2019-11-27 15:23:29,312 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x1000bc750090001
2019-11-27 15:23:29,885 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45507
2019-11-27 15:23:29,887 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45507
2019-11-27 15:23:29,888 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45507 (no session established for client)
2019-11-27 15:23:30,528 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35905
2019-11-27 15:23:30,529 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35905
2019-11-27 15:23:30,530 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35905 (no session established for client)
2019-11-27 15:23:39,773 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33802
2019-11-27 15:23:39,775 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33802
2019-11-27 15:23:39,780 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090002 with negotiated timeout 4000 for client /127.0.0.1:33802
2019-11-27 15:23:39,882 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39623
...
+ source /conf/env.sh
++ DOMAIN=onap-headless.zookeeper.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=onap-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=onap-1
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ onap-1 =~ (.*)-([0-9]+)$ ]]
+ NAME=onap
+ ORD=1
+ MYID=2
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
++ cat /data/myid
+ EXISTING_ID=2
+ [[ 2 == \2 ]]
+ WRITE_CONFIGURATION=false
+ set +e
+ nslookup onap-headless.zookeeper.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

Name:      onap-headless.zookeeper.svc.cluster.local
Address 1: 10.233.67.152 10-233-67-152.onap-client.zookeeper.svc.cluster.local
+ [[ 0 -eq 1 ]]
+ set -e
+ set +e
++ zkConnectionString
++ set +e
++ nslookup onap-client
++ [[ 0 -eq 1 ]]
++ set -e
++ echo onap-client:2181
+ ZKURL=onap-client:2181
+ set -e
++ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar get-all onap-client:2181
Connecting to Zookeeper onap-client:2181
+ CONFIG='server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
version=300000000'
++ echo 'server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
version=300000000'
++ grep server.2=
++ wc -l
+ REGISTERED=0
+ [[ 0 -eq 1 ]]
+ REGISTER_NODE=true
+ WRITE_CONFIGURATION=true
Writing myid: 2 to: /data/myid.
+ [[ true == true ]]
+ echo 'Writing myid: 2 to: /data/myid.'
+ echo 2
+ [[ 2 -eq 1 ]]
+ [[ 2 -ne 1 ]]
+ [[ true == false ]]
+ [[ true == true ]]
+ ROLE=observer
++ zkConfig
++ echo 'onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181'
+ ZKCONFIG='onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181'
+ echo Registering node and writing local configuration to disk.
+ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar add onap-client:2181 2 'onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181' /data/zoo.cfg.dynamic
Registering node and writing local configuration to disk.
Connecting to Zookeeper onap-client:2181
+ echo 'Starting zookeeper service'
+ zkServer.sh start-foreground
Starting zookeeper service
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2019-11-27 15:23:25,496 [myid:] - INFO  [main:QuorumPeerConfig@130] - Reading configuration from: /conf/zoo.cfg
2019-11-27 15:23:25,502 [myid:] - INFO  [main:QuorumPeerConfig@366] - clientPort is not set
2019-11-27 15:23:25,502 [myid:] - INFO  [main:QuorumPeerConfig@380] - secureClientPort is not set
2019-11-27 15:23:25,510 [myid:] - WARN  [main:QuorumPeerConfig@641] - No server failure will be tolerated. You need at least 3 servers.
2019-11-27 15:23:25,513 [myid:2] - WARN  [main:QuorumPeerConfig@706] - Peer type from servers list (OBSERVER) doesn't match peerType (PARTICIPANT). Defaulting to servers list.
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-11-27 15:23:25,516 [myid:2] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-11-27 15:23:25,528 [myid:2] - INFO  [main:QuorumPeerMain@141] - Starting quorum peer
2019-11-27 15:23:25,540 [myid:2] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-11-27 15:23:25,545 [myid:2] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
2019-11-27 15:23:25,553 [myid:2] - INFO  [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2019-11-27 15:23:25,584 [myid:2] - INFO  [main:Log@186] - Logging initialized @552ms
2019-11-27 15:23:25,641 [myid:2] - WARN  [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@1d057a39{/,null,null} contextPath ends with /*
2019-11-27 15:23:25,641 [myid:2] - WARN  [main:ContextHandler@1350] - Empty contextPath
2019-11-27 15:23:25,659 [myid:2] - INFO  [main:QuorumPeer@1430] - Local sessions disabled
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1441] - Local session upgrading disabled
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1408] - tickTime set to 2000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1452] - minSessionTimeout set to 4000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1463] - maxSessionTimeout set to 40000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1478] - initLimit set to 10
2019-11-27 15:23:25,673 [myid:2] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-11-27 15:23:25,675 [myid:2] - INFO  [main:QuorumPeer@2036] - QuorumPeer communication is not secured!
2019-11-27 15:23:25,675 [myid:2] - INFO  [main:QuorumPeer@2065] - quorum.cnxn.threads.size set to 20
2019-11-27 15:23:25,679 [myid:2] - INFO  [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.0
2019-11-27 15:23:25,693 [myid:2] - INFO  [main:Server@327] - jetty-9.2.18.v20160721
2019-11-27 15:23:25,730 [myid:2] - INFO  [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@1d057a39{/,null,AVAILABLE}
2019-11-27 15:23:25,735 [myid:2] - INFO  [main:AbstractConnector@266] - Started ServerConnector@6737fd8f{HTTP/1.1}{0.0.0.0:8080}
2019-11-27 15:23:25,736 [myid:2] - INFO  [main:Server@379] - Started @707ms
2019-11-27 15:23:25,736 [myid:2] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-11-27 15:23:25,742 [myid:2] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-1.onap-headless.zookeeper.svc.cluster.local/10.233.66.89:3888
2019-11-27 15:23:25,752 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1136] - LOOKING
2019-11-27 15:23:25,753 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@893] - New election. My id =  2, proposed zxid=0x8000000000000000
2019-11-27 15:23:25,760 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:25,762 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:25,762 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:25,763 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:25,962 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 400
2019-11-27 15:23:25,965 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:25,965 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:25,966 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:25,967 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:26,363 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 800
2019-11-27 15:23:26,367 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:26,368 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:26,368 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:26,369 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:27,164 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 1600
2019-11-27 15:23:27,168 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:27,168 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:27,169 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:27,169 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:28,766 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 3200
2019-11-27 15:23:28,769 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:28,770 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:28,770 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:28,771 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:31,967 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 6400
2019-11-27 15:23:31,970 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:31,971 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:31,971 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:31,972 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:37,591 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36965
2019-11-27 15:23:37,603 [myid:2] - INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-11-27 15:23:37,603 [myid:2] - INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[stat, envi, conf, ruok, mntr, srvr, crst, cons]]
2019-11-27 15:23:37,604 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:36965
2019-11-27 15:23:37,610 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:36965 (no session established for client)
2019-11-27 15:23:38,367 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 12800
2019-11-27 15:23:38,374 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:38,374 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:38,374 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:23:38,375 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:39,383 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:46681
2019-11-27 15:23:39,384 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:46681
2019-11-27 15:23:39,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:46681 (no session established for client)
2019-11-27 15:23:47,574 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44881
2019-11-27 15:23:47,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44881
2019-11-27 15:23:47,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44881 (no session established for client)
2019-11-27 15:23:49,386 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44397
2019-11-27 15:23:49,387 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44397
2019-11-27 15:23:49,388 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44397 (no session established for client)
2019-11-27 15:23:51,168 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 25600
2019-11-27 15:23:51,174 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:51,174 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:51,174 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:23:51,175 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:57,581 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40065
2019-11-27 15:23:57,583 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40065
2019-11-27 15:23:57,584 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40065 (no session established for client)
2019-11-27 15:23:59,400 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36819
2019-11-27 15:23:59,401 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:36819
2019-11-27 15:23:59,402 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:36819 (no session established for client)
2019-11-27 15:24:07,572 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40383
2019-11-27 15:24:07,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40383
2019-11-27 15:24:07,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40383 (no session established for client)
2019-11-27 15:24:09,397 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45489
2019-11-27 15:24:09,399 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45489
2019-11-27 15:24:09,399 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45489 (no session established for client)
2019-11-27 15:24:16,769 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 51200
2019-11-27 15:24:16,778 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:24:16,778 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:24:16,778 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:24:16,780 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:24:17,577 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38589
2019-11-27 15:24:17,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38589
2019-11-27 15:24:17,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38589 (no session established for client)
2019-11-27 15:24:19,383 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:37545
2019-11-27 15:24:19,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:37545
2019-11-27 15:24:19,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:37545 (no session established for client)
2019-11-27 15:24:27,578 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43155
2019-11-27 15:24:27,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43155
2019-11-27 15:24:27,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43155 (no session established for client)
2019-11-27 15:24:29,392 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44137
2019-11-27 15:24:29,393 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44137
2019-11-27 15:24:29,393 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44137 (no session established for client)
2019-11-27 15:24:37,575 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38211
2019-11-27 15:24:37,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38211
2019-11-27 15:24:37,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38211 (no session established for client)
2019-11-27 15:24:39,395 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43741
2019-11-27 15:24:39,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43741
2019-11-27 15:24:39,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43741 (no session established for client)
2019-11-27 15:24:47,586 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35941
2019-11-27 15:24:47,587 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35941
2019-11-27 15:24:47,587 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35941 (no session established for client)
2019-11-27 15:24:49,391 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43899
2019-11-27 15:24:49,392 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43899
2019-11-27 15:24:49,392 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43899 (no session established for client)
2019-11-27 15:24:57,572 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43943
2019-11-27 15:24:57,574 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43943
2019-11-27 15:24:57,574 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43943 (no session established for client)
2019-11-27 15:24:59,395 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41531
2019-11-27 15:24:59,395 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:41531
2019-11-27 15:24:59,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:41531 (no session established for client)
2019-11-27 15:25:07,592 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42705
2019-11-27 15:25:07,593 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:42705
2019-11-27 15:25:07,593 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:42705 (no session established for client)
2019-11-27 15:25:07,970 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 60000
2019-11-27 15:25:07,975 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:25:07,975 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:25:07,976 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:25:07,977 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:25:09,384 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38363
2019-11-27 15:25:09,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38363
2019-11-27 15:25:09,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38363 (no session established for client)
...
waynz0r commented 4 years ago

Hi @sylvainOL,

we have a custom Pilot and Mixer component we use it internally - especially for these kind of setups and a better multicluster support. In Istio 1.4 there were lots of changes and fixes around headless services but unfortunately it is still not good enough to run services as ZK. These continuous patching are not a way going forward, thus we are working on a solution for Zookeeper that will work without a headless service and provide a more stable way to operate ZK within an Istio based service mesh.

I suggest to check back here as this will be part of a forthcoming release of the operator (we need this as well as we operate Kafka clusters on K8s with/without Istio as well).

sylvainOL commented 4 years ago

hi @waynz0r, thanks for the quick reply! OK, I'll wait for the release then!

In the meantime, I'll put ZK out of the mesh!

thanks again for the heads up

baluchicken commented 4 years ago

Hi, @sylvainOL we just opened a PR to pravega/zookeeper-operator which allows to use Zookeeper with Istio. While that PR is under review we recommend to use our docker images, which contains the changes addressed in the PR. For the operator: banzaicloud/zookeeper-operator:0.1.0 For Zookeeper: banzaicloud/zookeeper:3.5.4-beta

To deploy the operator you can safely use the helm chart here To deploy a Zk cluster:

kubectl create -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: example-zookeepercluster
  namespace: zookeeper
spec:
  image: 
    repository: banzaicloud/zookeeper
    tag: 3.5.4-beta
    pullPolicy: Always
  replicas: 3
  isIstioEnabled: true
  config:
    initLimit: 10
    tickTime: 2000
    syncLimit: 5
    quorumListenOnAllIPs: true
EOF