pravega / zookeeper-operator

Kubernetes Operator for Zookeeper
Apache License 2.0
368 stars 202 forks source link

Zookeeper Cluster can't start when istio is used #102

Closed sylvainOL closed 4 years ago

sylvainOL commented 4 years ago

Hello,

I've deployed a kubernetes cluster with istio. When trying to deploy a 3 nodes ZooKeeper Cluster, the second one can't start because of immediate closed connections.

As banzai cloud has made a blog post on Kafka (+ZK?) on Istio (https://banzaicloud.com/blog/kafka-on-istio-performance/), and they propose to use your operator (https://github.com/banzaicloud/kafka-operator), I assumed it's possible but don't see how :-/

I've tried to disable mTLS but doesn't seem to be the issue.

here's the way I deployed:

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/
helm repo update
kubectl create ns zookeeper
kubectl label namespace zookeeper istio-injection=enabled
helm install --name zookeeper-operator --namespace=zookeeper banzaicloud-stable/zookeeper-operator
kubectl apply -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: onap
  namespace: zookeeper
spec:
  replicas: 3
  persistence:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 3G
    storageClassName: ssd
EOF

here's the logs:

+ source /conf/env.sh
++ DOMAIN=onap-headless.zookeeper.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=onap-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=onap-0
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ onap-0 =~ (.*)-([0-9]+)$ ]]
+ NAME=onap
+ ORD=0
+ MYID=1
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
++ cat /data/myid
+ EXISTING_ID=1
+ [[ 1 == \1 ]]
+ WRITE_CONFIGURATION=false
+ set +e
+ nslookup onap-headless.zookeeper.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'onap-headless.zookeeper.svc.cluster.local': Name does not resolve
+ [[ 1 -eq 1 ]]
+ set -e
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=false
+ [[ true == true ]]
+ echo 'Writing myid: 1 to: /data/myid.'
+ echo 1
Writing myid: 1 to: /data/myid.
+ [[ 1 -eq 1 ]]
+ [[ false == false ]]
+ ROLE=participant
+ echo Initial initialization of ordinal 0 pod, creating new config.
Initial initialization of ordinal 0 pod, creating new config.
++ zkConfig
++ echo 'onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ ZKCONFIG='onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ echo Writing bootstrap configuration with the following config:
Writing bootstrap configuration with the following config:
onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181
+ echo 'onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
+ echo 1
+ echo 'server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;2181'
Starting zookeeper service
+ echo 'Starting zookeeper service'
+ zkServer.sh start-foreground
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2019-11-27 15:22:48,324 [myid:] - INFO  [main:QuorumPeerConfig@130] - Reading configuration from: /conf/zoo.cfg
2019-11-27 15:22:48,329 [myid:] - INFO  [main:QuorumPeerConfig@366] - clientPort is not set
2019-11-27 15:22:48,329 [myid:] - INFO  [main:QuorumPeerConfig@380] - secureClientPort is not set
2019-11-27 15:22:48,335 [myid:] - WARN  [main:QuorumPeerConfig@641] - No server failure will be tolerated. You need at least 3 servers.
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-11-27 15:22:48,339 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-11-27 15:22:48,340 [myid:1] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-11-27 15:22:48,351 [myid:1] - INFO  [main:QuorumPeerMain@141] - Starting quorum peer
2019-11-27 15:22:48,361 [myid:1] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-11-27 15:22:48,364 [myid:1] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
2019-11-27 15:22:48,369 [myid:1] - INFO  [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2019-11-27 15:22:48,399 [myid:1] - INFO  [main:Log@186] - Logging initialized @527ms
2019-11-27 15:22:48,460 [myid:1] - WARN  [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@1d057a39{/,null,null} contextPath ends with /*
2019-11-27 15:22:48,460 [myid:1] - WARN  [main:ContextHandler@1350] - Empty contextPath
2019-11-27 15:22:48,481 [myid:1] - INFO  [main:QuorumPeer@1430] - Local sessions disabled
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1441] - Local session upgrading disabled
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1408] - tickTime set to 2000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1452] - minSessionTimeout set to 4000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1463] - maxSessionTimeout set to 40000
2019-11-27 15:22:48,482 [myid:1] - INFO  [main:QuorumPeer@1478] - initLimit set to 10
2019-11-27 15:22:48,496 [myid:1] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-11-27 15:22:48,498 [myid:1] - INFO  [main:QuorumPeer@2036] - QuorumPeer communication is not secured!
2019-11-27 15:22:48,498 [myid:1] - INFO  [main:QuorumPeer@2065] - quorum.cnxn.threads.size set to 20
2019-11-27 15:22:48,505 [myid:1] - INFO  [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.100000111
2019-11-27 15:22:48,532 [myid:1] - INFO  [main:Server@327] - jetty-9.2.18.v20160721
2019-11-27 15:22:48,561 [myid:1] - INFO  [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@1d057a39{/,null,AVAILABLE}
2019-11-27 15:22:48,566 [myid:1] - INFO  [main:AbstractConnector@266] - Started ServerConnector@71bbf57e{HTTP/1.1}{0.0.0.0:8080}
2019-11-27 15:22:48,567 [myid:1] - INFO  [main:Server@379] - Started @698ms
2019-11-27 15:22:48,567 [myid:1] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-11-27 15:22:48,571 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888
2019-11-27 15:22:48,580 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1136] - LOOKING
2019-11-27 15:22:48,581 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@893] - New election. My id =  1, proposed zxid=0x20000001e
2019-11-27 15:22:48,584 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@687] - Notification: 2 (message format version), 1 (n.leader), 0x20000001e (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-11-27 15:22:48,787 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=LeaderElection]
2019-11-27 15:22:48,789 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1224] - LEADING
2019-11-27 15:22:48,795 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@67] - TCP NoDelay set to: true
2019-11-27 15:22:48,797 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@87] - zookeeper.leader.maxConcurrentSnapshots = 10
2019-11-27 15:22:48,797 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@89] - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:zookeeper.version=3.5.4-beta-7f51e5b68cf2f80176ff944a9ebd2abbc65e7327, built on 05/11/2018 16:27 GMT
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:host.name=onap-0.onap-headless.zookeeper.svc.cluster.local
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.version=1.8.0_181
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.vendor=Oracle Corporation
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2019-11-27 15:22:48,812 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.class.path=/zookeeper-3.5.4-beta/bin/../build/classes:/zookeeper-3.5.4-beta/bin/../build/lib/*.jar:/zookeeper-3.5.4-beta/bin/../lib/slf4j-log4j12-1.7.25.jar:/zookeeper-3.5.4-beta/bin/../lib/slf4j-api-1.7.25.jar:/zookeeper-3.5.4-beta/bin/../lib/netty-3.10.6.Final.jar:/zookeeper-3.5.4-beta/bin/../lib/log4j-1.2.17.jar:/zookeeper-3.5.4-beta/bin/../lib/jline-2.11.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-util-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/zookeeper-3.5.4-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/zookeeper-3.5.4-beta/bin/../lib/jackson-mapper-asl-1.9.11.jar:/zookeeper-3.5.4-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/zookeeper-3.5.4-beta/bin/../lib/commons-cli-1.2.jar:/zookeeper-3.5.4-beta/bin/../lib/audience-annotations-0.5.0.jar:/zookeeper-3.5.4-beta/bin/../zookeeper-3.5.4-beta.jar:/zookeeper-3.5.4-beta/bin/../src/java/lib/*.jar:/conf:
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.io.tmpdir=/tmp
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:java.compiler=<NA>
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.name=Linux
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.arch=amd64
2019-11-27 15:22:48,813 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.version=4.9.0-8-amd64
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.name=root
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.home=/root
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:user.dir=/zookeeper-3.5.4-beta
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.free=444MB
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.max=966MB
2019-11-27 15:22:48,814 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Environment@109] - Server environment:os.memory.total=487MB
2019-11-27 15:22:48,816 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@928] - minSessionTimeout set to 4000
2019-11-27 15:22:48,816 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@937] - maxSessionTimeout set to 40000
2019-11-27 15:22:48,817 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ZooKeeperServer@160] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /data/version-2 snapdir /data/version-2
2019-11-27 15:22:48,818 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@432] - LEADING - LEADER ELECTION TOOK - 30 MS
2019-11-27 15:22:48,820 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FileTxnSnapLog@367] - Snapshotting: 0x20000001e to /data/version-2/snapshot.20000001e
2019-11-27 15:22:48,831 [myid:1] - ERROR [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-11-27 15:22:48,836 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Leader@1264] - Have quorum of supporters, sids: [ [1],[1] ]; starting up and setting last processed zxid: 0x300000000
2019-11-27 15:22:48,836 [myid:1] - ERROR [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1596] - Error closing file:
2019-11-27 15:22:48,847 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):CommitProcessor@255] - Configuring CommitProcessor with 1 worker threads.
2019-11-27 15:22:48,856 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):PrepRequestProcessor@99] - zookeeper.skipACL=="yes", ACL checks will be skipped
2019-11-27 15:22:48,858 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2019-11-27 15:22:59,878 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40039
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[stat, envi, conf, ruok, mntr, srvr, crst, cons]]
2019-11-27 15:22:59,893 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40039
2019-11-27 15:22:59,899 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40039 (no session established for client)
2019-11-27 15:23:00,535 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44549
2019-11-27 15:23:00,537 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44549
2019-11-27 15:23:00,539 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44549 (no session established for client)
2019-11-27 15:23:09,895 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42815
2019-11-27 15:23:09,897 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:42815
2019-11-27 15:23:09,898 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:42815 (no session established for client)
2019-11-27 15:23:10,549 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35831
2019-11-27 15:23:10,551 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35831
2019-11-27 15:23:10,551 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35831 (no session established for client)
2019-11-27 15:23:19,893 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45287
2019-11-27 15:23:19,896 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45287
2019-11-27 15:23:19,897 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45287 (no session established for client)
2019-11-27 15:23:20,517 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43115
2019-11-27 15:23:20,519 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43115
2019-11-27 15:23:20,520 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43115 (no session established for client)
2019-11-27 15:23:23,856 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33672
2019-11-27 15:23:23,862 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33672
2019-11-27 15:23:23,864 [myid:1] - INFO  [SyncThread:1:FileTxnLog@201] - Creating new log file: log.300000001
2019-11-27 15:23:23,877 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090000 with negotiated timeout 4000 for client /127.0.0.1:33672
2019-11-27 15:23:24,224 [myid:1] - WARN  [NIOWorkerThread-1:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x1000bc750090000, likely client has closed socket
2019-11-27 15:23:24,225 [myid:1] - INFO  [NIOWorkerThread-1:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=127.0.0.1,name5=0x1000bc750090000]
2019-11-27 15:23:24,226 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:33672 which had sessionid 0x1000bc750090000
2019-11-27 15:23:24,613 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33678
2019-11-27 15:23:24,615 [myid:1] - INFO  [NIOWorkerThread-2:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33678
2019-11-27 15:23:24,620 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090001 with negotiated timeout 4000 for client /127.0.0.1:33678
2019-11-27 15:23:24,635 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@432] - skipACL is set, reconfig operation will skip ACL checks!
2019-11-27 15:23:24,635 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@474] - Incremental reconfig
2019-11-27 15:23:24,639 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::QuorumPeer$QuorumServer@173] - Failed to resolve address: onap-1.onap-headless.zookeeper.svc.cluster.local
java.net.UnknownHostException: onap-1.onap-headless.zookeeper.svc.cluster.local
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at java.net.InetAddress.getByName(InetAddress.java:1076)
    at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:171)
    at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:727)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:682)
    at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1519)
    at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1556)
    at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1074)
    at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
    at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:906)
    at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:145)
2019-11-27 15:23:24,644 [myid:1] - WARN  [ProcessThread(sid:1 cport:-1)::QuorumCnxManager@660] - Cannot open channel to 2 at election address onap-1.onap-headless.zookeeper.svc.cluster.local:3888
java.net.UnknownHostException: onap-1.onap-headless.zookeeper.svc.cluster.local
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:638)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:695)
    at org.apache.zookeeper.server.quorum.QuorumPeer.connectNewPeers(QuorumPeer.java:1519)
    at org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier(QuorumPeer.java:1556)
    at org.apache.zookeeper.server.quorum.Leader.propose(Leader.java:1074)
    at org.apache.zookeeper.server.quorum.ProposalRequestProcessor.processRequest(ProposalRequestProcessor.java:78)
    at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:906)
    at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:145)
2019-11-27 15:23:24,645 [myid:1] - ERROR [ProcessThread(sid:1 cport:-1)::QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-11-27 15:23:24,647 [myid:1] - ERROR [SyncThread:1:QuorumPeer@1596] - Error closing file:
2019-11-27 15:23:24,649 [myid:1] - WARN  [SyncThread:1:QuorumPeer@1527] - Restarting Leader Election
2019-11-27 15:23:24,650 [myid:1] - INFO  [onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888:QuorumCnxManager$Listener@892] - Leaving listener
2019-11-27 15:23:24,651 [myid:1] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-0.onap-headless.zookeeper.svc.cluster.local/10.233.67.152:3888
2019-11-27 15:23:25,005 [myid:1] - WARN  [NIOWorkerThread-2:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x1000bc750090001, likely client has closed socket
2019-11-27 15:23:25,006 [myid:1] - INFO  [NIOWorkerThread-2:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=127.0.0.1,name5=0x1000bc750090001]
2019-11-27 15:23:25,007 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:33678 which had sessionid 0x1000bc750090001
2019-11-27 15:23:27,586 [myid:1] - INFO  [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@469] - WorkerSender is down
2019-11-27 15:23:27,588 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@439] - WorkerReceiver is down
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:ZooKeeperServer@392] - Expiring session 0x1000bc750090000, timeout of 4000ms exceeded
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1000bc750090000
2019-11-27 15:23:29,311 [myid:1] - INFO  [SessionTracker:ZooKeeperServer@392] - Expiring session 0x1000bc750090001, timeout of 4000ms exceeded
2019-11-27 15:23:29,312 [myid:1] - INFO  [SessionTracker:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x1000bc750090001
2019-11-27 15:23:29,312 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x1000bc750090000
2019-11-27 15:23:29,312 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x1000bc750090001
2019-11-27 15:23:29,885 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45507
2019-11-27 15:23:29,887 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45507
2019-11-27 15:23:29,888 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45507 (no session established for client)
2019-11-27 15:23:30,528 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35905
2019-11-27 15:23:30,529 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35905
2019-11-27 15:23:30,530 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35905 (no session established for client)
2019-11-27 15:23:39,773 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:33802
2019-11-27 15:23:39,775 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /127.0.0.1:33802
2019-11-27 15:23:39,780 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x1000bc750090002 with negotiated timeout 4000 for client /127.0.0.1:33802
2019-11-27 15:23:39,882 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39623
...
+ source /conf/env.sh
++ DOMAIN=onap-headless.zookeeper.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=onap-client
++ CLIENT_PORT=2181
+ source /usr/local/bin/zookeeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=onap-1
+ DATA_DIR=/data
+ MYID_FILE=/data/myid
+ LOG4J_CONF=/conf/log4j-quiet.properties
+ DYNCONFIG=/data/zoo.cfg.dynamic
+ [[ onap-1 =~ (.*)-([0-9]+)$ ]]
+ NAME=onap
+ ORD=1
+ MYID=2
+ WRITE_CONFIGURATION=true
+ REGISTER_NODE=true
+ '[' -f /data/myid ']'
++ cat /data/myid
+ EXISTING_ID=2
+ [[ 2 == \2 ]]
+ WRITE_CONFIGURATION=false
+ set +e
+ nslookup onap-headless.zookeeper.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

Name:      onap-headless.zookeeper.svc.cluster.local
Address 1: 10.233.67.152 10-233-67-152.onap-client.zookeeper.svc.cluster.local
+ [[ 0 -eq 1 ]]
+ set -e
+ set +e
++ zkConnectionString
++ set +e
++ nslookup onap-client
++ [[ 0 -eq 1 ]]
++ set -e
++ echo onap-client:2181
+ ZKURL=onap-client:2181
+ set -e
++ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar get-all onap-client:2181
Connecting to Zookeeper onap-client:2181
+ CONFIG='server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
version=300000000'
++ echo 'server.1=onap-0.onap-headless.zookeeper.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
version=300000000'
++ grep server.2=
++ wc -l
+ REGISTERED=0
+ [[ 0 -eq 1 ]]
+ REGISTER_NODE=true
+ WRITE_CONFIGURATION=true
Writing myid: 2 to: /data/myid.
+ [[ true == true ]]
+ echo 'Writing myid: 2 to: /data/myid.'
+ echo 2
+ [[ 2 -eq 1 ]]
+ [[ 2 -ne 1 ]]
+ [[ true == false ]]
+ [[ true == true ]]
+ ROLE=observer
++ zkConfig
++ echo 'onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181'
+ ZKCONFIG='onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181'
+ echo Registering node and writing local configuration to disk.
+ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar add onap-client:2181 2 'onap-1.onap-headless.zookeeper.svc.cluster.local:2888:3888:observer;2181' /data/zoo.cfg.dynamic
Registering node and writing local configuration to disk.
Connecting to Zookeeper onap-client:2181
+ echo 'Starting zookeeper service'
+ zkServer.sh start-foreground
Starting zookeeper service
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2019-11-27 15:23:25,496 [myid:] - INFO  [main:QuorumPeerConfig@130] - Reading configuration from: /conf/zoo.cfg
2019-11-27 15:23:25,502 [myid:] - INFO  [main:QuorumPeerConfig@366] - clientPort is not set
2019-11-27 15:23:25,502 [myid:] - INFO  [main:QuorumPeerConfig@380] - secureClientPort is not set
2019-11-27 15:23:25,510 [myid:] - WARN  [main:QuorumPeerConfig@641] - No server failure will be tolerated. You need at least 3 servers.
2019-11-27 15:23:25,513 [myid:2] - WARN  [main:QuorumPeerConfig@706] - Peer type from servers list (OBSERVER) doesn't match peerType (PARTICIPANT). Defaulting to servers list.
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-11-27 15:23:25,515 [myid:2] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-11-27 15:23:25,516 [myid:2] - INFO  [main:ManagedUtil@46] - Log4j found with jmx enabled.
2019-11-27 15:23:25,528 [myid:2] - INFO  [main:QuorumPeerMain@141] - Starting quorum peer
2019-11-27 15:23:25,540 [myid:2] - INFO  [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-11-27 15:23:25,545 [myid:2] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 2 worker threads, and 64 kB direct buffers.
2019-11-27 15:23:25,553 [myid:2] - INFO  [main:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:2181
2019-11-27 15:23:25,584 [myid:2] - INFO  [main:Log@186] - Logging initialized @552ms
2019-11-27 15:23:25,641 [myid:2] - WARN  [main:ContextHandler@1339] - o.e.j.s.ServletContextHandler@1d057a39{/,null,null} contextPath ends with /*
2019-11-27 15:23:25,641 [myid:2] - WARN  [main:ContextHandler@1350] - Empty contextPath
2019-11-27 15:23:25,659 [myid:2] - INFO  [main:QuorumPeer@1430] - Local sessions disabled
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1441] - Local session upgrading disabled
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1408] - tickTime set to 2000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1452] - minSessionTimeout set to 4000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1463] - maxSessionTimeout set to 40000
2019-11-27 15:23:25,660 [myid:2] - INFO  [main:QuorumPeer@1478] - initLimit set to 10
2019-11-27 15:23:25,673 [myid:2] - INFO  [main:ZKDatabase@117] - zookeeper.snapshotSizeFactor = 0.33
2019-11-27 15:23:25,675 [myid:2] - INFO  [main:QuorumPeer@2036] - QuorumPeer communication is not secured!
2019-11-27 15:23:25,675 [myid:2] - INFO  [main:QuorumPeer@2065] - quorum.cnxn.threads.size set to 20
2019-11-27 15:23:25,679 [myid:2] - INFO  [main:FileSnap@83] - Reading snapshot /data/version-2/snapshot.0
2019-11-27 15:23:25,693 [myid:2] - INFO  [main:Server@327] - jetty-9.2.18.v20160721
2019-11-27 15:23:25,730 [myid:2] - INFO  [main:ContextHandler@744] - Started o.e.j.s.ServletContextHandler@1d057a39{/,null,AVAILABLE}
2019-11-27 15:23:25,735 [myid:2] - INFO  [main:AbstractConnector@266] - Started ServerConnector@6737fd8f{HTTP/1.1}{0.0.0.0:8080}
2019-11-27 15:23:25,736 [myid:2] - INFO  [main:Server@379] - Started @707ms
2019-11-27 15:23:25,736 [myid:2] - INFO  [main:JettyAdminServer@112] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2019-11-27 15:23:25,742 [myid:2] - INFO  [QuorumPeerListener:QuorumCnxManager$Listener@854] - My election bind port: onap-1.onap-headless.zookeeper.svc.cluster.local/10.233.66.89:3888
2019-11-27 15:23:25,752 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumPeer@1136] - LOOKING
2019-11-27 15:23:25,753 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@893] - New election. My id =  2, proposed zxid=0x8000000000000000
2019-11-27 15:23:25,760 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:25,762 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:25,762 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:25,763 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:25,962 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 400
2019-11-27 15:23:25,965 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:25,965 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:25,966 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:25,967 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:26,363 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 800
2019-11-27 15:23:26,367 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:26,368 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:26,368 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:26,369 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:27,164 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 1600
2019-11-27 15:23:27,168 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:27,168 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:27,169 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:27,169 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:28,766 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 3200
2019-11-27 15:23:28,769 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:28,770 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:28,770 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:28,771 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:31,967 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 6400
2019-11-27 15:23:31,970 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:31,971 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:31,971 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:23:31,972 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:37,591 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36965
2019-11-27 15:23:37,603 [myid:2] - INFO  [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-11-27 15:23:37,603 [myid:2] - INFO  [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[stat, envi, conf, ruok, mntr, srvr, crst, cons]]
2019-11-27 15:23:37,604 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:36965
2019-11-27 15:23:37,610 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:36965 (no session established for client)
2019-11-27 15:23:38,367 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 12800
2019-11-27 15:23:38,374 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:38,374 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:38,374 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:23:38,375 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:39,383 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:46681
2019-11-27 15:23:39,384 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:46681
2019-11-27 15:23:39,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:46681 (no session established for client)
2019-11-27 15:23:47,574 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44881
2019-11-27 15:23:47,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44881
2019-11-27 15:23:47,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44881 (no session established for client)
2019-11-27 15:23:49,386 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44397
2019-11-27 15:23:49,387 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44397
2019-11-27 15:23:49,388 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44397 (no session established for client)
2019-11-27 15:23:51,168 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 25600
2019-11-27 15:23:51,174 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:23:51,174 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:23:51,174 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:23:51,175 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:23:57,581 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40065
2019-11-27 15:23:57,583 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40065
2019-11-27 15:23:57,584 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40065 (no session established for client)
2019-11-27 15:23:59,400 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:36819
2019-11-27 15:23:59,401 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:36819
2019-11-27 15:23:59,402 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:36819 (no session established for client)
2019-11-27 15:24:07,572 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:40383
2019-11-27 15:24:07,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:40383
2019-11-27 15:24:07,575 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:40383 (no session established for client)
2019-11-27 15:24:09,397 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:45489
2019-11-27 15:24:09,399 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:45489
2019-11-27 15:24:09,399 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:45489 (no session established for client)
2019-11-27 15:24:16,769 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 51200
2019-11-27 15:24:16,778 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:24:16,778 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:24:16,778 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1069] - Exception when using channel: for id 1 my id = 2 error = java.net.SocketException: Broken pipe (Write failed)
2019-11-27 15:24:16,780 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:24:17,577 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38589
2019-11-27 15:24:17,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38589
2019-11-27 15:24:17,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38589 (no session established for client)
2019-11-27 15:24:19,383 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:37545
2019-11-27 15:24:19,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:37545
2019-11-27 15:24:19,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:37545 (no session established for client)
2019-11-27 15:24:27,578 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43155
2019-11-27 15:24:27,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43155
2019-11-27 15:24:27,579 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43155 (no session established for client)
2019-11-27 15:24:29,392 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:44137
2019-11-27 15:24:29,393 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:44137
2019-11-27 15:24:29,393 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:44137 (no session established for client)
2019-11-27 15:24:37,575 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38211
2019-11-27 15:24:37,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38211
2019-11-27 15:24:37,576 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38211 (no session established for client)
2019-11-27 15:24:39,395 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43741
2019-11-27 15:24:39,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43741
2019-11-27 15:24:39,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43741 (no session established for client)
2019-11-27 15:24:47,586 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:35941
2019-11-27 15:24:47,587 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:35941
2019-11-27 15:24:47,587 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:35941 (no session established for client)
2019-11-27 15:24:49,391 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43899
2019-11-27 15:24:49,392 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43899
2019-11-27 15:24:49,392 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43899 (no session established for client)
2019-11-27 15:24:57,572 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:43943
2019-11-27 15:24:57,574 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:43943
2019-11-27 15:24:57,574 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:43943 (no session established for client)
2019-11-27 15:24:59,395 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:41531
2019-11-27 15:24:59,395 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:41531
2019-11-27 15:24:59,396 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:41531 (no session established for client)
2019-11-27 15:25:07,592 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42705
2019-11-27 15:25:07,593 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:42705
2019-11-27 15:25:07,593 [myid:2] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:42705 (no session established for client)
2019-11-27 15:25:07,970 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@927] - Notification time out: 60000
2019-11-27 15:25:07,975 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1148] - Connection broken for id 1, my id = 2, error =
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1133)
2019-11-27 15:25:07,975 [myid:2] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@1151] - Interrupting SendWorker
2019-11-27 15:25:07,976 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1064] - Interrupted while waiting for message on queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
    at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1215)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:75)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1052)
2019-11-27 15:25:07,977 [myid:2] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@1074] - Send worker leaving thread  id 1 my id = 2
2019-11-27 15:25:09,384 [myid:2] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38363
2019-11-27 15:25:09,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:38363
2019-11-27 15:25:09,385 [myid:2] - INFO  [NIOWorkerThread-2:NIOServerCnxn@627] - Closed socket connection for client /127.0.0.1:38363 (no session established for client)
...
seecsea commented 4 years ago

Occur the same: I use the yaml from git repo,not helm k8s:1.14.5

Readiness probe failed: HTTP probe failed with statuscode: 503

kubectl describe pod zookeeper-operator-8449df9744-lkx9j Events: Type Reason Age From Message


Normal Scheduled 2d11h default-scheduler Successfully assigned default/zookeeper-operator-8449df9744-lkx9j to k8stian-n3 Normal Pulled 2d11h kubelet, k8stian-n3 Container image "docker.io/istio/proxy_init:1.2.2" already present on machine Normal Created 2d11h kubelet, k8stian-n3 Created container istio-init Normal Started 2d11h kubelet, k8stian-n3 Started container istio-init Normal Pulled 2d11h kubelet, k8stian-n3 Container image "docker.io/istio/proxyv2:1.2.2" already present on machine Normal Started 2d11h kubelet, k8stian-n3 Started container istio-proxy Normal Created 2d11h kubelet, k8stian-n3 Created container istio-proxy Normal Pulling 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Pulling image "pravega/zookeeper-operator:latest" Normal Pulled 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Successfully pulled image "pravega/zookeeper-operator:latest" Normal Created 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Created container zookeeper-operator Normal Started 2d11h (x2 over 2d11h) kubelet, k8stian-n3 Started container zookeeper-operator Warning Unhealthy 2m18s (x106771 over 2d11h) kubelet, k8stian-n3 Readiness probe failed: HTTP probe failed with statuscode: 503

[root@k8stian-m2:/usr/local/src/deploy/zookeeper-operator/deploy/crds]# kubectl logs -f zookeeper-operator-8449df9744-lkx9j Error from server (BadRequest): a container name must be specified for pod zookeeper-operator-8449df9744-lkx9j, choose one of: [zookeeper-operator istio-proxy] or one of the init containers: [istio-init] [root@k8stian-m2:/usr/local/src/deploy/zookeeper-operator/deploy/crds]# kubectl logs -f zookeeper-operator-8449df9744-lkx9j -c zookeeper-operator {"level":"info","ts":1575336376.1156788,"logger":"cmd","msg":"zookeeper-operator Version: 0.2.3-17"} {"level":"info","ts":1575336376.1157143,"logger":"cmd","msg":"Git SHA: cea93ee"} {"level":"info","ts":1575336376.1157634,"logger":"cmd","msg":"Go Version: go1.12.10"} {"level":"info","ts":1575336376.1157687,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1575336376.1157732,"logger":"cmd","msg":"operator-sdk Version: v0.3.0"} {"level":"info","ts":1575336376.1163554,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1575336376.294069,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1575336376.3139248,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1575336376.3809543,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1575336376.3813946,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575336376.3815124,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575336376.3815713,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575336376.381645,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"zookeepercluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1575336376.3816953,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1575336376.5820937,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"zookeepercluster-controller"} {"level":"info","ts":1575336376.6822746,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"zookeepercluster-controller","worker count":1}

spiegela commented 4 years ago

@seecsea @sylvainOL Thanks for reporting this. I'll take a look ASAP.

alihalabyah commented 4 years ago

Any updates here?

pbelgundi commented 4 years ago

To fix this we perhaps need to set quorumListenOnAllIPs to true in Zookeeper:

https://github.com/istio/istio.io/blob/master/content/en/faq/applications/zookeeper.md

https://github.com/helm/charts/issues/17183

https://github.com/istio/istio/issues/19280

AceHack commented 4 years ago

Yes it's still not working for me as well.

AceHack commented 4 years ago

When you turn on quorumListenOnAllIPs you are going to start having this problem but if you use my health checks it will work around the problem.

https://github.com/pravega/zookeeper-operator/issues/128