操作zookeeper超时，导致整个机房的同步任务停止

[yanghongkjxy]

1.跨机房同步，有一个异城机房访问主机房的zk超时，导致到这个机房的任务全部停止，并且这个机房的node全部假死

栈信息里有看到操作zk在等待

"Load-Rpc-Async-0" #488 daemon prio=5 os_prio=0 tid=0x00007f355401a000 nid=0x3073 in Object.wait() [0x00007f33c9156000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)

• locked <0x0000000660230a48> (a org.apache.zookeeper.ClientCnxn$Packet) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1180) at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:125) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx$10.call(ZkClientx.java:870) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx$10.call(ZkClientx.java:866) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx.retryUntilConnected(ZkClientx.java:775) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx.readData(ZkClientx.java:866) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx.readData(ZkClientx.java:861) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx.readData(ZkClientx.java:850) at com.alibaba.otter.shared.common.utils.zookeeper.ZkClientx.readData(ZkClientx.java:844) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess.processDelete(NormalTerminProcess.java:148) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess.doProcess(NormalTerminProcess.java:135) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess.process(NormalTerminProcess.java:63) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess$$FastClassByCGLIB$$8078899c.invoke() at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at com.alibaba.otter.shared.arbitrate.impl.interceptor.LogInterceptor.invoke(LogInterceptor.java:53) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.termin.NormalTerminProcess$$EnhancerByCGLIB$$80507060.process() at com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.TerminZooKeeperArbitrateEvent.single(TerminZooKeeperArbitrateEvent.java:152) at com.alibaba.otter.shared.arbitrate.impl.setl.rpc.LoadRpcArbitrateEvent$1.run(LoadRpcArbitrateEvent.java:108) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

[yanghongkjxy]

@longdafeng

[yanghongkjxy]

停止前有一些处理超时的错误

[agapple]

otter对于zookeeper是一个强依赖