Zookeeper connection error Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]]

[honghaic]

2015-02-08 16:23:49,016 - [INFO] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7-SendThread(10.65.196.37:2181) Opening socket connection to server 10.65.196.37/10.65.196.37:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-08 16:23:49,312 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2 Could not read WebJar RequireJS config for: dustjs-linkedin 2.4.0 Please file a bug at: http://github.com/webjars/dustjs-linkedin/issues/new

2015-02-08 16:23:49,354 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2 Could not read WebJar RequireJS config for: json 20121008 Please file a bug at: http://github.com/webjars/json/issues/new

2015-02-08 16:23:49,378 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2 Could not read WebJar RequireJS config for: requirejs 2.1.10 Please file a bug at: http://github.com/webjars/requirejs/issues/new

2015-02-08 16:23:58,829 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2 error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms] akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[akka-actor_2.11-2.3.7.jar:na] at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[scala-library-2.11.4.jar:na] at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[scala-library-2.11.4.jar:na] at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[scala-library-2.11.4.jar:na] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[akka-actor_2.11-2.3.7.jar:na] at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40-MS]

2015-02-08 16:24:04,065 - [ERROR] - from org.apache.curator.ConnectionState in kafka-manager-system-akka.actor.default-dispatcher-7 Connection timed out for connection string (10.65.196.35:2181,10.65.196.36:2181,10.65.196.37:2181) and timeout (15000) / elapsed (15114) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.7.0.jar:na] at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [curator-client-2.7.0.jar:na] at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) ~[curator-client-2.7.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:671) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423) ~[curator-framework-2.7.0.jar:na] at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) ~[curator-framework-2.7.0.jar:na] at kafka.manager.KafkaManagerActor$$anonfun$6.apply(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at kafka.manager.KafkaManagerActor$$anonfun$6.apply(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at scala.util.Try$.apply(Try.scala:191) ~[scala-library-2.11.4.jar:na] at kafka.manager.KafkaManagerActor.(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_40-MS] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_40-MS] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_40-MS] at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_40-MS] at akka.util.Reflect$.instantiate(Reflect.scala:66) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.Props.newActor(Props.scala:252) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.ActorCell.newActor(ActorCell.scala:552) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.ActorCell.create(ActorCell.scala:578) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) ~[akka-actor_2.11-2.3.7.jar:na] at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) ~[akka-actor_2.11-2.3.7.jar:na] at akka.dispatch.Mailbox.run(Mailbox.scala:220) ~[akka-actor_2.11-2.3.7.jar:na] at akka.dispatch.Mailbox.exec(Mailbox.scala:231) ~[akka-actor_2.11-2.3.7.jar:na] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.11.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) ~[scala-library-2.11.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) ~[scala-library-2.11.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.11.4.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.11.4.jar:na]

[honghaic]

2015-02-08 16:40:44,135 - [INFO] - from org.apache.zookeeper.ZooKeeper in kafka-manager-system-akka.actor.default-dispatcher-7 Client environment:user.dir=D:\app\KafkaManager.A\siphon

2015-02-08 16:40:44,137 - [INFO] - from org.apache.zookeeper.ZooKeeper in kafka-manager-system-akka.actor.default-dispatcher-7 Initiating client connection, connectString=10.65.196.35:2181,10.65.196.36:2181,10.65.196.37:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4c0dfaf6

2015-02-08 16:40:44,142 - [DEBUG] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7 zookeeper.disableAutoWatchReset is false

2015-02-08 16:40:44,159 - [INFO] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7-SendThread(10.65.196.36:2181) Opening socket connection to server 10.65.196.36/10.65.196.36:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-08 16:40:45,054 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2 error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms] akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[akka-actor_2.11-2.3.7.jar:na] at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[scala-library-2.11.4.jar:na] at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[scala-library-2.11.4.jar:na] at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[scala-library-2.11.4.jar:na] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[akka-actor_2.11-2.3.7.jar:na] at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[akka-actor_2.11-2.3.7.jar:na] at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40-MS]

2015-02-08 16:40:45,395 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6 Could not read WebJar RequireJS config for: dustjs-linkedin 2.4.0 Please file a bug at: http://github.com/webjars/dustjs-linkedin/issues/new

2015-02-08 16:40:45,431 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6 Could not read WebJar RequireJS config for: json 20121008 Please file a bug at: http://github.com/webjars/json/issues/new

2015-02-08 16:40:45,454 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6 Could not read WebJar RequireJS config for: requirejs 2.1.10 Please file a bug at: http://github.com/webjars/requirejs/issues/new

[rycoxiao]

you can use the comand as below: kafka-manager -Dkafka-manager.zkhosts="server-01:2181". the shell can't load the rigth config file.

[miguno]

Unfortunately I confirm this problem I assume it's because of incorrect Play/sbt-native-packager settings in kafka-manager.

How to reproduce

A kafka-manager packaged and deployed via a zipfile (./sbt clean dist) is apparently not loading conf/application.conf after a deployment.

$ ./sbt clean dist    # >>> creates a zipfile
$ unzip kafka-manager-1.0-SNAPSHOT.zip
$ cd kafka-manager-1.0-SNAPSHOT 

# Now modify `conf/application.conf` to e.g. configure your ZK connection string via the
# config setting `kafka-manager.zkhosts`.  Say, set `kafka-manager.zkhosts="myzookeeper:2181".
$ vi conf/application.conf

Now, when you launch kafka-manager without any additional CLI options you will run into the following error:

$ bin/kafka-manager
Play server process ID is 57526
[info] play - Application started (Prod)
[info] play - Listening for HTTP on /0.0.0.0:9000
[INFO] [02/10/2015 10:13:53.250] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] Starting curator...
[error] o.a.c.f.i.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.net.UnknownHostException: kafka-manager-zookeeper: kafka-manager-zookeeper
        at java.net.InetAddress.getAllByName0(InetAddress.java:1348) ~[na:1.7.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1257) ~[na:1.7.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1182) ~[na:1.7.0]
        at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[org.apache.zookeeper.zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) ~[org.apache.zookeeper.zookeeper-3.4.6.jar:3.4.6-1569965]
[INFO] [02/10/2015 10:13:53.275] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] zk=kafka-manager-zookeeper:2181
[INFO] [02/10/2015 10:13:53.275] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] baseZkPath=/kafka-manager
[error] k.m.ApiError - error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[com.typesafe.akka.akka-actor_2.11-2.3.7.jar:na]
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[com.typesafe.akka.akka-actor_2.11-2.3.7.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[org.scala-lang.scala-library-2.11.4.jar:na]
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[org.scala-lang.scala-library-2.11.4.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[org.scala-lang.scala-library-2.11.4.jar:na]
[error] o.a.c.ConnectionState - Connection timed out for connection string (kafka-manager-zookeeper:2181) and timeout (15000) / elapsed (15053)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
        at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) [org.apache.curator.curator-framework-2.7.0.jar:na]
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) [org.apache.curator.curator-framework-2.7.0.jar:na]

Note the java.net.UnknownHostException: kafka-manager-zookeeper error: kafka-manager-zookeeper is the default host for kafka-manager.zkhosts. So kafka-manager is not reading the app configuration file, because if it had, it would have used myzookeeper as the ZK host.

Workaround

Explicitly provide the location of the application.conf file by setting the config.file System property on the command line when launching kafka-manager:

$ bin/kafka-manager -Dconfig.file=/path/to/kafka-manager/conf/application.conf

[mindscratch]

+1 for the work around.

[spuder]

I'm also running into this problem on both ubuntu 14.04 and mac os x. Running the latest commit as of right now.

a9fbba0cc9cb7e68f55e8a1b678b5b5437d6d4d6

Specifying the config directory is the only way I can get this to work.

[spuder]

Duplicate of #11

[lewison]

i download the newest version of kafka-manager. and try to start it by the command above which said by miguno. but I still meet the error of load conf file error.

only I set the zookeeper in the command : -Dkafka-manager-zkhosts="127.0.0.1:2181".

I can start the kafka-manager in a right way now .

so I think this issue had no been fixed yet.

[zheolong]

@lewison Same problem, do you know where the default zk urls was stored when sbt clean dist

[guangping0215]

i hit error "Bad application path: -Dconfig.file=../config/application.conf" when i use command line to start it. help~

[memelet]

I have just started to see this today. Sometimes the page renders, sometimes I get this error.

[vijaygupta358]

For mine this command was working - bin/kafka-manager -Dconfig.file=conf/application.conf -Dkafka-manager.zkhosts="localhost:2181"

make sure , u are using both parameter -Dconfig.file and -Dkafka-manager.zkhost .

If I was trying with individual parameter , it was giving same error . Try with use of both parameter .

[dinar-dalvi]

I am encountering the same issue Yikes! Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager/prod/kafka-state)]] after [2000 ms]

I am running the kafka-manager on a node where i can confirm that i can telnet into the ZK on 2181.

I am running with /home/ubuntu/kafka-manager-1.3.3.13/bin/kafka-manager -Dconfig.file=/home/ubuntu/kafka-manager-1.3.3.13/conf/application.conf -Dhttp.port=9292 -Dkafka-manager.zkhosts=kafka1:2181,kafka2:2181,kafka3:2181,kafka4:2181 > /var/log/kafka-manager.log 2>/home/ubuntu/kafka-manager-1.3.3.13/error.log &

When i load the test cluster it's fine. Help needed please.