search

vivint-smarthome / ceph-on-mesos

Ceph on Mesos
http://vivint-smarthome.github.io/ceph-on-mesos/
Apache License 2.0
20 stars 4 forks source link

Better error message for zookeeper root node missing (when namespace specified in connection string) #4

Open chjhaijiang opened 7 years ago

chjhaijiang commented 7 years ago

Hi, I was trying to deploy this framework, but finally get the errors as below, can you please provide some tips for the failure?

============= std error logs 22:54:39.115 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 15,3 replyHeader:: 15,55834603366,-101 request:: '/frameworks/ceph-on-mesos,F response::
22:54:39.170 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 16,1 replyHeader:: 16,55834603367,-101 request:: '/frameworks/ceph-on-mesos,,v{s{31,s{'world,'anyone}}},0 response::
22:54:45.237 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 17,1 replyHeader:: 17,55834603368,-101 request:: '/frameworks/ceph-on-mesos/master-lock/locks/_c_f707a2bb-25f0-42d6-ae36-24f2e13775a7-lock-,#3139322e3136382e3131312e3434,v{s{31,s{'world,'anyone}}},3 response::
22:54:45.241 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 18,3 replyHeader:: 18,55834603368,-101 request:: '/frameworks/ceph-on-mesos,F response::
22:54:45.295 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 19,1 replyHeader:: 19,55834603369,-101 request:: '/frameworks/ceph-on-mesos,,v{s{31,s{'world,'anyone}}},0 response::
22:54:45.298 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 20,4 replyHeader:: 20,55834603369,-101 request:: '/frameworks/ceph-on-mesos/ceph.conf,F response::
22:54:45.299 [ceph-on-mesos-akka.actor.default-dispatcher-8] ERROR com.vivint.ceph.TaskActor - acquiring lock: failure org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ceph-on-mesos at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:741) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:718) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:714) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) at org.apache.curator.framework.recipes.locks.StandardLockInternalsDriver.createsTheLock(StandardLockInternalsDriver.java:54) at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:217) at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:232) at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:89) at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.internalAcquire1Lease(InterProcessSemaphoreV2.java:351) at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.acquire(InterProcessSemaphoreV2.java:284) at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.acquire(InterProcessSemaphoreV2.java:212) at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex.acquire(InterProcessSemaphoreMutex.java:46) at com.vivint.ceph.kvstore.ZookeeperStore$$anonfun$lock$1.apply(ZookeeperStore.scala:117) at com.vivint.ceph.kvstore.ZookeeperStore$$anonfun$lock$1.apply(ZookeeperStore.scala:114) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

chadleeshaw commented 7 years ago

Do you have a child in the root of zookeeper called "frameworks"? If not you need to create it.

chjhaijiang commented 7 years ago

No. Oh, I see. I thought it will be created by this frameworks. Can the error message get improved for this case? Like frameworks child does not exist or something else?

chadleeshaw commented 7 years ago

Yeah, there isn't a great error message around it. Also, the instructions could be updated to reflect this requirement. The video does mention it though.

chjhaijiang commented 7 years ago

OK, Thanks for your help.

timcharper commented 7 years ago

@chjhaijiang I think the error message and docs should improve. Thank you for reporting ! At the very least people might search for and find this ticket :)

tobilg commented 7 years ago

Wouldn't it be possible for the framework to create the node if it's not yet there?

timcharper commented 7 years ago

Yes, if we parse the connection string and connect twice, I think? The driver is a bit weird

Sent from my iPhone

On Nov 16, 2016, at 01:41, Tobi notifications@github.com wrote:

Wouldn't it be possible for the framework to create the node if it's not yet there?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

chjhaijiang commented 7 years ago

Agreed, but It may need to consider more test scenarios? E.g. an empty node or not conflict with other root nodes, etc...

timcharper commented 7 years ago

How would it conflict with another root node? A node in zookeeper can be a prefix and contain a value

chjhaijiang commented 7 years ago

what if the root node that pointed by user is an existed folder used by other frameworks? I was trying to say the case of user error. Is that possible once the root node is decided to be created by ceph-on-mesos framework?