search

vivint-smarthome / ceph-on-mesos

Ceph on Mesos
http://vivint-smarthome.github.io/ceph-on-mesos/
Apache License 2.0
20 stars 4 forks source link

Framework gets stuck in crash loop #12

Closed timcharper closed 1 year ago

timcharper commented 7 years ago

Ceph on mesos got in a crash loop from which it couldn't recover. Here's the logs:

java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
    at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:08.941 [ceph-on-mesos-akka.actor.default-dispatcher-4527] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-187/flow-2178-0-unknown-operation#1053547476]] terminated abruptly
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4521] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4521] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-188/flow-2180-0-unknown-operation#-1470175774]] terminated abruptly
22:27:10.033 [ceph-on-mesos-akka.actor.default-dispatcher-4527] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:10.033 [ceph-on-mesos-akka.actor.default-dispatcher-4527] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:11.139 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:13.870 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59393"

22:27:13.871 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59393"

22:27:14.903 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:15.871 [ceph-on-mesos-akka.actor.default-dispatcher-4472] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59394"

22:27:15.871 [ceph-on-mesos-akka.actor.default-dispatcher-4472] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59394"

22:27:15.905 [ceph-on-mesos-akka.actor.default-dispatcher-4472] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:16.880 [ceph-on-mesos-akka.actor.default-dispatcher-4509] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59395"

22:27:16.880 [ceph-on-mesos-akka.actor.default-dispatcher-4509] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59395"

22:27:16.907 [ceph-on-mesos-akka.actor.default-dispatcher-4509] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:24.481 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:25.051 [ceph-on-mesos-akka.actor.default-dispatcher-4480] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
    at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:25.051 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-189/flow-2182-0-unknown-operation#116359274]] terminated abruptly
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4508] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4480] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-190/flow-2185-0-unknown-operation#590294963]] terminated abruptly
22:27:26.232 [ceph-on-mesos-akka.actor.default-dispatcher-4527] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:26.232 [ceph-on-mesos-akka.actor.default-dispatcher-4527] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:33.901 [ceph-on-mesos-akka.actor.default-dispatcher-4530] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59397"

22:27:33.901 [ceph-on-mesos-akka.actor.default-dispatcher-4530] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59397"

22:27:33.931 [ceph-on-mesos-akka.actor.default-dispatcher-4530] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:37.828 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:41.251 [ceph-on-mesos-akka.actor.default-dispatcher-4531] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
    at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:41.251 [ceph-on-mesos-akka.actor.default-dispatcher-4531] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-191/flow-2187-0-unknown-operation#-612276060]] terminated abruptly
22:27:41.253 [ceph-on-mesos-akka.actor.default-dispatcher-4532] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:41.253 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:41.253 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:41.254 [ceph-on-mesos-akka.actor.default-dispatcher-4527] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-192/flow-2189-0-unknown-operation#-1953777267]] terminated abruptly
22:27:42.312 [ceph-on-mesos-akka.actor.default-dispatcher-4535] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:42.312 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:44.920 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59399"

22:27:44.920 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59399"

22:27:44.947 [ceph-on-mesos-akka.actor.default-dispatcher-4534] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:45.920 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59400"

22:27:45.920 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59400"

22:27:45.948 [ceph-on-mesos-akka.actor.default-dispatcher-4534] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:46.920 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59401"

22:27:46.920 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59401"

22:27:47.951 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:51.161 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:57.330 [ceph-on-mesos-akka.actor.default-dispatcher-4535] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
    at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:57.330 [ceph-on-mesos-akka.actor.default-dispatcher-4535] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-193/flow-2191-0-unknown-operation#-260710826]] terminated abruptly
22:27:57.333 [ceph-on-mesos-akka.actor.default-dispatcher-4535] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:57.333 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:57.333 [ceph-on-mesos-akka.actor.default-dispatcher-4501] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:58.503 [ceph-on-mesos-akka.actor.default-dispatcher-4501] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:58.503 [ceph-on-mesos-akka.actor.default-dispatcher-4501] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:03.950 [ceph-on-mesos-akka.actor.default-dispatcher-4536] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59402"

22:28:03.950 [ceph-on-mesos-akka.actor.default-dispatcher-4536] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59402"

22:28:03.972 [ceph-on-mesos-akka.actor.default-dispatcher-4536] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:04.498 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:28:13.521 [ceph-on-mesos-akka.actor.default-dispatcher-4533] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
    at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:28:13.521 [ceph-on-mesos-akka.actor.default-dispatcher-4533] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-195/flow-2195-0-unknown-operation#1011439711]] terminated abruptly
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4533] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4531] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4536] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-196/flow-2198-0-unknown-operation#-1031974059]] terminated abruptly
22:28:14.673 [ceph-on-mesos-akka.actor.default-dispatcher-4538] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:14.673 [ceph-on-mesos-akka.actor.default-dispatcher-4538] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:14.961 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59403"

22:28:14.961 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59403"

22:28:14.989 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:15.960 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59404"

22:28:15.960 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59404"

22:28:15.991 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:17.832 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:28:17.971 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59406"

22:28:17.971 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59406"

22:28:17.993 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:29.690 [ceph-on-mesos-akka.actor.default-dispatcher-4507] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
    at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
    at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:28:29.691 [ceph-on-mesos-akka.actor.default-dispatcher-4507] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-197/flow-2200-0-unknown-operation#-1749189163]] terminated abruptly
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4538] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4538] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4480] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4538] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-198/flow-2202-0-unknown-operation#-1071175607]] terminated abruptly
22:28:30.743 [ceph-on-mesos-akka.actor.default-dispatcher-4539] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:30.743 [ceph-on-mesos-akka.actor.default-dispatcher-4539] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:31.179 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms

Unsure of the reason. Killing the framework resolved it. Potential solution is to suicide after 10 failed retries?

ssiahetiong commented 7 years ago

I was able to fix this issue by manual deleting the file on exhibitor: ceph-on-mesos -> master-lock -> locks -> lock file

Melozzola commented 6 years ago

I have the same problem, but deleting the lock file file didn't help. Did you get to the bottom of the issue? Is there somebody investigating and working on a fix?