search

mesos / kafka

Apache Kafka on Apache Mesos
Apache License 2.0
414 stars 140 forks source link

Automatically remove failed broker in STAGING state #285

Closed deric closed 7 years ago

deric commented 7 years ago

After one broker crashed I had to manually remove broker and add it again to cluster. However mesos-kafka still thinks that the task is STAGING:

After restarting the scheduler there's still zombie kafka broker:

Reconciling 1/3 state of broker 0, task kafka-0-dc981a68-7d50-4d67-a7e4-d273c72653f5
Reconciling 1/3 state of broker 3, task kafka-3-e3cf24d0-6516-41f8-a65a-e68c2b4724da
Reconciling 1/3 state of broker 2, task kafka-2-ff56c1f1-7965-4c85-9e75-0219a4fdd5e0
Reconciling 1/3 state of broker 1, task kafka-1-fb5bcb20-3de2-42cf-96b9-a6b7c2bc6c3c
[statusUpdate] kafka-1-fb5bcb20-3de2-42cf-96b9-a6b7c2bc6c3c TASK_RUNNING slave:#57-S2 message:Reconciliation: Latest task state
Finished reconciling of broker 1, task kafka-1-fb5bcb20-3de2-42cf-96b9-a6b7c2bc6c3c
[statusUpdate] kafka-1-027fd5bd-ad0c-4d92-9803-8381d3a66a96 TASK_STAGING slave:#8-S20 reason:REASON_RECONCILIATION message:Reconciliation: Latest task state
Got unexpected task state: TASK_STAGING
[statusUpdate] kafka-3-e3cf24d0-6516-41f8-a65a-e68c2b4724da TASK_RUNNING slave:#8-S21 message:Reconciliation: Latest task state

Is it possible to remove (or try restarting) such stalled task? Probably after some timeout.

steveniemitz commented 7 years ago

Is there an executor running on the slave (s8 in this case) still for that task? Otherwise mesos shouldn't have it in STAGING anymore (it should have transitioned to TASK_LOST or TASK_FAILED).

Also, what version of kafka-mesos are you running? As well as your mesos version. It looks like an older version of the framework, 0.10.0 might handle this, but a quick glance looks like it might not handle TASK_STAGING for an unknown broker either (relevant code here ).

deric commented 7 years ago

Yeah you're right, Mesos should remove the task. Mesos agent on that node has been restarted and the old task was not removed. We're running on:

mesos 1.1.0-2.0.107.debian81
kafka 0.9.0.1
kafka-mesos 0.9.5.1

We're using the latest stable version. The relevant code doesn't seem to handle such case.

deric commented 7 years ago

After restarting Mesos master the STAGING task was removed (probably we should leave this to Mesos). I've restarted kafka scheduler, but it's still in quite strange state. Any attempt for rebalance fails:

Error: java.io.IOException: 400 - rebalance is already running

Is this possible to fix without shutting down whole kafka cluster?

steveniemitz commented 7 years ago

You'll need to figure out why your rebalance is stuck (check out the kafka controller's logs).

If also else fails you can delete the rebalance znode in zookeeper and restart whichever node is the current controller to force stop it.

deric commented 7 years ago

Ok, thanks. I ended up with deleting znode

delete /kafka/admin/reassign_partitions

and now rebalance is working again. I'm closing this task because the case should be handled by Mesos master.