Rolling restart

[olegkovalenko]

MOTIVATION: Rolling restart manually:

1. stop one broker
2. update its configurations
3. start this one broker
4. repeat for next broker

Would be great to embed this steps into scheduler in single action.

PROPOSED CHANGE:

CLI changes:

add `broker restart <broker-expr>` command

command will send HTTP POST request to HTTP API, HTTP server will stop then start
each broker (one by one, sequentially, proceed to next only when previous restarted),
when timeout occurs process stops and returns JSON describing status "timeout" and
message "broker $id timeout on <stop|start>".

usage example:

  ./kafka-mesos.sh broker update 0..9 --options file:server.properties

  NOTE: allow ability to update running broker (currently only stopped broker can be udpated)

./kafka-mesos.sh broker restart 0..9 --timeout 2m

restarted brokers: ...same output as for list of brokers

  timeout output will be:

./kafka-mesos.sh broker restart 0..9

Error: broker 0 timeout on stop

help broker restart:

  ./kafka-mesos.sh help broker restart

Start broker Usage: broker start [options]

Option Description

---

--timeout Time to wait until broker restarts. Defaults to 2m.

...

cli `broker list` affected by change, because in order to communicate back to user that broker
has been modified but not restarted flag needsRestart will be added to broker model whenever
broker is modified via `broker update` cmd flag will be set to `true` (default value is `false`),
once broker stopped (task update will be received in onTaskStopped) or right before launching task 
flag `modfied` will be set to `false`. Thus if broker has been udpated flag `needsRestart` will be set to
`true`, once `onTaskStopped` is called `needsRestart` will be set to `false`.

when broker `stopped`, flag `needsRestart` aren't shown to user

./kafka-mesos.sh broker list brokers:

id: 0 active: false state: stopped (modified, needs restart)

when broker `running`, `starting`, `stopping`, `reconciling` 

./kafka-mesos.sh broker list brokers:

id: 0 active: true state: running (modified, needs restart) ...

broker JSON representation changes: added `needsRestart` flag

{ "id": "0", ... "needsRestart": true }

HTTP API changes:

add call for restart
POST /broker/restart
parameters:
  broker
  timeout

response format for success:

{ "status": "restarted", "brokers": [ { "id": "0", ... }... ] }

response format for timeout:

{ "status": "timeout", "message": "broker $id timeout on [start|stop]" }

Scheduler changes:

update attribute `needsRestart` set to `false` when 
- invoked `onTaskStopped`
- before task launch

RESULT: easy way to restart brokers (fixes #165)

[dmitrypekar]

Merged. Thank you for the update.

[olegkovalenko]

Thanks!