Progress for restart

[olegkovalenko]

MOTIVATION Node restart takes significant amount of time.

Default timeout is 5m per node and it is unclear for the user what is the current overall progress of the operation (consider we are restarting 10 nodes, for instance).

It would be good to print some progress in CLI like:

#./dse-mesos.sh node restart 0..9
stopping node 0 ... done
starting node 0 ... done
...
stopping node 9 ... done
starting node 9 ... done

PROPOSED CHANGE

Definition of stopping node for progress is transition of node state from starting, running, stopping to starting.

Definition of starting node for progress is transition of node state from starting to running.

Use streaming (transfer-encoding chunked) to display progress, however it requires fundamental backward compatible changes for:

• HTTP Server /node/restart
• CLI should be able to handle chunked responses

when HTTP Server handle request for /node/restart and parameter progress presents, server writes "stopping node $id ... " before stopping node and when it has been stopped writes "done", before starting node it writes "starting node $id ... " and when it is running writes "done", on client side CLI, handles lines as they arrive and passes them to callback function, when chunk starts with "{" it parses it and returns control to caller (accumulating everything after "{" into single string and return it). Some how easier implementation on CLI side and deadlock due to ordering of starting/stopping nodes impossible however requires backward compatible changes on HTTP Server and CLI part.

CLI changes:

• handle chunked responses (since there are no direct access to underlying stream thus handle lines)

HTTP Server changes:

• respond in chunks when progress parameter preset for /node/restart API call
• each chunk (line) will contain item of progress, the last chunk will be JSON (describing error or node list)

NEW OR CHANGED PUBLIC INTERFACES:

• added parameter onChunk: String => Unit for net.elodina.mesos.dse.Cli.sendRequest

    private[dse] def sendRequest(uri: String, params: Map[String, String], urlPathPrefix: String = "api", parseJson: Boolean = true, onChunk: String => Unit = null): Any

• added parameter progress to /node/restart API call

MIGRATION PLAN AND COMPATIBILITY:

• changes are backward compatible

REJECTED ALTERNATIVES:

• on CLI side poll /node/list while restart in progress and watch for node state change, once changed write to console (deadlock possible)
• simplify output to just "restarting node $id ... " and once node changes its state from whatever to running and having different task id output "done" (deadlock possible due to changed ordering)

RESULT: having better user experience, ability to view progress of restart

[dmitrypekar]

Great PR. Everything works perfectly, except 3 small points, which I have commented. Please fix them for merge.

[dmitrypekar]

Merged. Thank you for great addition.

[olegkovalenko]

@dmitrypekar thanks for review and valuable feedback!