[Nerve] avoid zk reporter/watcher reaping on network disconnection (cont.)

[lap1817]

This PR is to follow the similar idea in the PR on Synapse (https://github.com/airbnb/synapse/pull/250).

The current logic in Nerve is that: if an error is raised from the reporter (e.g. zookeeper reporter) when ping?, report_up, report_down, the watcher thread will stop the reporter (so it tries to remove the znode). Then the Nerve main thread will reap the watcher and relaunch it.

This caused unnecessary Nerve status flipping while in short network disconnection. The ping? function on reporter is most frequently called by the watcher thread. E.g.,

• when a disconnection happens while the the reporter is ping, it raises an exception
• the reporter then tries to remove the znode which succeeded as the disconnection was short.
• after the watcher is relaunched, the znode is added again

Flipping nerve status triggers more Synapse read, which make the network saturation more likely to happen.

Changes

• reporter.ping?, report_up, report_down will catch ZK connection exceptions.
• reporter.report_up, reporter.report_down will return a bool value. -- if everything goes good, they return true -- if zk connection exception was captured, return false -- other exceptions, raise it
• service watcher now has a a circuit break. a counter keeps tracking repeated check_report failures (as returned by reporter). if that goes beyond a threshold, the run() thread will end.

Test

Rspec added/updated Tested on mango-test with simulated packet loss

Reviewers: @igor47 @jolynch @juchem @ken-experiment

[lap1817]

@juchem , I added the logic to bubble up the connection errors when they happened too many. PTAL

[lap1817]

@igor47 , PTAL. Rspec are updated to address your previous comments on using context

[lap1817]

@igor47 , addressed the comments. please take a look

[lap1817]

@igor47 , PTAL

[igor47]

the change and accompanying specs look much better, thanks