Mitigate 100% CPU edge case under ringpop churn

[Raynos]

When the ring churns we get a large amount of ringpop ring changes.

This causes updateServiceChannels() to get called at a high frequency.

This method is incredibly CPU intensive.

• https://people.uberinternal.com/~jakev/hyperbahn-outage-flame1.svg
• https://people.uberinternal.com/~jakev/hyperbahn-outage-flame2.svg
• https://people.uberinternal.com/~jakev/hyperbahn-outage-flame3.svg
• https://people.uberinternal.com/~jakev/hyperbahn-outage-flame4.svg

To avoid high CPU usage under high frequency ringpop changes we should cap the maximum number of updateServiceChannels() calls per minute.

By doing so we add a small amount of staleness but that's a far better trade off then being unavailable and burning a lot of CPU.

Right here ( https://github.com/uber/hyperbahn/blob/master/service-proxy.js#L282-L284 ) instead of scheduling a updateServiceChannels() immediately we should schedule one in the future ( 5 seconds or something ? ) and check to see how many times we've updated in the last minute, maybe have a cap of 12 updates per minute ?

[blampe]

Resolved by #307?

[Raynos]

Correct.