fandrei / AppMetrics

Apache License 2.0
8 stars 2 forks source link

Emergency stop button #106

Closed mrdavidlaing closed 12 years ago

mrdavidlaing commented 12 years ago

Overnight support start need a super simple "emergency stop" system to stop the Latency Collectors hitting the live API in the case of some kind of live emergency.

Currently we've given them a set of PowerShell scripts that will stop all the nodes tagged as isLatencyCollector=true; but this method is a little error prone. It also won't stop none AWS nodes (once we get those running).

Ideally we need a simple web interface they can log into, and a single "STOP" button.

fandrei commented 12 years ago

I've added enable/disable buttons here: config.metrics.labs.cityindex.com/Config.aspx It works by creating a file "~/CIAPILatencyCollectorConfig/stop.txt" under config site, and when this file is present all monitors don't execute any polling.

mrdavidlaing commented 12 years ago

Cool. How long does stopping take?

And how can you tell if it has worked?

On Tuesday, August 14, 2012, Andrei Faber wrote:

I've added enable/disable buttons here: config.metrics.labs.cityindex.com/Config.aspx It works by creating a file "~/CIAPILatencyCollectorConfig/stop.txt" under config site, and when this file is present all monitors don't execute any polling.

— Reply to this email directly or view it on GitHubhttps://github.com/fandrei/AppMetrics/issues/106#issuecomment-7719316.

David Laing Open source @ City Index - github.com/cityindex http://davidlaing.com Twitter: @davidlaing

fandrei commented 12 years ago

Clients check settings every 15 seconds, so stopping time should be something around this. You can try to disable and check if current latency report shows no data.

mrdavidlaing commented 12 years ago

http://config.metrics.labs.cityindex.com/Config.aspx gives me

  401 - Unauthorized: Access is denied due to invalid credentials.

Without asking for any.

fandrei commented 12 years ago

You should call it from the server only.

mrdavidlaing commented 12 years ago

It needs to be accessibly via the web by anyone with the correct username password.

Getting / giving the overnight support guys RDP access introduces too many challenges with firewalls & training

On 15 August 2012 04:55, Andrei Faber notifications@github.com wrote:

You should call it from the server only.

— Reply to this email directly or view it on GitHubhttps://github.com/fandrei/AppMetrics/issues/106#issuecomment-7747417.

David Laing Open source @ City Index - github.com/cityindex http://davidlaing.com Twitter: @davidlaing

fandrei commented 12 years ago

done

mrdavidlaing commented 12 years ago

Awesome; can confirm it works, and is pretty responsive

However, can we have a little more UI explaining:

(1) What this is (2) How to check whether polling is working or not.

eg: http://webwhiteboard.com#mcxrc8ht

fandrei commented 12 years ago

Done.