kenahrens / nr-synthetics-plugin

Query New Relic Synthetics and publish new metrics as Plugin
9 stars 4 forks source link

Newrelic (lack of) alerts when the plugin sends no data #5

Closed lparry closed 7 years ago

lparry commented 7 years ago

This plugin looks great! It seems like it would solve a huge pain point in my team.

I've got a concern around what happens if the machine running this stops (maybe it went down or had network issues reaching newrelic), and thought maybe you might know the solution.

I've tried configuring alerts looking for "> x" Component/Overall/Failure, and "< x" Component/Overall/Success, and then killed the plugin. While the graphs in insights drop to 0 it seems like the lack of data is handled differently by Newrelic's alerting and does not trigger either.

Do you know if there's some special way that alerts can be configured so that if the plugin goes quiet the alert will still trigger? It's a little unnerving to tie site uptime monitoring to something the relies on the machine running the plugin being up, with the plugin working correctly and reporting in.

Just to be clear, I know this isn't the fault of your plugin. I was just hoping maybe you'd been down this path already and had a good fix.

kenahrens commented 7 years ago

Currently New Relic does not have a "plugin not reporting" alert. Here are some ideas:

But otherwise I don't have a way right now, once the data stops reporting in New Relic it also is not evaluated for alert conditions...

kenahrens commented 7 years ago

The NRQL Alerts feature of Alerts New is GA. This plugin will report ExtraSyntheticsInfo events when it is running so you can use NRQL Alerts to check that event and alert if it's not there. Like this:

screen shot 2017-06-21 at 4 37 43 pm