building alerting system for grafana

[Dieterbe]

Hi everyone, I recently joined raintank and I will be working with @torkelo, @mattttt , and you, on alerting support for Grafana.

From the results of the Grafana User Survey it is obvious that alerting is the most commonly missed feature for Grafana. I have worked on/with a few alerting systems in the past (nagios, bosun, graph-explorer, etsy's kale stack, ...) and I'm excited about the opportunity in front of us: we can take the best of said systems, but combine them with Grafana's focus on a polished user experience, resulting in a powerful alerting system, well-integrated and smooth to work with.

First of all, terminology sync:

• alerting: executing logic (threshold checks or more advanced) to know the state of an entity. (ok, warning, critical)
• notifications: emails, text messages, posts to chat, etc to make people aware of a state change
• monitoring: this term covers everything about monitoring (data collection, visualizations, alerting) so I won't be using it here.

I want to spec out requirements, possible implementation ideas and their pro's/cons. With your feedback, we can adjust, refine and choose a specific direction.

General thoughts:

• integration with existing tools vs built-in: there's some powerfull alerting systems out there (bosun, kale) that deserve integration. Many alerting systems are more basic (define expression/threshold, get notification when breached), for those it seems integration is not worth the pain (though I won't stop you)
  The integrations are a long term effort. I think the low hanging fruit ("meet 80% of the needs with 20% of the effort") can be met with a system that is more closely tied to Grafana, i.e. compiled into the grafana binary. That said, a lot of people confuse seperation of concerns with "must be different services". If the code is sane, it'll be decoupled packages but there's nothing necessarily wrong with compiling them together. i.e. you could run:
  • 1 grafana binary that does everything (grafana as you know it + all alerting features) for simplicity
  • multiple grafana binaries in different modes (visualization instances and alerting instances) even highly available/redundant setups if you want to, using an external worker queue

That said, we don't want to reinvent the wheel: we want alerting code and functionality to integrate well with Grafana, but if high-quality code is compatible, we should use it. In fact, I have a prototype that leverages some existing bosun code. (see "Current state")

• polling vs stream processing: they have different performance characteristics, but they should be able to take the same or similar alerting rule definitions (thresholds, boolean logic, ..), they mostly are about how the actual rules are executed and don't change much about how rules are defined. Since polling is much simpler and should be able to scale fairly far this should IMHO be our initial focus.

  Current state

The raintank/grafana version currently has an alerting package with a simple scheduler, an in-process worker bus as well as rabbitmq based, an alert executor and email notifications. It uses the bosun expression libraries which gives us the ability to evaluate arbitrarily complex expressions (use several metrics, use boolean logic, math, etc). This package is currently raintank-specific but we will merge a generic version of this into upstream grafana. This will provide an alert execution platform but notably still missing is

1. an interface to create and manage alerting rules
2. state management (acknowledgements etc)

these are harder problems, which I hope to tackle with your input.

Requirements, Future implementations

First off, I think bosun is a pretty fantastic system for alerting (not so much for visualization) You can make your alerting rules as advanced as you want, and it enables you to fine-tune over time, backtest on historical data, so you can get them just right. And it has a good state machine. In theory we could just compile bosun straight into grafana, and leverage bosun via its REST api instead of Golang api, but then we have less finegrained control and for now I feel more comfortable trying out piece by piece (piece meaning golang package) and make the integration decision on a case by case basis. Though the integration may look different down the road based on experience and as we figure out what we want our alerting to look like.

Either way, we don't just want great alerting. We want great alerting combined with great visualizations, notifications with context, and a smooth workflow where you can manage your alerts in the same place you manage your visualizations. So it needs to be nicely integrated into Grafana. To that end, there's a few things to consider:

1. some visualized metrics (metrics plotted on graphs) are not alerted on
2. some visualized metrics are alerted on:
   • A: with simple threshold checks: easy to visualize alerting logic
   • B: with more advanced logic: (e.g. look at standard deviation of the series being plotted, compare current median against historical median, etc): can't easily be visualized nex to the input series
3. some metrics used in alerting logic are not to be vizualized

Basically, there's a bunch of stuff you may want visualized (V), and a bunch of stuff you want alerts (A), and V and A have some overlap. I need to think about this a bit more and wonder what y'all think. There will definitely need to be 1 central place where you can get an overview of all the things you're alerting on, irrespective of where those rules are defined.

There's a few more complications which I'll explain through an example sketch of how alerting could look like:

let's say we have a timeseries for requests (A) and one for errorous requests (B) and this is what we want to plot. we then use fields C,D,E to put stuff that we don't want to alert on. C contains the formula for ratio of error requests against the total.

we may for example want to alert (see E) if the median of this ratio in the last 5min ago is more than 1.5 of what the ratio was in the same 5minute period last week, and also if the errors seen in the last 5min is worse than the errors seen since 2 months ago until 5min ago.

notes:

• some queries use different timeranges than what is rendered
• in addition to processing by tsdb (such as Graphite's sum(), divide() etc which return series) we need to be able to reduce series to single numbers. fairly easy to implement (and in fact currently the bosun library does this for us)
• we need boolean logic (bosun also gives us this)
• in this example the expression only uses variables defined within the same panel, but it might make sense to include expressions of other panels/graphs.

other ponderings:

• do we integrate with current grafana graph threshold settings (which are currently for viz only, not for processing) ? if the expression is a threshold check, we could automatically display a threshold line
• using the letters is a bit clunky, could we refer to the aliases instead? like #requests and #errors?
• if the expression are stats.$site.requests and stats.$site.errors, and we want to have seperate alert instances for every site (but only set up the rule once)? what if we only want it for a select few of the sites. what if we want different parameters based on which site? bosun actually supports all these features, and we could expose them though we should probably build a UI around them.

I think for an initial implementation every graph could have two fields, like so:

warn: - expression
         - notification settings (email,http hook, ..)
crit: - expression
        -notification settings

where the expression is something like what I put in E in the sketch. for logic/data that we don't want to visualize, we just toggle off the visibility icon. grafana would replace the variables in the formula's, execute the expression (with the current bosun based executor). results (state changes) could be fed into something like elasticsearch and displayed via the annotations system.

Thoughts? Do you have concerns or needs that I didn't addres?

[kokarn]

For me number 2 seems better in that you can really minimize the number of things you have to configure for everything to work smoothly. In our current setup we would go with that.

Just so it has been said if everybody else disagrees ;)

Quick edit: Awesome slides. If the ending product comes out looking half as good as that then it's amazing.

[aferrari-technisys]

+1 I agree that internal notification handler with this integrations is perfect! for most common use case.

I'll be happy to be Beta Testing :) and slides be amazing!

[mattttt]

I think @Dieterbe's last post clears things up quite a bit, but I wanted to post this quick diagram to further clarify.

Alerting in Grafana is really two things, the self-service alert config (thanks @jaimegago, couldnt have said it better myself) and the handler itself.

We'll be shipping a Grafana alert handler, but also providing the framework to integrate with your alerting software of choice:

[BarthV]

+1 for building a kind of bridge to other alerting systems (maybe we could think about implementing some generic alerting plugin system :-) )

[grobie]

You could add Prometheus also on the "External Alert Handlers" part. The first Prometheus alertmanager version is in production in several companies and a complete rewrite is currently in the works. SoundCloud might use Grafana to configure alerts, but very certainly only if the Prometheus alertmanager is used as alert handler.

[mattttt]

@grobie, good catch, fixed in original comment.

[jaimegago]

@mattttt @Dieterbe that's great ! Looks like you're on the path of "batteries included but removable" which is IMHO the best of both worlds. Have you already thought about how you are going to pass authorization data to the alert handler? I'm thinking a story like this: As a Grafana user I'd like to be alerted via email and/or pagerduty and/or foo when (some condition built via Grafana alerting UI) happens. That user should only be able to send alerts to the notifications system he is authorized for, this is a requirement for self service and will need to be addressed someway somehow. Since Grafana 2 we have a SQL backend + users authentication/authorization with LDAP integration, so it doesn't seem too far fetch to have that capability from day one of alerting? With Sensu (which is the tool I'd be plugging in) passing the alert target (e.g. email address) via the handler should be quite straight forward, can't say about the others.

[Crapworks]

Hi all, I am happy to see that this afford is getting pushed forward, since I love the self-service alert configuration approach.

Personally, I don't need a specific alert handler. I would like to see a generic HTTP POST handler, that is just triggered as soon as an alert it thrown. I think most admins can quickly build something that is capable of accepting HTTP and then doing whatever they need to do with it (sending it to nagios, riemann, younameit). So I would be happy with an HTTP handler that sends all informations about an alert as JSON data.

Talking about alerting via grafana, are you planning to add something like flapping detection? Or is this something that the external monitoring system should take care of?

Keep up the good work guys!

Cheers

[Dieterbe]

I would like to see a generic HTTP POST handler, that is just triggered as soon as an alert it thrown. I think most admins can quickly build something that is capable of accepting HTTP and then doing whatever they need to do with it (sending it to nagios, riemann, younameit)

so if an alert fires (say "web123 has critical cpu idle!, value 1 lower than threshold 15" ) and we do an http post of that data, how would you handle that in nagios? you mean nagios would take it in as a passive service check, and then nagios would send the notifications?

Talking about alerting via grafana, are you planning to add something like flapping detection? Or is this something that the external monitoring system should take care of?

this is also something we need to think about more. This can get messy and if people use something like pagerduty or flapjack then they can use that to aggregate events/surpress duplicates so we're looking if we can avoid implementing that in the grafana handler, though we may have to. Note also that because you'll be able to set alerts on arbitrary metrics query expressions, you'll have a lot of power to take past data into account in the actual expression, and so you can create a more robust signal in the expression that doesn't change state as often.

[Crapworks]

so if an alert fires (say "web123 has critical cpu idle!, value 1 lower than threshold 15" ) and we do an >http post of that data, how would you handle that in nagios? you mean nagios would take it in as a >passive service check, and then nagios would send the notifications?

Kind of. I'm actually looking forward to grafana alerting to get rid of nagios. Using the HTTP handler, you need to configure passive checks for nagios to be able to submit the results there. But I would like to have grafana as the one source where you can configure alerting. In our case, the people who are allowed to add alerts are the actual sysadmin who would also configure the checks in nagios.

With the http handler grafana would have everything we need for that: a dashboard for real time monitoring, an API, easy alerting configuration and a http handler where we can forward alerts to our internal notifcation system.

Cheers

[csadai]

Although I can see the logic in this integration strategy, I cannot help but think it's a little bit overkill. To what I understand (and what I could read in the thread), the only reason most Grafana users keep using a standalone alerting technology is that Grafana doesn't propose one. So, wouldn't be more "lean" to focus first on the Grafana Alerting part, and develop it as a component that communicates with the rest of the stack through the API, so that other contributors would be able to mimic the behaviour and create specific adapters later?

tl;dr: by focusing on building its own "batteries" first, Grafana would have a full-featured alerting system, that can later evolve into a service for integration with 3rd party alerting tools.

[j1n6]

Minor concern, if this hasn't been addressed. The traditional alerting system does not scale well for cloud infrastructure, because resources are very dynamic (provisioned & destroyed). Alerting on metrics should support tempting or grouping feature (with exceptions override, sometimes workloads are different). A templated or grouped alert should be able to discover new group set.

[dahendel]

Thanks for the update! In my use case built-in alerting in Grafana is all I would need at this time. I have been patiently impatiently waiting for Grafana alerting.

[simmel]

As I promised in IRC, here is our use case for this:

We have a legacy Rails-app which searches our logs for patterns and has an HTTP API to answer if a certain pattern has crossed it's thresholds and thus has a status of {OK,WARNING,CRITICAL}.

Threshold can be either:

• a status of CRITICAL if pattern exists at all.
• that status is WARNING if pattern is found more than X times and status is CRITICAL if found more than Y times.
• if pattern is less than 1 hour old then the status is OK, less than 3 hours status is WARNING and otherwise status is CRITICAL.

If I understand this feature correctly, Grafana will support this usage pattern (via Logstash and Elasticsearch of course) when this feature and the Elasticsearch data source is fully implemented?

[mgravlin]

@Dieterbe @mattttt your slides and mock-ups look absolutely amazing! This is truly a game changer. To me the internal Grafana alert handler would suit our needs the best. Reasons:

• Self-service - Very important. Our users said loud and clear they want to create alerts themselves end-to-end inside Grafana.
• Unified workflow - I want to minimize moving parts not increase them. As @Dieterbe pointed out, a 3rd party alert handler would require at least 4 steps where an internal alert handler would require just 1 (maybe 2 if you need to define notification method for each threshold? - unsure from the presentation).
• Tight integration and no dependency on 3rd party alerting infrastructure.

A few concerns:

• What is the frequency checking thresholds?
• How does it handle polling frequency that is too fast for it to get the data back? Log, alerted and queued or dropped?
• For scaling, concerned that Grafana may not be able to keep up with the sheer number of checks, fast frequency and especially with latency between datasources that we will likely need to add/scale Grafana servers to support internal alerting. I know this because we need several 3rd party alert handler instances now. In this case how would we be able to seamlessly assign or queue threshold checks among a cluster of Grafana servers, especially if checks are from the same datasource? From a user experience I'd like users to seamlessly create thresholds through a load balanced, cluster of Grafana servers without having users go to a particular "assigned" instance of Grafana for a particular check.
• For notifications, would this support some kind of plugin architecture so notifications can easily be developed and integrated? In general we need something that can perform HTTP POSTs. This is most common with the likes of PagerDuty, xMatters, VictorOps, Opsgenie, etc. Each one requires a slightly different format, authentication, etc. As previously mentioned in this thread, perhaps a generic HTTP POST handler would work that would send to a custom HTTP service capable of doing whatever you want with it. Alternatively a custom script capability should work too.
• I assume thresholds could be set, retrieved and get violations through an API? I think this would be helpful

[sknolin]

I think it's ideal to be able to integrate alerting into existing alert systems. There are some tough and ugly problems like flap detection as mentioned that have been dealt with and it seems wasteful to reinvent everything from the start. I'd hate to see this buried under the weight of feature creep.

But I don't think this really needs to be a tight integration into all of these alert handlers. A good, well documented api should allow people familiar with these system to integrate with little effort. So the slide with 'grafana api -> handler' is what looks attractive to me.

Scott

[jds86930]

Hi all -- I'm coming in late to this discussion, but I have some expertise on this topic, and I'm the lead developer of one of the tools that has attempted to solve the alerting problem. Our tool, StatsAgg is comparable to programs like bosun. StatsAgg aims to cover flexible alerting, alert suspensions, and notifications & is pretty mature/stable at this point (although the API is not ready).

Anyway, some thoughts on the subject of alerting:

• Alerting by individual metrics sucks. I work at a company that manages thousands of servers, and having to create/configure/manage separate alerts for each metric series of 'free disk space %' is logistically unfeasible. Enterprise monitoring tools often tie multiple metric-series together with regular expressions (or just wildcarded expressions). StatsAgg was built on the same premise; multiple metric-series are tied together, and then the group of metrics has the alert threshold checks executed against it by a single 'alert'. At scale, this sort of capability is a necessity.
• If one accepts my previous assertion that an alerting tool should not alert off of individual metrics, then it follows that the tool must have a mechanism to quickly get a list of qualifying metrics & of the metric values. Many tools rely on the querying data-stores to get the list of metrics & metric values, and this solution frankly doesn't work very well at scale. Alerting logic, by its nature, needs to run often (every X seconds, or as each new qualifying datapoint rolls in). The datastores (graphite, opentsdb, influxdb, etc) were just not built to handle constant querying of 'give me the current list of metrics that conform to this pattern' and 'show me the last X values for these Y metrics'. They either don't have the appropriate API/query language, or they simply can't handle the load. To be clear, I'm talking about scales of running alert logic against 10,000 metric-series when there are 10,000,000 available metric-series in the datastore. This is not everyone's use-case, but it is my company's.
• I've found that tackling the problem via stream processing is the only viable way to address the issues raised by my last bullet point. That's why StatsAgg was built to sit in front of the datastore. The alerting logic can run against the metrics without touching the datastore, and the metrics just pass through to the datastore. The main conceits of this approach are that 1) newly created alerts can't/won't use old/archived metric values for alert evaluation 2) if the stream processing program (ex- StatsAgg) crashes, then datapoints don't make it into the datastore 3) metric values that are needed for alert evaluation are stored in memory, which could be a server stability concern. 4) the stream-processing program must be able to deconstruct & reconstruct the incoming metrics (which InfluxDB hasn't made easy over the last year...). Even with these conceits, this solution has worked very well for my company, and at a very large scale. At times we've had 200,000+ live metric-series, averages of 30k+ incoming metrics/sec, hundreds of alerts that evaluate thousands of metric-series, and a server running StatsAgg that barely breaks a sweat. All the while, the datastore doesn't get queried at all.

Those are the main things that I wanted to mention. There are lots of other important aspects to alerting too (notifications, suspensions, etc), but those solutions are easy to bolt on once one has the architecture of core problem solved. I realize that the scale of our needs are not the same as the average user, but hopefully you all can appreciate this perspective.

[blysik]

I'd like to suggest launching with a notification handler that can send data to Alerta: https://github.com/guardian/alerta

Alerta has a very sane REST API for receiving notifications.

[kirbmeister]

I prefer a lean grafana only implementation! I think its worth re-evaluating after everyone has had experience with this feature in the typical fantastic Grafana UX.

[007reader]

There will be many complex cases and/or custom backend systems people will want to integrate with. You can see many on this thread, mostly open source, but there are so many commercial products out there as well ! Don't bother with individual handlers - it will be a rat whole and you always will be in a catching mode

I'd strongly advise to implement only two types of handlers. One is definitely HTTP POST, it will be the most versatile and flexible tool. The other one is custom scripting, so the users can implement integration with their specific tool of choice. Plugin model is not bad, but it forces to use specific plugin language which is limiting. External scripts are better - as long as you pass to a script all the details the script can be written in any language - shell script, Python, etc.

[shanielh]

I'm with @007reader

[j1n6]

I agree. As long as common integration methods provided, custom implementation can be separate project or deployment.

For example, the recent CloudWatch release is decent, however I would love to make it as a separate project by only synchronous selected metrics to alternative storage. It will give us full retention instead of only 2 weeks data.

[Dieterbe]

hey everyone, my grafanacon presentation video is online! it's at https://www.youtube.com/watch?v=C_H2ew8e5OM i think it will clear up a lot, though as you can see. the specifics of integrations are still to be figured out and was also a topic a lot of people wanted to discuss. (though there was limited time and i asked people to continue the conversation here so everybody can participate)

@simmel yes exactly. you'd use an ES query and set a rule on that. @activars re grouping and discovery, i think a lot of that will depend on your datasource, but most common requirements should be addressed if you use something like graphite or ES who i know are very good at "auto discovering" previously unseen metrics/series/documents that match the given expression (with wildcards) for graphite or query (for ES). not sure about the other sources. your comment about needing to apply exceptions to rules is a trickier one, we'll probably need to address that at some point but I think that can wait until things are clearer and more settled down. maybe we can avoid needing that somehow. @mgravlin frequency will be a setting in the rule, dealing with too slow datasources, not sure yet. but don't do it ;-) . also handler dependent. scale-out deployment should be possible, also with included handler so pretty sure we'll look at that. but maybe not a priority for first release. yes, notification plugins will be key and we'll make sure you can use what you want/need. re api: yes :) @sknolin if i understand correctly, in your view, grafana would do the alert scheduling, execution, trigger notification hooks etc, even when using another system like nagios/bosun. then what exactly would be the role of the external system (nagios/bosun/...) . this seems also similar to what @Crapworks was talking about. @jds86930 StatsAgg looks quite interesting. I think here also an integration with grafana would make sense. I think stream processing is a valid approach that has a place as an alternative to repeated querying. but the latter is just simpler to get started with and just simpler in general I think. But both should be supported. So yes in grafana you will be able to set up patterns/queries that match a spectrum of data, and potentially cover new series/data as it becomes live. in our view though, you would just leverage whatever functionality your datasource has (for example graphite is pretty good at this with its wildcards, glob expressions etc, and elasticsearch rich data and query model), or if somebody would use grafana+StatsAgg they could just use StatsAgg to solve that. Are you saying grafana itself should do anything specific here? I think if your datasource is not fast enough, solve the datasource problem. get something faster, that has caching for metric metadata, maybe a memory server in front or stream processing. but either way as far as Grafana is concerned, not much would change that I can think of? @blysik yes looks interesting. so many alerting tools out there we should integrate with :) to be clear, do you like the idea of managing alerting rules and visualizing them in grafana the way it's been presented so far, but you want to use alerta to take care of the notifications? would alerta be the primary place where you go look at the state of your alerts (that seems like a reasonable thing to do), but want to make sure i fully understand how you see the integration.

@007reader , @shanielh , @activars just to be clear, this integration via a generic HTTP post or script, what would be the goal. to tell the external system "there's a new rule, here's the query, the thresholds, frequency, etc. now go please execute it" ? or would grafana be the thing executing the rules and then updating the external systems with new state?

[blysik]

@blysik yes looks interesting. so many alerting tools out there we should integrate with :) to be clear, do you like the idea of managing alerting rules and visualizing them in grafana the way it's been presented so far, but you want to use alerta to take care of the notifications? would alerta be the primary place where you go look at the state of your alerts (that seems like a reasonable thing to do), but want to make sure i fully understand how you see the integration.

Correct. Alerta is shaping up to be our notifications hub. All sorts of things sending alerts into it. For example: Custom scripts, Cabot, Zenoss, vCenter, and maybe Grafana. That gives ops a single place to see all the alerts. And then that is the single place which drives notifications to the oncall engineer.

[sknolin]

@sknolin https://github.com/sknolin if i understand correctly, in your view, grafana would do the alert scheduling, execution, trigger notification hooks etc, even when using another system like nagios/bosun. then what exactly would be the role of the external system (nagios/bosun/...) . this seems also similar to what @Crapworks https://github.com/Crapworks was talking about.

I guess I didn't explain well. That's not what I'd want, grafana not doing all that stuff. @Crapworks (that's fun to type) is talking passive service checks, I'd just use active polling.

So all I would want is an api where I can read grafana alerts status. External systems do everything else.

That doesn't mean if it didn't develop somehow into a great general alerting tool I wouldn't use it. Just what I'd do now.

Scott

[Dieterbe]

@sknolin

So all I would want is an api where I can read grafana alerts status.

how would that status be updated in grafana? what process would be executing alerts and updating the status in grafana?

[sknolin]

Each time it's polled grafana updates alert status, with some kind of caching interval to deal with multiple systems polling it.

[sknolin]

I see the point that this still requires grafana to do logic for the alerts and present it. So thinking about it, no I don't need alerts of any kind.

I think I could do whatever alerting needed if I were able to retrieve the current value for a metric on a graph panel. For example where we derive a rate from the sum of several counter metrics and graph it, would be nice to poll for the current value with the monitoring system. Maybe that's totally doable now and I'm just being obtuse.

Scott

[shanielh]

@Dieterbe The latter :

grafana be the thing executing the rules and then updating the external systems with new state

[jds86930]

@Dieterbe I agree that polling the datasource (Graphite, OpenTSDB, etc) using the datasource's native query syntax is simplest/easiest & is probably the quickest way to get some form of alerting natively into Grafana. For a lot of people, this sort of solution will meet their needs, and I think this is the best solution for the initial Grafana implementation (in my opinion). My main point was that there is a ceiling on alert configurability & performance that will be hard to work past with the 'polling the datasource' model.

In terms of directions Grafana could go for long-term alerting solutions, I could see a few options:

• Work with the datastore maintainers to build a faster/better purpose-built APIs for the alerting use-case. I dislike this option because many of those projects move at a slower pace (months to years) & they may not accept some/all of the enhancement requests. They'd probably also want to code in the native language of their datastores, which aren't always fast languages (ex- Graphite in python).
• Build stream-processing/caching layers for each datastore as separate raintank projects. I think this would ultimately have a better outcome than trying to coax the various datastore maintainers to build solutions for their projects. This would also allow you to continue expand on the work you're already doing (using the existing datastore query mechanisms). You could even build your own custom APIs into the stream-processing/caching layers that could simplify the Grafana's query syntax to the datastore.
• Stick with the native solution you're working toward & make it work well. 3rd party tools like StatsAgg, bosun, etc will be around for use-cases that are more demanding/specialized/complex. Having Grafana integrate with these tools would definitely be an added benefit for the user, but it would add non-trivial complexity to Grafana. That said, it looks like you may end up doing this anyway (I'm looking at 'Alerting Backends' on slide 35 of your presentation right now). I'm personally open to implementing a Grafana-friendly set of APIs in StatsAgg; we'd just have to figure how what the APIs are & get some API protocol documentation drafted. Feel free to message/email me if you'd like to discuss any of that.

[Crapworks]

Hi all,

@Dieterbe I just watched your presentation and the stuff looks awesome. I really appreciate that you are trying to build an alerting system in the "right" way, using a lot of community input! Thanks!

I also want to make my point a bit clearer, since I don't think it was obvious what I was trying to say. I DO NOT require grafana to be aware of any other monitoring system as Nagios, Icinga, Bosun, etc. I actually only need this:

• The awesome user interface you showed off in your presentation or whatever it looks like when it's completely finshed
• A generic HTTP POST handler (like some other people here also suggested) that is completely configurable (I will give you an example later on)

The event-flow I am thinking of:

• You visualize your data in grafana
• You add thresholds for alerting in grafana
• As soon as a threshold is exceeded, the HTTP POST handler is triggered
• From that point on, grafanas work is done

Like @mgravlin and @007reader mentioned, most notification and alerting services use HTTP POST, requiering diffent kinds of data. So the most generic thing I could think of, is let the user define their POST data and headers, so you can feed several systems with one handler, using different templates. Pseudo code example:

"notificator": {
    "httppost": {
        "data": {
            "host": "$hostname",
            "alert": "$alertname",
            "state": "$state"
        },
        "header": {
            "content-type": "application/json"
        }
    }
}

Giving the user enough variables to use here, will be powerfull enough to feed a ton of backends.

And again, with that kind of handler, any sysadmin with some coding knowledge can build it's own http post receiver and transform how he likes to, for example, feed backends that doesn't understand http post.

Since this is stateless, it also scales out. Just place a load balancer in front of the backend/api/whatever and you are good to go.

At least, this would solve most/nearly all of my problems ;)

Cheers

[javiermarcon]

Thanks for building this feature. Is there an aproximate release date for it?

[kokarn]

torkelo said ROUGHLY 3 months on IRC. If I understood him correctly that's a really rough estimate and should be treated as such.

[doanerock]

I am excited for the ability to do alerting with grafana. I think that this is the one feature that is keeping grafana from being the ultimate monitoring tool.

[j1n6]

If you have an early alpha/beta release, I would love to test & give early feedback with production data.

[anryko]

++

[lebernardo]

Me 2 lol

+1

Em seg, 16 de nov de 2015 às 21:03, Jing Dong notifications@github.com escreveu:

If you have a early alpha/beta release, I would love to test & give early feedback with production data.

— Reply to this email directly or view it on GitHub https://github.com/grafana/grafana/issues/2209#issuecomment-157202686.

[chaosong]

+1 If you have an early alpha/beta release, I would love to test & give early feedback with production data.

[lebernardo]

+1 me 2

2015-11-21 14:44 GMT-02:00 chaosong notifications@github.com:

+1 If you have an early alpha/beta release, I would love to test & give early feedback with production data.

— Reply to this email directly or view it on GitHub https://github.com/grafana/grafana/issues/2209#issuecomment-158661279.

[yuankui]

+1

[Dieterbe]

great to see all the +1's but FWIW they're not really needed. we already know it's the most eagerly awaited new feature, and once we have tangible progress, the code will appear in a separate branch that anyone can play with. BTW we're also bringing in more people to work full-time on grafana so stay tuned everyone :)

[vchekan]

Yes please, there are 484 people "watching" this issue. Each time you "+1" it, 484 people get email notification. Just subscribe to notification and it'll be indication of your interest to the issue.

[simmel]

+1 >; P

On Mon, 2015-11-30 at 10:33:52 -0800, Vadim Chekan wrote:

Yes please, there are 484 people "watching" this issue. Each time you "+1" it, 484 people get email notification.

[gsaray101]

Sory, I know you guys are working very hard on this. Is there any time line for first release?

[anlutro]

I would be more than happy with being able to configure alerting metrics and thresholds (either through the web interface or API) and a Grafana cronjob/daemon that checks these metrics and does a HTTP POST with JSON or invokes a script with JSON in stdout. It would be extremely simple for individuals to write a simple python script that passes this information on to Pagerduty, Slack, IRC, SMS, email or whatever.

While I would highly appreciate the convenience, I don't think it's Grafana's job to tightly integrate with third-party tools, and would rather see a minimalist implemetation sooner than a well fleshed out one later.

[bemeyert]

I completely agree with @anlutro . Please provide something simple to get started. The most interesting thing for me is to enable the people to set simple alerts themselves (self-service). Grafana shouldn't try to replace existing alerting/escalating solutions.

[elvarb]

I agree with @anlutro as well. But instead of just providing a simple api rather have the alerting part able to handle custom plugins that interact with the api. That way the base package could include email, pagerduty and a few others then the community could add to it as needed. Similar to how Logstash plugins are handled now.

[MachineCity]

+1

[mrh666]

Any news on alerting system? Any estimates?

[amitgilad3]

+1