Thold plugin expose alerting information for external script

[f0cus3d]

I think much of alerting ; or good alerting, is moving away from email, would it be possible to expose threshold information ( host, value exceeded, etc) via an environment variables so that it can be passed to an external script for alerting.

Better would be to have information sent to a web hook callback.

There is https://github.com/jamesboswell/thold-slack which parses the email output from sendmail which is great but a solution like that wouldn't be needed with the above suggestion, or would make it easier to implement as not all notifications are ingested through email :)

[netniV]

The current version of THold does have the ability to run an external script on alert statuses (edit a threshold to see the new fields).

[TheWitness]

@netniV, do you think it makes sense to have a separate settings for Slack notifications? Should that setting URL and key be a part of notification lists?

[netniV]

The problem comes in that we would always have some system that people want to integrate with. As long as we can give them a method to run a custom script that provides substituted info from the threshold, that provides the greatest flexibility.

For example, the following are all things people would likely ask for integration to:

• Email
• Syslog
• SMS/Pager
• WebHooks
• Web Push Notifications
• IRC
• Slack
• Hangouts
• PushBullet
• WhatsApp
• Skype
• Helpdesk Systems
• Pushover
• Boxcar
• Others I've most definitely missed

Many of those require their own custom way of doing things. It's almost like we need a notification plugin system separate to actual plugins.

[f0cus3d]

The current version of THold does have the ability to run an external script on alert statuses (edit a threshold to see the new fields). Thanks for the info!

The problem comes in that we would always have some system that people want to integrate with. As long as we can give them a method to run a custom script that provides substituted info from the threshold, that provides the greatest flexibility.

For example, the following are all things people would likely ask for integration to:

* Email

* Syslog

* SMS/Pager

* WebHooks

* Web Push Notifications

* IRC

* Slack

* Hangouts

* PushBullet

* WhatsApp

* Skype

* Helpdesk Systems

* Pushover

* Boxcar

* Others I've most definitely missed

Many of those require their own custom way of doing things. It's almost like we need a notification plugin system separate to actual plugins.

I have often wondered why there wasn't a mature notification system already apart of cacti. Email is great but as you mentioned there are many different types of systems that people will likely want to use. I haven't tested the substituted information in a script yet but its likely the best option available at the moment.

Do you know if this is only per threshold ; meaning it can't be applied to a template for a group of devices ?

[netniV]

I have not tested out that functionality yet myself. It was introduced by @TheWitness as part of a large set of mods.

One thing we have always wanted is a more mature notification system within the core of cacti, the problem is trying to work out what is best for the core, core+plugins and/or plugins alone. THold's notify lists, for example, should really be a core feature since it could then be utilised for email.

It may even be a case of designing some form of notification system with plugin capabilities to get more info. The issue then is converting whatever message is sent into one that can be handled by the target messaging system.

[TheWitness]

We already have hooks for some 'event' types, so you could write a small webhook plugin that get's triggered as a part of each event (host down, host came up, threshold breach, threshold trigger, etc.)

The hook is there for threshold events, but not there right now for host/device events. Simple thing to add I think.

On the where to put the commands, careful thought has to be given to what is supported in the monitor plugin for example, where the concept of device criticality is introduced. @cigamit actually was going to redesign the whole plugin some years ago to flatten things out. At some point, maybe that will happen. For now, in the pending 1.2.1 release, since I'm working on this, I may just add the device hooks so that a simple plugin can be written to augment thold with Slack, etc webhooks.

[TheWitness]

Hooks added.

[TheWitness]

Also, the script notification is already there too. So, you can run any script, and the thold data is in the environment of the script. So, you can use whatever service you like. However, I'll keep this open for a time as it's a pretty good laundry list of notification types to be thought of for a future enhancement.

[gh-ghost]

I add a php file and one line in thold_function.php can let thold send $subject to slack or telegram, of course, if I add two php file and two lines in thold_function.php, i think it can send $subject to slack and telegram together, if someone interested in this?

If yes, I can show the php code I added, let you see and think how to integrate into thold, let us can use one of them or both together. (sending to 'slack' or 'telegram' or 'slack + telegram') then maybe we can enable what function we want to use, for example, maybe we just want to use telegram, don't use e-mail.

But I just tried this two IM, because they can use url to send message directly...

[netniV]

Show us what you changed as it could be useful in the future. Plugin development has been a bit short in the past few months due to lack of time even for core stuff.

[TheWitness]

Moving this to the Thold 3.0 Milestone. The real questions is how do we create the proper plugin architecture for this so that we can define a simple YAML file for each plugin to define what is a part of the configuration and what is a part of what must be provided by the plugin. Really need a spec here. Anyway. Command execution is there today and we already, at my office, use it to send to Pager Duty and Service Now. It's simply not as elegant as solutions such as Splunk and Data Dog for example.