netdata / netdata-cloud

The public repository of Netdata Cloud. Contribute with bug reports and feature requests.
GNU General Public License v3.0
41 stars 16 forks source link

[Feat]: Dynamic Alert Content #934

Open jarrodlovas opened 1 year ago

jarrodlovas commented 1 year ago

Problem

No response

Description

NetData provides many difference approaches to how we can receive our alerts. A nice little feature for alert notifications would be dynamic content that allows us to customize our alerts using dynamic content flags.

For my use case, it would be really awesome to have dynamic titles. For example Alert - {alert.title}, {alert.status}. From here, using dynamic content in conjunction with our automation tools, we would be able to build out escalation workflows based on the severity of the alert and the impacted system.

Importance

really want

Value proposition

  1. Using dynamic content from subject to build out automated workflows
  2. Improve incident response time by being able to leverage dynamic content with workflows
  3. For a lot of Systems Engineers, being able to look at an alert subject to determine if it require action just from the notification subject can better allow us to determine if it requires our immediate attention or not. ...

Proposed implementation

For my implementation, It would be awesome to have this configurable from the web interface that also provides a preview or a method to test what an incoming alert would look like.

For Example:

Alert - CPU on HOST-01 has reached 90% utilization. (New - High)

Embedded Content

Alert - CPU on HOST-01 has reached 90% utilization. (Updated - Medium)

Embedded Content

Doing this way would ensure that when the notification is received by the administrator, they can quickly judge next steps or IT teams can use those subjects to build out automated workflows.

Alert - {alert.title,} (alert.status - {alert.priority})

MrZammler commented 1 year ago

Hi @RodoggA, thanks for the suggestion!

We have merged a few days ago the ability to have custom notification subjects (https://learn.netdata.cloud/docs/alerting/health-configuration-reference#alert-line-summary).

However, it is not a complete custom subject replacement as you would like. As an example of an email notification:

Warning, Ram utilization = 90.2%, on 6am

The Ram utilization part of the above is the summary field in alert configuration and can be anything you like (and can include chart labels). The rest of them though (i.e. the Warning, = 90.2%, on 6am) are currently hardcoded to construct the rest of the email subject.

We are looking for ways to further customize the notifications though, and introducing more variables like you suggested was also considered, but since we wanted the summary field to appear in the dashboard as well, we choose to keep it simpler. Perhaps we can introduce them and allow full customization if a user chooses to do so, and keep a fallback in other cases...

hugovalente-pm commented 1 year ago

hey @RodoggA , thanks for opening the FR. will move this to the netadata-cloud repo

ralphm commented 9 months ago

I think my suggestion on alert annotations in this discussion would go a long way into addressing this request.

hugovalente-pm commented 9 months ago

yes @ralphm the annotations approach suggested on that discussions seems an important foundation to address this but there is still a lot to be discussed/designed to have the full e2e approach on this from the UI, configurations from the UI, etc.

this is still in our backlog, we will for sure look at this as a whole when we pick this up