Create alarms

[darrylmendillo]

Create alerts with event driven and batch driven applications stop creating stdout logs.

Sev 1 alerts driven by application status

Event Driven:

• [x] event_writer
• [x] vent* (Any vent applications)

Batch Driven

• [x] Palliative Care (24 hrs)
• [x] Conversation Connect (24 hrs)
• [x] CDI (24 hrs)
• [x] RIDS (15 min)

Use Alertmanager to create alerts for:

• General node down status
• Service down status
• pennsignals application performance
  • missed messages
  • error messages

[darrylmendillo]

Create Sev 1 alarms

• Establish notification methods for critical issues (slack, text, email, IS sev 1 ticket, etc)

Create recovered (back online status)

[Mdraugelis]

Set up two thresholds for Signals and VentCue systems by monitoring the primary Mongo collections. flowsheet_metrics, location_metrics, cue_metrics, mar_metrics

1. (Sev 1) Alert when Metrics is below the min_threshold for 15 minutes. min_threshold (default 1 essential)
2. (Recovery) Alert when Metrics go from below threshold to above threshold 15 minutes.

Potential approach Set up grafana to monitor the number of events written to the following collections:

mon_db.collection_names()
'flowsheet_metrics', 'location_metrics', 'cue_metrics', 'mar_metrics'

Example:

[darrylmendillo]

Grafana does not natively support MongoDB as a data source.

Solutions:

• Use 3rd party plugin for grafana (https://github.com/JamesOsgood/mongodb-grafana)
  • This requires more overhead and image customization
• Use telegraf to monitor then send to prometeus
  • Strong support and the ability to use telegraf for future connections

[Mdraugelis]

Got it. Based on the conversation this morning. I'll remove the notes on Mongo and update to "logs".

[darrylmendillo]

Created two sets of criteria for the above applications. Each event is evaluated every minute

1. Event Driven:

   
   # data_rate = events/15min
   # 1 event / 15min = 0.0011 events/sec

alert_rate = 0.001

execute every minute

data_rate = len(samples_total_15m) / (60 * 15)

if (max(data_rate) == None) or min(data_rate) < alert_rate: pending = true pending_time+= 1 else: pending = false pending_time = 0

has been pending for 15 minutes

if pending and pending_time >= 15: fire_alert()

2. Batch:

data_rate = events/24hr

1 event / 24hr = 0.000012 events/sec

alert_rate = 0.00001

execute every minute

data_rate = len(samples_total_24hr) / (60 60 24)

if (max(data_rate) == None) or min(data_rate) < alert_rate: pending = true pending_time+= 1 else: pending = false pending_time = 0

has been pending for 120 minutes

if pending and pending_time >= 120: fire_alert()