Implementation: Threshold Analysis (Alerts) & Notification

[Yapping72]

Will be using Telegram bot for notification, email service requires SMTP server. Setting up our own personal SMTP server is deemed to be too time and labour intensive, with very high chance of emails being directed to draft / spam. Commercially available SMTP services e.g, SendGrid has strict restrictions i.e., type of domain and only allows up to 100 emails per day. Following discussion with team and ITP advisor, telegram will be the means of notification.

[Yapping72]

Telegram bot created, added to group and verified that notifciations can be sent to group

[Yapping72]

Created alerting rules for temperature and co2 sensor readings.

TODO:

1. get desired threshold temperature from crop scientist, threshold currently at 19 degrees celsius for temp.
2. create alerts for other sensors

[Yapping72]

Configuring alerts for when light turns on or off. For rack 1, when growlight first switches on difference between latest and previous point will be approximately +390W. When it switches off it will be approximately -390W. This holds true for past data points.

Flux query will be used to determine the change in power consumption for each hourly interval. If change >= +360 W growlight went from off to on state. If change <= -360 W growlight went from on to off state. Using a 30W buffer since sometimes the change ranges from +391 to +393. This way alerts will only be sent when growlight turns on and when growlight turns off. This logic will apply to all other growlight power consumption alerts.

[Yapping72]

Created rules

1. CO2 Evaluation Group: Evaluated every 30minutes, sends an alert if rack CO2 levels above 1500ppm.
2. Growlight Power Consumption Evaluation group: Evaluated every 1hr, sends an alert if growlight switched on or off.
3. Temperature Evaluation Group: Evaluated every 30minutes, sends an alert if rack temp > 19.
4. Water Flow Evaluation Group: Evaluated every 30minutes, sends an alert if water flow is stopped / low , water flow reading <50.
5. Water Pump Power Consumption Evaluation group: Evaluated every 1hr, sends an alert if water pump power consumption < 5 W.

TODO:

1. Monitor if alerts have been configured correctly.
2. Monitor if alerts are being triggered for the correct data-point. Alert interval is started based on when the rule was created, i.e., if created at 4pm with 30min evaluation interval next evalutation will be at 430pm. However, if our sensor hasnt sent a new data by 430pm the evaluation will still be for the old data.

   TOFIX: Align the start time of each alert to when new data will be recieved. i.e., sensor sends data at 432pm, alert should start only from 432pm. Intervals can stay the same.

3. Obtain threshold values for dangerous co2 levels (lit review) and high-temp alerts.

[Yapping72]

TOFIX:

1. Align the start time of each alert to when new data will be received. i.e., sensor sends data at 432pm, alert should start only from 432pm. Intervals can stay the same.
2. Water pump disconnected alert not working.
3. Whenever alert is triggered, the alert should also be reflected in the dashboard.

[Yapping72]

1. Verified That Growlight switched off alerts can be triggered.

2. Verified that water pump disconnected alerts can be triggered.

FIXED:

• Aligned start times of alerts to when sensor sends data.
• Removed FOR-evaluation times.
• Changed influxdb query which should resolve bug in water-pump disconnected alert. Observed that alert would return no-result state for alert, even when data point exists, not sure if caused by bad influx db query.

To do:

• Monitor if growlight switched on alert can be triggered.
• Monitor if water pump power disconnected alert returning [no-data] states.
• Get threshold values for high-temp, setting threshold value at 19 results in alerts being triggered every 30minutes during the day.
• Align dashboard and telegram alerts.

[Yapping72]

TODO:

• Change threshold of temperature alerts to 26.
• Attempt to create own AlertManager to manage growlight switch on/off alerts. Growlight on/off alerts via Grafana has unexpected behavior. For racks 2 and 3. Rack 1 alerts can be triggered correctly via grafana, rack 2 alerts sent at wrong time, while rack 3 alert switched on alert cannot be triggered. All alerts configured using the same query hence not sure why not working.

[Yapping72]

Implemented own alerting module in python. This is in hopes of circumventing grafana alerting limitations. todo: configure nodata handling as per prof fauzis guidance i.e., send NODATA alerts only when X-alerts missed.

[Yapping72]

Custom telegram alerts working can detect for all racks.

[Yapping72]

Alerts working as intended, triggered when difference in power consumption greater than 0. Not reliant on fixed growlight times.

e.g., Rack 2,3 - 7am ON, Rack 1- 8 am ON. (19 July)

Can also capture and report unexpected behaviour, Rack 2 switched on at 7am, off at 8am and switched back on at 10am.