Automatic monitoring

[mankoff]

To support auto-updating with minimal human interaction, we should implement some level of automatic monitoring.

• Implement some algorithm that looks at the last timestamp relative to previous ones, and flags out of limit values
  • e.g. if latest discharge is > 2 sigma from historical, flag it
• Mail graphic and/or tabular summaries to key personnel
• Delay pushing (or publishing?) the update to the Dataverse by ~24 hours, giving key personnel an opportunity to stop the update if necessary.

[mankoff]

Example algorithm: If the changes in the most recent update are more than 20 % over the max historical (last 3 year) change, flag it.

import xarray as xr
import numpy as np

ds = xr.open_dataset('./out/gate.nc')
df = ds['discharge'].T.to_dataframe().unstack()

past = df[df.index > df.index[-1] - pd.DateOffset(years=3)]

flag = np.abs(past.diff().iloc[-1]) > np.abs(past.diff().iloc[:-1].max()*1.20)
flag = flag[flag == True]

if flag.size == 0:
    print("No flags")
else:
    df[flag.index].loc['2016':].plot(drawstyle='steps-post')