ADR 020 real-time alerts and logging draft

[marinamatic]

Draft for ADR 020 on logging and real time alerts. Would be great to get some input from all of you team leaders!!

[angelicalmcgowan]

Thanks for this! I have to admit that I find it a bit difficult to comment on this without insight into what either the logs or alerts will/already do look like. Do you by chance have any examples to share to help me get a better picture of what this will entail?

Suggestions and questions thus far (feel free to ignore):

• If alerts are to be sent to / reviewed by more audiences than those with deep understanding of the concerned code, it would be great with some human-friendly error messages instead of only numerical error codes (if this is not already in place).
• Does this ADR relate solely to the model pipeline, or also to data ingestion?
• For data ingestion and the monthly run: Could there be value in assigning 1-2 people (preferably 2 to prevent disruptions in case of illness, end of contracts etc) as managers of each ingestion script / model / ensemble / etc (regardless of who initially wrote it) to instill a better sense of ownership of the code for troubleshooting and maintenance? I know this is naturally the case for the sub-models at the moment, so this mostly relates to ingestion scripts and any other scripts that may not naturally fall under the responsibility of specific model authors. If so, it might be helpful if a directed error message that requests action is sent to those specific people, in addition to the general audience groups in the ADR above.
• Have there been any discussions on possible means to compare the data we ultimately ingest with the source data to ensure that there aren't any semantic issues in our ingestion scripts that may result in large data errors, which the input drift detection may not catch if it bases its assessments on previously ingested data? I'm not sure if this is even possible, or at all feasible, but (for context) the question comes from use cases such as us filtering out specific data from the UCDP that may, at times and by definition, exclude important events that may be relevant for us to incorporate on a case-by-case basis; and on us basing our ACLED ingestion on strings in their ever-changing API that – when they change their API – may generate non-zeros in the ingested data but still be incorrect compared to what we intended to ingest. In the case of ACLED, a manual review of their API documentation every month (...) might be sufficient to remedy this, but automated solutions are of course helpful. Wishful thinking here, but thought I'd at least raise the question for review.

[marinamatic]

@angelicalmcgowan Thanks for looking over this, great points!

I think there are still some things that need to be decided before this can be more specific (such as other logging ADRs etc), nonetheless I think your point about messages being more human-friendly is really important – they should be as informative as possible, no matter who is receiving them.

As for some of the other points you brought up, here is a bit of general clarification:

• this ADR is solely for the pipeline (since we are also separating the ingestion processes from the pipeline), however that does not stop us from developing more of such things for the ingestion side
• for the managers of scrips, models etc.: there is a lot of such things which will get resolved with the new pipeline, i.e. it will not be as modular (and all over the place), so hopefully it will not require people to manage it in that way. However, of course it is important to have designated people who are responsible (fully responsible) for both the ingestion and the runs, so that things do not fall through the cracks.
• your last point is really important, and some parts are resolved by the drift detection. It goes without saying that the more checks we implement, the less room we have for things to go wrong/unnoticed. However, this is again a bit more related to the ingestion part, rather than the pipeline itself. Thus, we should definitely have a moment to dedicate proper attention to all the ingestion issues and see how we can make it more similar to the pipeline in terms of fail-safes.

Hope this clarifies some things, and thank you again for great input!