Issue: Draft ADR 020 - Log Files and Real-Time Alerts

[Polichinel]

Issue: Draft ADR 020 - Log Files and Real-Time Alerts

Description
Create an Architectural Decision Record (ADR 020) to outline the strategy for handling log files and implementing real-time alerts for the pipeline. This ADR will define alert channels, logging levels, detail requirements, and notification flow to ensure timely and relevant error handling for critical issues.

Objectives and Requirements

1. Define Alert Channels:

   • Identify and outline key channels (e.g., Slack, email, Weights & Biases, Prefect) for distributing alerts.
   • Specify the types of errors each channel should handle and the target audience for each.

2. Set Logging Levels:

   • Establish logging thresholds (INFO, WARNING, ERROR, CRITICAL) and specify which levels are sent to each channel.
   • Define what constitutes a “critical” vs. “non-critical” alert to inform alert prioritization.

3. Determine Detail Levels for Alerts:

   • Define the level of detail required in alerts for each channel, ensuring both immediate response capabilities and sufficient context for follow-up.
   • Examples can include:
     • Slack: Quick summaries and links to full logs.
     • Email: Expanded summaries with context and resolution steps.
     • W&B and Prefect: Task-level detail and performance metrics.

4. Review and Revise:

   • Review the ADR draft with relevant team members to align on alerting needs and logging details.
   • Refine based on team input to produce a final version for approval.

Tasks

• [x] Draft the initial ADR 020 based on the outline above, incorporating example alert flows and log detail levels.
• [ ] Review and gather feedback from key stakeholders (e.g., Marina and team leads).
• [ ] Finalize the ADR with team feedback, preparing it for submission as a pull request.

Labels
ADR, logging, alerting, error-handling, documentation