Improve Airflow Slack notifications

[mddilley]

We talked about improving the Airflow Slack notification behavior in our retro recently - What Could Be Improved.

Pain points:

• Too much noise, hard to tell what needs action
• Can't tell if a DAG runs frequently and will heal itself if it comes out of failure
• VPN is speed bump to checking error messages

Ideas:

• Utilizing different alert levels
  • We have an existing critical level which mentions @channel - CRIS DAG for example
• Showing the schedule (could be human readable? https://github.com/bradymholt/cron-expression-descriptor)
• Global rule to differentiate between repeated failures or initial failure
• Enhance alert messages with metadata available through Airflow metadata (or DB or API)
• Make dashboard accessible over the public internet

[mddilley]

@mddilley Break out ideas into separate issue and make this one into adding the schedule to the Slack messages https://github.com/cityofaustin/atd-airflow/blob/production/dags/utils/slack_operator.py

[mddilley]

@Charlie-Henry could you check this out and add any ideas you have to make Slack alerts better? We agreed to start with adding the cron schedule to the messages and then move to other ideas that would take more work.

[Charlie-Henry]

Maybe there's a way to configure some sort of timeout for airflow slack alerts? Like if we've already sent a slack message for a DAG in the past 3 hours we don't need to send an alert again?

Maybe an easy solution for "critical" DAGs is setting up a separate slack channel atd-airflow-critical or something and send those alerts there?