hackoregon / civic-devops

Master collection point for issues, procedures, and code to manage the HackOregon Civic platform
MIT License
11 stars 4 forks source link

Create a notifications stream to publish via Slack, email or other channels outside of AWS console #154

Closed MikeTheCanuck closed 5 years ago

MikeTheCanuck commented 6 years ago

Problem Statement

Projects who publish updates to their APIs have good visibility via Travis into whether their containers have been successfully pushed to ECS.

However, it's not uncommon, especially when there aren't good tests validating container health in Travis, that the pushed container isn't able to start, or doesn't publish the expected routes, or otherwise doesn't survive the ALB health check, and there's no visibility for API developers into why their changes haven't been pushed to production - they have to contact the few members of Hack Oregon's DevOps squad and ask them to look into where the problem lies.

For DevOps squadroneers, we're trolling through various levels of AWS consoles in ECS, or CloudWatch logs, until we see where there's some signal to tell us what caused the container to fail to survive to "productionalization". (That's a word, right?)

Proposal

Create a channel for publishing the ongoing success/failure status of Travis "deploys" (task definition updates), and what details can be pulled out of ECS (and later from CloudWatch as well) to indicate why something is failing.

Possibilities to explore include: https://github.com/segersniels/aws-custom-slack-notifications#pushing-ecs-task-state-changes-to-slack https://aws.amazon.com/blogs/compute/monitor-cluster-state-with-amazon-ecs-event-stream/