Solr on ECS - Cloudwatch - SNS - Lambda Restarts

[nickumia-reisys]

Related to

• https://github.com/GSA/data.gov/issues/3904
• https://github.com/GSA/catalog.data.gov/pull/534
• https://github.com/GSA/data.gov/issues/3714

Important Changes:

• [BREAKING CHANGE] Solr Follower naming was updated to follow consistency pattern
• Slack Webhook Url was manually setup in AWS Accounts (ssb-production and ssb-development, ssb-staging was not set up)

References:

• Cloudwatch Alarms
  • Terraform resource sns_topic
  • AWS Docs - Cloudwatch ECS Metrics
  • AWS Docs - Cloudwatch Metrics Expressions
  • Terraform cloudwatch_metric_alarm
• Lambda Functions
  • Terraform sns_topic_subscription
  • Terraform lambda_permission
  • Terraform lambda_function
  • AWS Support - Debug Lambda Function
  • AWS Docs - Passing Python Code as a Zip to Lambda
  • AWS Docs - Python Zip Example + External Libraries
  • Terraform aws_secretsmanager_secret_version
  • ~AWS Docs - Lambda Building on top of Base Image~
  • ~Example AWS Lambda Python Docker Image~
  • ~AWS Docs - Lambda Python Docker Image~
• Slack API
  • Slack API Docs - Build an App (interactive)
  • Slack API Docs - Formatting
  • Slack API Docs - Webhooks

[nickumia-reisys]

Proof that the alarm was triggered and Solr was restarted:

Proof that the alarm went back to OK after restart: