search

hackoregon / civic-devops

Master collection point for issues, procedures, and code to manage the HackOregon Civic platform
MIT License
11 stars 4 forks source link

Convert 2017 Housing API from EC2 to Fargate #265

Open MikeTheCanuck opened 4 years ago

MikeTheCanuck commented 4 years ago

Addresses #244 for the 2017 Housing API. Mirrors the work in #259 and #260, and implements similar changes as hackoregon/hackoregon-aws-infrastructure#84. Uses the migration procedure documented here.

Acceptance Criteria

Tests that will confirm the container has successfully migrated:

  1. CloudFormation will report UPDATE_COMPLETE for the enhanced stack
  2. ECS will report (a) "Launch Type FARGATE" for the service, (b) 1 Task Running at the Task level, and (c) at the Task detail level a "Started at" timestamp that is at least 5 minutes into the past (which indicates that the container stayed RUNNING long enough to pass the ALB health check cycle)
  3. latest log in CloudWatch group for the associated service will show entries of the form 10.180.9.210 [09/Aug/2019:20:52:04 +0000] GET /housing/ HTTP/1.1 200 23929 - ELB-HealthChecker/2.0 0.103165
  4. browser requests to https://service.civicpdx.org/housing/prodvscost/ will display a Swagger-schema-prettified response with pages of records in JSON format
  5. latest CloudWatch logs will display a recent web request to /housing/prodvscost/ with a 200 response code
MikeTheCanuck commented 4 years ago

Container/API upgrades

The 2017 API containers have drifted pretty far from the 2018/2019 API configuration and CI/CD setup, so it is first necessary to upgrade the 2017 API, container and Travis configurations before the CloudFormation migration will ever succeed:

  1. Adapt the changes here to the current project: https://github.com/hackoregon/team-budget/pull/173
  2. If the container still won't deploy to EC2, check for additional tips here: https://github.com/hackoregon/civic-devops/blob/master/docs/HOWTO-Update-an-API-repo-to-use-Fargate.md
  3. If things go really haywire, read up on #158 and see if there's other corrections needed.

CloudFormation migration strategy

Follow the steps outlined here: https://github.com/hackoregon/civic-devops/issues/260#issuecomment-520180942

The actual changes to master.yaml ultimately result in:

  1. adding a 2017Housing Resource to the # 2017 API Services - Fargate section (currently around here)
  2. removing the HousingService Resource block from master.yaml.

Travis updates

The newly-created Resource will have a new Service name in the ECS interface (e.g. hacko-integration-2017Budget-YOMCBA6UTARG-Service-1F3RKQ7TN48S) which will need to be used as the ECS_SERVICE_NAME variable in the associated Travis repo's Settings: https://travis-ci.org/hackoregon/housing-backend/settings

Until the Travis configuration knows of the new ECS Service Name, any deployment via Travis will fail its attempt to deploy the Travis-build container image to the old Service (which will be gone as soon as the CloudFormation migration strategy is underway).

Delete the old value and create a new one (and feel free to make it non-secret - it's more important to be able to review the Travis env var settings when troubleshooting future problems, than it is to protect configuration parameters like this that aren't really secret. NOTE: you will need elevated privileges in the associated GitHub repo to be able to access and update the Travis repo Settings.