Update our EC2 launch configs for ECS to use latest AMI for modern agent #108

Closed ghost closed 3 years ago

In GitLab by @infinitewarp on Aug 12, 2020, 13:42

Important note!

This work is effectively blocked on https://github.com/cloudigrade/cloudigrade/issues/738 because 738's implementation might handle this issue "for free".

Summary

As a cloudigrade sysop, I want my houndigrade ECS cluster EC2 launch configuration to use modern versions of available ECS AMIs so that I am more confident about having a supported and working Docker agent for ECS.

In our recent investigations, we discovered that most (all?) of our existing ECS setups were using very old AMIs, and they presented warnings in the AWS web console that they were not running recent versions of their agents. We suspected this could contribute to some of our mysteriously failing inspection tasks. This means we want to periodically get the latest supported ECS EC2 AMI and update our launch definitions to use that latest AMI.

To perform this update, we should consider implementing an independent scheduled process that is responsible for checking for recent AMIs and updating the various things in AWS to make ECS use that latest AMI. Consider making this the responsibility of the "reaper" in GitLab CI. Consider renaming "reaper" to something more appropriate like "janitor"?

Let's do this with a GitLab CI pipeline that behaves like our old cloudigrade release pipeline:

CI and QA automatically get "latest"
- updating stage is a manual step that takes the version used in CI/QA
- updating prod is a manual step that takes the version used in stage

Acceptance Criteria

[ ] Verify a periodic job exists to perform the following actions.
[ ] Verify we have new AWS EC2 Launch Configuration with the newly chosen AMI ID.
[ ] Verify we have updated the AWS EC2 Auto Scaling group with the new Launch Configuration's ID.
[ ] Verify that houndigrade actually runs through the normal inspection process.
[ ] Verify the job applies changes to:
- [ ] prod ECS cluster
- [ ] stage ECS cluster
- [ ] qa ECS cluster
- [ ] ci ECS cluster
- [ ] NOT each of our personal ECS clusters
[ ] QE to add some test case to verify that the updated reaper/janitor actually gets a recent AMI ID.
- This might mean to check and compare two things: the current latest AMI ID from AWS ECS and the currently configured value assigned to the Launch Configuration.
- This is not a fully-automated test. This is a one-time verification check that may be manual if that's easier.
[ ] QE to run manually at least once the IQE test suite to verify inspection still works after at least one run of the new reaper/janitor update process.
- This is not a fully-automated test. This is a one-time verification check that may be manual if that's easier.

Assumptions and Questions

Keep in mind that we may need to manually coordinate config changes in app-interface if the Auto Scaling group name changes!

In GitLab by @infinitewarp on Aug 27, 2020, 10:53

changed the description

In GitLab by @infinitewarp on Sep 3, 2020, 10:41

changed the description

In GitLab by @infinitewarp on Sep 3, 2020, 10:58

changed the description

In GitLab by @infinitewarp on Sep 3, 2020, 10:59

changed the description

In GitLab by @infinitewarp on Sep 9, 2020, 16:30

changed the description

In GitLab by @infinitewarp on Sep 28, 2020, 15:41

marked this issue as related to cloudigrade#732

Included in 0.4.2 release with the rest of cloudigrade/cloudigrade#738 changes.

Since we closed https://github.com/cloudigrade/cloudigrade/issues/738, I think we're clear to close this one too. Thanks @katherine-black ! 🙂

cloudigrade / houndigrade