search

cloud-gov / pages-core

cloud.gov Pages is a publishing platform for modern 21st Century IDEA websites.
https://cloud.gov/pages
Other
276 stars 68 forks source link

Consolidate SQS instances #1647

Closed jseppi closed 6 years ago

jseppi commented 6 years ago

Description of feature or bug

While working though some SQS permissions issues, @erik-burgess identified that there are three SQS instances apparently belonging to Federalist:

It seems likely that federalist-builds was intended to be our production instance, but was forgotten in some move.

We can probably swap production over to using that one and have the current prod queue (federalist-builds-cloudgov) retired.

Definition of done

Depends on what we decide to do.

After evaluating, edit this part:

Level of effort - low

Implementation outline (if higher than "low" effort):

erik-burgess commented 6 years ago

For visibility, I've already added the sqs:GetQueueAttributes permission to the unused federalist-builds queue.

wslack commented 6 years ago

Is us-west-2 in GovCloud? @erik-burgess do you have any preference where these are?

wslack commented 6 years ago

I do want them in the same spot, ideally located alongside other active services (vs being isolated)

erik-burgess commented 6 years ago

@wslack All of these are in a standard AWS account, meaning not GovCloud. I have no preference where they are located at this point in time as long as things are documented correctly in the SSP. With one of these being in a different region, it was easy to miss as it wasn't "default." If there's a business justification or technical reason behind that, it's no problem to keep them separated.

wslack commented 6 years ago

Ok. I endorse James' suggested change, though we should execute on it with a fallback plan in case something doesn't work as predicted.

jseppi commented 6 years ago

It should be just as easy as:

  1. Change the manifest.yml for federalist-builder in prod to point to the new prod SQS url
  2. Change the federalist-builder-env service to contain credentials for the new prod SQS queue
    • I don't have those credentials. Would need help from Erik to get.
  3. Change the related user-provided service variables for the federalist app to the new queue address and credentials
  4. Redeploy federalist
    • doing this first will make sure the new queue starts filling up and the old queue can drain before swapping over federalist-builder
  5. Redeploy federalist-builder after the old queue has drained (ie builds have been completed)
    • the changes won't take effect until it is redeployed, so there shouldn't even be any downtime.

We could test this all out in staging.

wslack commented 6 years ago

👍 SGTM

erik-burgess commented 6 years ago

Happy to pair any time on this.

wslack commented 6 years ago

moved to ready

jseppi commented 6 years ago

@jmhooper: want to check my steps above? https://github.com/18F/federalist/issues/1647#issuecomment-367827518

jmhooper commented 6 years ago

lgtm :+1:

jseppi commented 6 years ago

Updated list that I'm working through now:

jseppi commented 6 years ago

Ran into a snag -- the production deployer account credentials have expired, so I'm getting new ones and will update CircleCI with them.

jseppi commented 6 years ago

Also had to update the deployer account credentials in the CircleCI config for federalist-builder.

jseppi commented 6 years ago

All done! I ran a site build on federalist.18f.gov after all the switches and all looks nominal!

jseppi commented 6 years ago

After sprint review, we can ask Erik to retire the old production SQS queue.

wslack commented 6 years ago

👍

wslack commented 6 years ago

@jseppi did we make this ask yet?

jseppi commented 6 years ago

Nope, will do now.

jseppi commented 6 years ago

https://gsa-tts.slack.com/archives/C039MHHF8/p1521039132000801

erik-burgess commented 6 years ago

Unused queue (federalist-builds-cloudgov in us-west-2) has been decommissioned.