Migrate dashboard to cloud.gov

adborden commented 2 years ago

User Story

In order to stop maintaining the FCS deployment, the data.gov team wants production service to be directed to our deployment on cloud.gov.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[x] WHEN we visit dashboard.data.gov in our browser \ THEN we see the expected pod-dashboard output \ AND we see requests in the pod-dashboard app's logs on cloud.gov

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!] This change will migrate us away from our old environment, which is harder to maintain and for which there are more things that we have to look after. The new environment has already been pen-tested and ATOd so we think it's going to be net win on attack surface overall.

Launch plan

Pre-launch

In the days leading up to the launch, these tasks should be completed:

[x] Review application configuration and all values are correct
[x] Ensure CAA record for letsencrypt.org exists for domain
[x] Familiarize yourself with the external domain service
[x] Open a DNS ticket for GSA to add _acme-challenge records for DNS validation (RITM0904225)
[x] Create domain cf create-private-domain gsa-datagov dashboard.data.gov
[x] Map the domain route to the application in the prod space cf map-route dashboard dashboard.data.gov
[x] Create the external domain service in the production space: cf create-service external-domain domain-with-cdn <app>-cdn -c '{"domains": "<domain>"}'
[x] Script datastore migrations them for repeatability
[x] Rehearse the launch and rollback plans below in staging, document any commands verbatim
[x] Open ticket with GSA DNS to create new CNAME record dashboard.data.gov CNAME dashboard.data.gov.external-domains-production.cloud.gov (RITM0905723)
[x] Open a ticket with GSA DNS to reduce TTL to 300 and coordinate the DNS switch over and update the DNS labs.data.gov CNAME labs.data.gov.external-domains-production.cloud.gov on launch day
[x] Confirm the production-ready checklist is complete
- [x] Application has a manifest.yml and is deployed from CI
- [x] Production application is running with at least 2 instances
- [x] Datastores are running "dedicated" or production grade plans
- [x] cf ssh access is disabled in the production space
- [x] NR Synthetics alerts are configured for this domain and reporting to #datagov-alerts

Launch

Tasks to be completed at the time of launch.

[x] Merge production config changes
[x] Check the application route is setup curl -v --fail https://dashboard.data.gov
[x] Disable app restarts in cloud.gov
[x] Disable craws in cloud.gov
[x] Disable crawls in FCS (from jumpbox pipenv run ansible-playbook dashboard-web.yml -e '{"crons_enabled": false}' -i inventories/production
[x] Migrate data services as described in these gists. These can happen in parallel.
- Migrate dashboard-db (~40 min)
- Migrate dashboard-s3 (9 hours with --clear, likely ~15 minutes without)
[x] Have GSA update the DNS labs.data.gov CNAME labs.data.gov.external-domains-production.cloud.gov

In the event a rollback is necessary, apply these tasks.

[ ] Revert DNS changes, should point to original values in FCS
[ ] Ensure application services are started and running in FCS

Post-launch

Once the launch is stable and successful, these tasks should be performed.

[x] Re-enable app restarts in cloud.gov
[x] Resume dashboard crawls in cloud.gov
[x] Create issues for decommissioning the FCS environment (REQ0654065)
[x] cf ssh access is disabled in the production space
[x] Consider adding an external domain-with-cdn service for staging environment and create an issue (https://github.com/GSA/datagov-deploy/issues/3580)
[x] Remove Synthetics check for dashboard-prod-datagov.app.cloud.gov
[x] Submit pull request or create issue to remove dashboard code from datagov-deploy
[x] Open DNS ticket to reset TTL to 3600
[x] Delete any unused service-keys

adborden commented 2 years ago

Hit a snag when creating the data.gov domain in cloud.gov. We're working through this with the cg-support team.

$ cf create-domain gsa-datagov data.gov
Creating private domain data.gov for org gsa-datagov as aaron.borden@gsa.gov...
The domain name "data.gov" cannot be created because "staging.api.data.gov" is already reserved by another domain
FAILED

adborden commented 2 years ago

Backing up the staging database took ~1m and restoring took ~9m.

   2021-11-30T19:25:00.23-0800 [APP/TASK/dashboard-backup/0] OUT restoring dashboard-db (mysql) from /dashboard/staging/dashboard-db-20211201-023643-migration.sql.gz...
   2021-11-30T19:34:15.32-0800 [APP/TASK/dashboard-backup/0] ERR real   9m15.091s
   2021-11-30T19:34:15.32-0800 [APP/TASK/dashboard-backup/0] ERR user   0m59.095s
   2021-11-30T19:34:15.32-0800 [APP/TASK/dashboard-backup/0] ERR sys    0m6.885s
   2021-11-30T19:34:15.32-0800 [APP/TASK/dashboard-backup/0] OUT ok
   2021-11-30T19:34:15.32-0800 [APP/TASK/dashboard-backup/0] OUT Exit status 0

adborden commented 2 years ago

Migrating S3 is going to be a pain.

Total Objects: 91405
   Total Size: 351.6 GiB

adborden commented 2 years ago

Migrating the S3 bucket on staging took about 8 1/2 hours.

real    505m10.448s
user    52m59.490s
sys     43m49.413s

adborden commented 2 years ago

DB backup on FCS production:

real    4m31.271s
user    5m29.634s
sys     1m4.459s
dashboard-db (prod) backup complete.

adborden commented 2 years ago

DB restore in cloud.gov production

real       32m29.414s
user       3m33.197s
sys        0m28.964s

adborden commented 2 years ago

Production s3 migration:

real    647m47.594s
user    67m56.898s
sys     43m5.665s

adborden commented 2 years ago

At this point, we're waiting on GSA to create the dashboard.data.gov record and I'm scheduled with Shaw to update the labs.data.gov record on Monday.

adborden commented 2 years ago

Removed service keys from dashboard-db and dashboard-s3 in both prod and staging spaces

GSA / data.gov