Decrease the time it takes to deploy to staging

edmund-dunn commented 4 months ago

User Story or Problem Statement

Staging deployments for BRD is currently taking two hours and causing problems with the deployment process (e.g. shield is persistently staging up). The delays has made staging unusable for developers.

Description or Additional Context

See: http://jenkins.vfs.va.gov/job/deploys/job/cms-vagov-staging/3576/consoleFull

11:13:55 Reset file permissions ----------------------------------------------- 1912.59s (31.8m) 11:13:55 sites/default/files set owner to cms & group to apache --------------- 1452.13s (24.2m) 11:13:55 Extract asset files to a temp dir ------------------------------------ 1420.01s (23.7m) 11:13:55 Parallel Rsync files to EFS ------------------------------------------ 1085.20s (18.0m) 11:13:55 Download Asset files from S3 ------------------------------------------ 281.64s (4.7m)


The current Lifecycle limit is maxed out at 7200 seconds (2 hours) so the job fails and restarts because the ec2 instance is abandoned. 

Historically these deployments take a long time because user files are copied down every time it deploys.

We are resetting file permissions twice which may be redundant on staging.

11:13:55  Reset file permissions ----------------------------------------------- 1912.59s
11:13:55  sites/default/files set owner to cms & group to apache --------------- 1452.13s

We need to test if we need to set the owner and group, which is taking 24 min, or if we can do away with that, just on staging.

Steps for Implementation

Review the existing permissions script to determine if there's opportunities to be more efficient with the script in regards to setting permissions for files.
Review the existing process to copy files into staging to determine if there's opportunities to be more efficient with the process.
Regroup in a pre-refinement to determine next steps.

Acceptance Criteria

[x] Test opportunities to be more efficient with setting permissions on cms-test.
[x] Determine if being more efficient with setting permissions reduces the build time for staging.
[x] Create a plan of action of implement changes to reduce build time in Staging.
[ ] Implement the plan of action.

### Tasks
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18887
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18888
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19048
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19050
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19049
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19051

7hunderbird commented 3 months ago

This is the job in Jenkins with build times. We can use this to help us define some metrics for improvement.

http://jenkins.vfs.va.gov/blue/organizations/jenkins/deploys%2Fcms-vagov-staging/activity

gracekretschmer-metrostar commented 3 months ago

To do: add Tyler's comment on this ticket.

7hunderbird commented 3 months ago

This issue is blocked until the following actions are resolved:

7hunderbird commented 2 months ago

We are going to break the release and testing into two phases.

Phase 1 - `cms-test`

Phase 2 - `cms`

Add the file sync job for cms
remove the file sync from the cms staging deploy
communicate out that we've moved the file sync out to a nightly job and ask that people report any thing out of the ordinary

7hunderbird commented 2 months ago

Good progress is being made on the "add file sync job" for cms-test. I've got the functionality to add a job to Jenkins working. Now I just need to develop the Ansible tasks that will do the file sync work.

https://github.com/department-of-veterans-affairs/va.gov-cms/issues/19048

7hunderbird commented 2 months ago

Collapsing all sub-tickets back to this one.

Closing these tickets:

Going with this PR to review for this current ticket to "Decrease the time it takes to deploy to staging".

https://github.com/department-of-veterans-affairs/devops/pull/14664

Once we're ready to merge that PR the work will be done.

department-of-veterans-affairs / va.gov-cms