Jenkins - Scheduled deploys of master and worker AMI

jbritt1 commented 4 years ago

Issue Description

Jenkins job fails a check on a daily basis that results in quite a bit of additional noise.
Also, Jenkins is not getting deployed on a regular basis, results in patches / apps / etc getting out of date.

Work was done in this pull request to make the OS Check jobs not rely so heavily on naming as it was missing instances that did not include the "-asg" portion of a Name tag. With that merged however, the OS Check for the utility environment fails daily due to how long it has been since Jenkins master has been deployed. This will continue to be a daily failure for that particular job until an automated method or even scheduled manual method is implemented to ensure the deployed master does not reach an age of >14 days.

After some discussions, the Operations team decided that it is worth the effort and time to ensure that Jenkins is reliable and secure until such time as it is no longer needed.

As of right now, the Worker AMI gets deleted every 30 days by the AMI cleanup script which causes a Jenkins outage like clockwork. This also needs to be resolved.

Tasks

[X] Team should decide on whether automated method is worth the time and effort in light of eventually sunsetting Jenkins or if we need manual intervention to hold us over a bit longer (see above)
[x] Move back to Jenkins LTS (#8321)
[ ] Determine if all plugin upgrades are safe (vagrant deploys of Jenkins to see if plugins explode)
[ ] Redeploy Jenkins

Acceptance Criteria

[ ] A process / solution in place to prevent Jenkins from being the cause of OS Check failures in the utility environment
[ ] Regular automatic OR Manual deploys of Jenkins on old infra are scheduled.
[ ] Solid documentation for deploying Jenkins problem-free?

Relation

8321 (consider doing this work simultaneously. needs more team discussion / buy-in)

jbritt1 commented 3 years ago

https://dsva.slack.com/archives/CJYRZK2HH/p1631279782326600

jhouse-solvd commented 3 years ago

Let's leave this in the icebox and either schedule some new monthly deploys, or decide to put in the work to do this.

jbritt1 commented 2 years ago

Should we close this? I think our intention is to continue to move towards sunsetting Jenkins vs spending time on more automation around it.

department-of-veterans-affairs / va.gov-team