7hunderbird
commented
1 week ago Took the alerts from Monday and Tuesday
To understand what's been happening created some statistic information as well.
These were the jobs that were failing:
- cms/cms-db-backup-prod (30)
- cms/cms-full-pipeline (30)
- cms/cms-efs-backup-prod (3)
- cms/cms-db-sanitize (1)
Key Questions
- How many of each?
- See number in list above
- What errors came up in each job?
- Did all of them run on the same or different Jenkins Agent?
- What were the time ranges?
- Monday from 9:46AM ET to 11:15AM ET
- Tuesday from 6:00AM ET to 12:15PM ET
cms/cms-db-backup-prod
cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109621/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109622/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109623/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109624/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109625/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109626/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109627/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109703/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109704/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109705/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109706/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109707/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109708/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109709/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109710/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109711/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109712/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109713/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109714/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109715/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109716/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109717/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109718/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109719/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109720/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109721/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109722/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109723/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109726/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109728/
cms/cms-full-pipeline
cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109442/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109443/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109444/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109445/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109446/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109447/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109448/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109524/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109525/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109526/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109527/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109528/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109529/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109530/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109531/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109532/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109533/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109534/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109535/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109536/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109537/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109538/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109539/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109540/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109541/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109542/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109543/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109544/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109547/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109549/
cms/cms-efs-backup-prod
cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42294/ cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42306/ cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42307/
cms/cms-db-sanitize
cms/cms-db-sanitize failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-sanitize/105053/
User Story or Problem Statement
The first couple of days this week (the week starting July 1st 2024) we've been receiving errors on jobs related to sanitizing and backing up the CMS database.
Let's identify the issue causing the failures and then determine the fix we can do to ensure the jobs run without failure.
Incident Examination
Monday 7/1
I came into work to see errors in the
#cms-notificationsroom and began to investigate. I knew that I had been working on changes to theefs-backupjob, so I was surprised to see errors from many other jobs likedb-backuporfull-pipeline.I looked at the most recent and saw that it had failed because of a lack of disk space. This was the cause of failures for the
efs-backupjob as well. But I didn't want to assume they were related and I needed to determine the cause.I didn't make further progress on Monday because of other tasks that came up.
Tuesday 7/2
When I saw that there were errors again I knew I needed to dig deeper
Wednesday 7/3
There were no errors on Wednesday.
Working Theories
Shared File System
efs-backupjob does delete it's temporary fileefs-backupjob initially fails, it fails in the middle of creating the large backupefs-backupjob is happening DURING execution not at the BEGINNING of executionTODO: Need to establish that there aren't external events happening to Jenkins agents
Job Dependencies
Acceptance Criteria