Identify and Repair Recent Jenkins Failed Jobs

7hunderbird commented 4 months ago

User Story or Problem Statement

The first couple of days this week (the week starting July 1st 2024) we've been receiving errors on jobs related to sanitizing and backing up the CMS database.

Let's identify the issue causing the failures and then determine the fix we can do to ensure the jobs run without failure.

Incident Examination

Monday 7/1

I came into work to see errors in the #cms-notifications room and began to investigate. I knew that I had been working on changes to the efs-backup job, so I was surprised to see errors from many other jobs like db-backup or full-pipeline.

I looked at the most recent and saw that it had failed because of a lack of disk space. This was the cause of failures for the efs-backup job as well. But I didn't want to assume they were related and I needed to determine the cause.

I didn't make further progress on Monday because of other tasks that came up.

Tuesday 7/2

When I saw that there were errors again I knew I needed to dig deeper

Wednesday 7/3

There were no errors on Wednesday.

Working Theories

Shared File System

There is a Shared file system attached to each Jenkins Agent
When the efs-backup job does delete it's temporary file
When the efs-backup job initially fails, it fails in the middle of creating the large backup
The initial disk space error in the efs-backup job is happening DURING execution not at the BEGINNING of execution

TODO: Need to establish that there aren't external events happening to Jenkins agents

Job Dependencies

full-pipeline calls
- efs-backup
- db-backup

Acceptance Criteria

[ ] Determine the cause of the failures

7hunderbird commented 4 months ago

Took the alerts from Monday and Tuesday

To understand what's been happening created some statistic information as well.

These were the jobs that were failing:

cms/cms-db-backup-prod (30)
cms/cms-full-pipeline (30)
cms/cms-efs-backup-prod (3)
cms/cms-db-sanitize (1)

Key Questions

How many of each?
- See number in list above
What errors came up in each job?
Did all of them run on the same or different Jenkins Agent?
What were the time ranges?
- Monday from 9:46AM ET to 11:15AM ET
- Tuesday from 6:00AM ET to 12:15PM ET

cms/cms-db-backup-prod

cms/cms-full-pipeline

cms/cms-efs-backup-prod

cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42294/ cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42306/ cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42307/

cms/cms-db-sanitize

cms/cms-db-sanitize failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-sanitize/105053/

7hunderbird commented 4 months ago

Broken up by day

Monday

cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109621/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109622/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109623/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109624/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109625/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109626/ cms/cms-db-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-backup-prod/109627/

cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109442/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109443/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109444/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109445/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109446/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109447/ cms/cms-full-pipeline failed http://jenkins.vfs.va.gov/job/cms/job/cms-full-pipeline/109448/

cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42294/

Tuesday

cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42306/ cms/cms-efs-backup-prod failed http://jenkins.vfs.va.gov/job/cms/job/cms-efs-backup-prod/42307/

cms/cms-db-sanitize failed http://jenkins.vfs.va.gov/job/cms/job/cms-db-sanitize/105053/

7hunderbird commented 4 months ago

The error 15:49:37 ("No usable temporary directory found in %s" % dirlist)) came up today for the efs-backup job. But I can't connect to the bad Jenkins agent to determine why it's not got a working temp directory.

When I try to connect this happens:

aws ssm start-session --target i-0989977e88bdc7422

Starting session with SessionId: Tyler.Bird-j54ytehnhh45jvithzgcai7xtu

SessionId: Tyler.Bird-j54ytehnhh45jvithzgcai7xtu : Plugin with name Standard_Stream not found. Step name: Standard_Stream

7hunderbird commented 2 months ago

Today a jenkins agent ran out of disk space. https://dsva.slack.com/archives/CJT90C0UT/p1724092308179419

7hunderbird commented 2 months ago

This connection to git has happened recently a few times. It could be an API throttling issue.

[2024-09-13T07:00:00.517Z] Started by timer
[2024-09-13T07:00:00.517Z] Checking out git https://github.com/department-of-veterans-affairs/devops.git into /var/jenkins_home/workspace/cms/cms-full-pipeline@script to read ansible/Jenkinsfiles/cms/cms-backup-pipeline.groovy
[2024-09-13T07:00:00.522Z] The recommended git tool is: git
[2024-09-13T07:00:00.522Z] using credential va-bot
[2024-09-13T07:00:00.575Z]  > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/cms/cms-full-pipeline@script/.git # timeout=10
[2024-09-13T07:00:00.582Z] Fetching changes from the remote Git repository
[2024-09-13T07:00:00.582Z]  > git config remote.origin.url https://github.com/department-of-veterans-affairs/devops.git # timeout=10
[2024-09-13T07:00:00.604Z] Fetching upstream changes from https://github.com/department-of-veterans-affairs/devops.git
[2024-09-13T07:00:00.604Z]  > git --version # timeout=10
[2024-09-13T07:00:00.608Z]  > git --version # 'git version 2.20.1'
[2024-09-13T07:00:00.608Z] using GIT_ASKPASS to set credentials Credentials for va-bot on GitHub
[2024-09-13T07:00:00.636Z]  > git fetch --tags --force --progress -- https://github.com/department-of-veterans-affairs/devops.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2024-09-13T07:00:06.221Z]  > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
[2024-09-13T07:00:06.279Z] Checking out Revision ba0e51e6665e2095dc05664ee0ed42ecaf23ab74 (refs/remotes/origin/master)
[2024-09-13T07:00:06.279Z]  > git config core.sparsecheckout # timeout=10
[2024-09-13T07:00:06.288Z]  > git checkout -f ba0e51e6665e2095dc05664ee0ed42ecaf23ab74 # timeout=10
[2024-09-13T07:00:06.984Z] Commit message: "92474 & 91677 (#14693)"
[2024-09-13T07:00:06.986Z]  > git rev-list --no-walk ba0e51e6665e2095dc05664ee0ed42ecaf23ab74 # timeout=10
[2024-09-13T07:00:07.090Z] Running in Durability level: MAX_SURVIVABILITY
[2024-09-13T07:00:07.107Z] Loading library va.gov-devops-jenkins-lib@master
[2024-09-13T07:00:07.108Z] Attempting to resolve master from remote references...
[2024-09-13T07:00:07.108Z]  > git --version # timeout=10
[2024-09-13T07:00:07.115Z]  > git --version # 'git version 2.20.1'
[2024-09-13T07:00:07.115Z] using GIT_ASKPASS to set credentials Credentials for va-vfs-bot on GitHub
[2024-09-13T07:00:07.115Z]  > git ls-remote -h -- https://github.com/department-of-veterans-affairs/va.gov-devops-jenkins-lib.git # timeout=10
[2024-09-13T07:00:07.450Z] Found match: refs/heads/master revision b50da3eb1f99b65623130e24bd28f24c3e2b120d
[2024-09-13T07:00:07.456Z] The recommended git tool is: NONE
[2024-09-13T07:00:07.456Z] using credential va-vfs-bot
[2024-09-13T07:00:07.488Z]  > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/cms/cms-full-pipeline@libs/va.gov-devops-jenkins-lib/.git # timeout=10
[2024-09-13T07:00:07.495Z] Fetching changes from the remote Git repository
[2024-09-13T07:00:07.495Z]  > git config remote.origin.url https://github.com/department-of-veterans-affairs/va.gov-devops-jenkins-lib.git # timeout=10
[2024-09-13T07:00:07.515Z] Fetching without tags
[2024-09-13T07:00:07.515Z] Fetching upstream changes from https://github.com/department-of-veterans-affairs/va.gov-devops-jenkins-lib.git
[2024-09-13T07:00:07.515Z]  > git --version # timeout=10
[2024-09-13T07:00:07.519Z]  > git --version # 'git version 2.20.1'
[2024-09-13T07:00:07.519Z] using GIT_ASKPASS to set credentials Credentials for va-vfs-bot on GitHub
[2024-09-13T07:00:07.549Z]  > git fetch --no-tags --force --progress -- https://github.com/department-of-veterans-affairs/va.gov-devops-jenkins-lib.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2024-09-13T07:00:07.980Z] Checking out Revision b50da3eb1f99b65623130e24bd28f24c3e2b120d (master)
[2024-09-13T07:00:07.980Z]  > git config core.sparsecheckout # timeout=10
[2024-09-13T07:00:07.991Z]  > git checkout -f b50da3eb1f99b65623130e24bd28f24c3e2b120d # timeout=10
[2024-09-13T07:00:08.041Z] Commit message: "Make all github api requests authenticated (#20)"
[2024-09-13T07:00:08.042Z]  > git rev-list --no-walk b50da3eb1f99b65623130e24bd28f24c3e2b120d # timeout=10
[2024-09-13T07:00:08.371Z] [Pipeline] Start of Pipeline
[2024-09-13T07:00:08.575Z] [Pipeline] node
[2024-09-13T07:00:08.608Z] Running on EC2 (jenkins-cloud) - vetsgov-general-purpose-1b (sir-67ies6bm) in /home/jenkins/workspace/cms/cms-full-pipeline
[2024-09-13T07:00:08.648Z] [Pipeline] {
[2024-09-13T07:00:08.718Z] [Pipeline] stage
[2024-09-13T07:00:08.731Z] [Pipeline] { (Declarative: Checkout SCM)
[2024-09-13T07:00:08.808Z] [Pipeline] checkout
[2024-09-13T07:00:08.828Z] The recommended git tool is: git
[2024-09-13T07:00:08.837Z] using credential va-bot
[2024-09-13T07:00:08.875Z] Cloning the remote Git repository
[2024-09-13T07:00:09.061Z] ERROR: Error cloning remote repo 'origin'
[2024-09-13T07:00:09.061Z] hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://github.com/department-of-veterans-affairs/devops.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
[2024-09-13T07:00:09.061Z] stdout: 
[2024-09-13T07:00:09.061Z] stderr: ssh_exchange_identification: Connection closed by remote host
[2024-09-13T07:00:09.061Z] fatal: Could not read from remote repository.
[2024-09-13T07:00:09.061Z] 
[2024-09-13T07:00:09.061Z] Please make sure you have the correct access rights
[2024-09-13T07:00:09.061Z] and the repository exists.
[2024-09-13T07:00:09.061Z] 
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2639)
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2079)
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:601)
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:830)
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
[2024-09-13T07:00:09.061Z]  at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
[2024-09-13T07:00:09.061Z]  at hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2024-09-13T07:00:09.061Z]  at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2024-09-13T07:00:09.061Z]  at hudson.remoting.Request$2.run(Request.java:376)
[2024-09-13T07:00:09.061Z]  at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
[2024-09-13T07:00:09.061Z]  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2024-09-13T07:00:09.061Z]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2024-09-13T07:00:09.061Z]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2024-09-13T07:00:09.061Z]  at java.lang.Thread.run(Thread.java:750)
[2024-09-13T07:00:09.061Z]  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-cloud) - vetsgov-general-purpose-1b (sir-67ies6bm)
[2024-09-13T07:00:09.061Z]      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
[2024-09-13T07:00:09.061Z]      at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2024-09-13T07:00:09.061Z]      at hudson.remoting.Channel.call(Channel.java:1001)
[2024-09-13T07:00:09.061Z]      at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
[2024-09-13T07:00:09.061Z]      at jdk.internal.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
[2024-09-13T07:00:09.061Z]      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-09-13T07:00:09.061Z]      at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2024-09-13T07:00:09.061Z]      at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
[2024-09-13T07:00:09.061Z]      at com.sun.proxy.$Proxy96.execute(Unknown Source)
[2024-09-13T07:00:09.061Z]      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1223)
[2024-09-13T07:00:09.061Z]      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1301)
[2024-09-13T07:00:09.061Z]      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
[2024-09-13T07:00:09.061Z]      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
[2024-09-13T07:00:09.061Z]      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
[2024-09-13T07:00:09.061Z]      at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2024-09-13T07:00:09.061Z]      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[2024-09-13T07:00:09.061Z]      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2024-09-13T07:00:09.061Z]      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2024-09-13T07:00:09.061Z]      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2024-09-13T07:00:09.061Z]      at java.base/java.lang.Thread.run(Thread.java:829)
[2024-09-13T07:00:09.092Z] [Pipeline] }
[2024-09-13T07:00:09.182Z] [Pipeline] // stage
[2024-09-13T07:00:09.227Z] [Pipeline] }
[2024-09-13T07:00:09.305Z] [Pipeline] // node
[2024-09-13T07:00:09.379Z] [Pipeline] stage
[2024-09-13T07:00:09.393Z] [Pipeline] { (Declarative: Post Actions)
[2024-09-13T07:00:09.476Z] [Pipeline] cleanWs
[2024-09-13T07:00:09.539Z] Error when executing always post condition:
[2024-09-13T07:00:09.540Z] org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
[2024-09-13T07:00:09.540Z] Perhaps you forgot to surround the code with a step that provides this, such as: node
[2024-09-13T07:00:09.540Z]  at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:88)
[2024-09-13T07:00:09.540Z]  at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:69)
[2024-09-13T07:00:09.540Z]  at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2024-09-13T07:00:09.540Z]  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[2024-09-13T07:00:09.540Z]  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2024-09-13T07:00:09.540Z]  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2024-09-13T07:00:09.540Z]  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2024-09-13T07:00:09.540Z]  at java.base/java.lang.Thread.run(Thread.java:829)
[2024-09-13T07:00:09.540Z] 
[2024-09-13T07:00:09.592Z] [Pipeline] slackSend
[2024-09-13T07:00:09.609Z] Slack Send Pipeline step running, values are - baseUrl: <empty>, teamDomain: dsva, channel: #cms-notifications, color: danger, botUser: false, tokenCredentialId: slack-token, notifyCommitters: false, iconEmoji: <empty>, username: <empty>, timestamp: <empty>
[2024-09-13T07:00:08.876Z] Cloning repository https://github.com/department-of-veterans-affairs/devops.git
[2024-09-13T07:00:08.876Z]  > git init /home/jenkins/workspace/cms/cms-full-pipeline # timeout=10
[2024-09-13T07:00:08.879Z] Fetching upstream changes from https://github.com/department-of-veterans-affairs/devops.git
[2024-09-13T07:00:08.879Z]  > git --version # timeout=10
[2024-09-13T07:00:08.880Z]  > git --version # 'git version 2.38.4'
[2024-09-13T07:00:08.880Z] using GIT_ASKPASS to set credentials Credentials for va-bot on GitHub
[2024-09-13T07:00:08.881Z]  > git fetch --tags --force --progress -- https://github.com/department-of-veterans-affairs/devops.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2024-09-13T07:00:09.967Z] [Pipeline] }
[2024-09-13T07:00:10.051Z] [Pipeline] // stage
[2024-09-13T07:00:10.112Z] [Pipeline] End of Pipeline
[2024-09-13T07:00:10.200Z] ERROR: Error cloning remote repo 'origin'
[2024-09-13T07:00:10.200Z] Finished: FAILURE

7hunderbird commented 2 months ago

CleanShot 2024-09-13 at 10 06 47

In fact all these were errors for connecting to the repo.

returned status code 128:
[2024-09-13T07:00:09.061Z] stdout: 
[2024-09-13T07:00:09.061Z] stderr: ssh_exchange_identification: Connection closed by remote host
[2024-09-13T07:00:09.061Z] fatal: Could not read from remote repository.

department-of-veterans-affairs / va.gov-cms