[ci.jenkins.io– Startup and build failures with `java.lang.NoClassDefFoundError: org/jenkinsci/plugins/gitclient/RepositoryCallback`error in the `datadog` plugin

[dduportal]

Service(s)

ci.jenkins.io

Summary

The 11 April 2024 at 13h20 UTC, after a set of plugin upgrades (with controller restart) on ci.jenkins.io (see the list below), the controller failed to restart twice with an error stack described below.

It started properly the 3rd time but each tentative of a build fails with the same error stack (below) in the controller logs:

Apr 11 13:01:57 controller docker-jenkins[2899]: java.lang.ClassNotFoundException: org.jenkinsci.plugins.gitclient.RepositoryCallback
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.net.URLClassLoader.findClass(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at jenkins.util.URLClassLoader2.findClass(URLClassLoader2.java:35)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]: Caused: java.lang.NoClassDefFoundError: org/jenkinsci/plugins/gitclient/RepositoryCallback
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.datadog.jenkins.plugins.datadog.model.BuildData.isGit(BuildData.java:512)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.datadog.jenkins.plugins.datadog.model.BuildData.populateEnvVariables(BuildData.java:370)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.datadog.jenkins.plugins.datadog.model.BuildData.<init>(BuildData.java:160)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.datadog.jenkins.plugins.datadog.logs.DatadogTaskListenerDecorator.<init>(DatadogTaskListenerDecorator.java:49)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.datadog.jenkins.plugins.datadog.logs.DatadogTaskListenerDecorator$Factory.of(DatadogTaskListenerDecorator.java:82)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator.lambda$apply$3(TaskListenerDecorator.java:164)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator.apply(TaskListenerDecorator.java:166)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:236)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:629)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:362)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at hudson.model.ResourceController.execute(ResourceController.java:101)
Apr 11 13:01:57 controller docker-jenkins[2899]:         at hudson.model.Executor.run(Executor.java:442)

This error is thrown by the datadog plugin, installed in the version 6.0.2 in this controller. Note that the plugin was NOT updated that day. The updated plugins were:

aws-java-sdk-ec2
aws-java-sdk-minimal
blueocean-bitbucket-pipeline
blueocean-commons
blueocean-config
blueocean-core-js
blueocean-dashboard
blueocean-events
blueocean-git-pipeline
blueocean-github-pipeline
blueocean-i18n
blueocean-jwt
blueocean-personalization
blueocean-pipeline-api-impl
blueocean-pipeline-editor
blueocean-pipeline-scm-api
blueocean-rest-impl
blueocean-rest
blueocean-web
blueocean
github-branch-source
jenkins-design-language
pipeline-graph-view
pipeline-model-api
pipeline-model-definition
pipeline-model-extensions
pipeline-stage-tags-metadata
script-security
ssh-agent
support-core

Reproduction steps

• Jenkins Core 2.440.2
• The following plugin set:

analysis-model-api: 12.1.0
ansicolor: 1.0.4
antisamy-markup-formatter: 162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api: 4.5.14-208.v438351942757
artifact-manager-s3: 845.848.ve968b_8576c1c
asm-api: 9.7-33.v4d23ef79fcc8
authentication-tokens: 1.53.v1c90fd9191a_b_
aws-credentials: 231.v08a_59f17d742
aws-global-configuration: 130.v35b_7b_96f53c3
aws-java-sdk-ec2: 1.12.696-451.v0651a_da_9ca_ec
aws-java-sdk-minimal: 1.12.696-451.v0651a_da_9ca_ec
azure-container-agents: 253.vd2f5cd5c5040
azure-credentials: 312.v0f3973cd1e59
azure-sdk: 157.v855da_0b_eb_dc2
azure-vm-agents: 921.v76424a_e24594
blueocean-bitbucket-pipeline: 1.27.12
blueocean-commons: 1.27.12
blueocean-config: 1.27.12
blueocean-core-js: 1.27.12
blueocean-dashboard: 1.27.12
blueocean-display-url: 2.4.2
blueocean-events: 1.27.12
blueocean-git-pipeline: 1.27.12
blueocean-github-pipeline: 1.27.12
blueocean-i18n: 1.27.12
blueocean-jwt: 1.27.12
blueocean-personalization: 1.27.12
blueocean-pipeline-api-impl: 1.27.12
blueocean-pipeline-editor: 1.27.12
blueocean-pipeline-scm-api: 1.27.12
blueocean-rest-impl: 1.27.12
blueocean-rest: 1.27.12
blueocean-web: 1.27.12
blueocean: 1.27.12
bootstrap5-api: 5.3.3-1
bouncycastle-api: 2.30.1.77-225.v26ea_c9455fd9
branch-api: 2.1152.v6f101e97dd77
build-discarder: 139.v05696a_7fe240
buildtriggerbadge: 251.vdf6ef853f3f5
caffeine-api: 3.1.8-133.v17b_1ff2e0599
checks-api: 2.2.0
cloud-stats: 336.v788e4055508b_
cloudbees-bitbucket-branch-source: 883.v041fa_695e9c2
cloudbees-folder: 6.901.vb_4c7a_da_75da_3
commons-lang3-api: 3.13.0-62.v7d18e55f51e2
commons-text-api: 1.11.0-95.v22a_d30ee5d36
config-file-provider: 968.ve1ca_eb_913f8c
configuration-as-code: 1775.v810dc950b_514
copyartifact: 722.v0662a_9b_e22a_c
coverage-badges-extension: 47.v41e62ecf0928
coverage: 1.13.0
credentials-binding: 657.v2b_19db_7d6e6d
credentials: 1337.v60b_d7b_c7b_c9f
dark-theme: 439.vdef09f81f85e
data-tables-api: 2.0.3-1
display-url-api: 2.200.vb_9327d658781
docker-commons: 439.va_3cb_0a_6a_fb_29
docker-workflow: 572.v950f58993843
durable-task: 550.v0930093c4b_a_6
echarts-api: 5.5.0-1
embeddable-build-status: 467.v4a_954796e45d
favorite: 2.208.v91d65b_7792a_c
font-awesome-api: 6.5.1-3
forensics-api: 2.4.0
generic-tool: 1.1
git-client: 4.7.0
git-forensics: 2.1.0
git: 5.2.1
github-api: 1.318-461.v7a_c09c9fa_d63
github-branch-source: 1787.v8b_8cd49a_f8f1
github-checks: 554.vb_ee03a_000f65
github-scm-trait-notification-context: 40.vfa_7f31a_b_d7f8
github: 1.38.0
groovy: 457.v99900cb_85593
gson-api: 2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api: 2.1.8-30.v7e777411b_148
htmlpublisher: 1.33
http_request: 1.18
instance-identity: 185.v303dc7c645f9
ionicons-api: 70.v2959a_b_74e3cf
jackson2-api: 2.17.0-379.v02de8ec9f64c
jakarta-activation-api: 2.1.3-1
jakarta-mail-api: 2.1.3-1
javax-activation-api: 1.2.0-6
javax-mail-api: 1.6.2-9
jaxb: 2.3.9-1
jenkins-design-language: 1.27.12
jjwt-api: 0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api: 3.1.19-2
joda-time-api: 2.12.7-29.v5a_b_e3a_82269a_
jquery3-api: 3.7.1-2
jsch: 0.2.16-86.v42e010d9484b_
json-api: 20240303-41.v94e11e6de726
json-path-api: 2.9.0-58.v62e3e85b_a_655
junit-attachments: 205.vc0677977deb_0
junit-realtime-test-reporter: 149.v05a_d403e2f48
junit: 1265.v65b_14fa_f12f0
kubernetes-client-api: 6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials: 0.11
kubernetes: 4203.v1dd44f5b_1cf9
ldap: 719.vcb_d039b_77d0d
lockable-resources: 1246.v28b_e4cc6fa_16
mailer: 472.vf7c289a_4b_420
matrix-auth: 3.2.2
matrix-project: 822.824.v14451b_c0fd42
metrics: 4.2.21-449.v6960d7c54c69
mina-sshd-api-common: 2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core: 2.12.1-101.v85b_e08b_780dd
okhttp-api: 4.11.0-172.vda_da_1feeb_c6e
parallel-test-executor: 473.v81c8f8d1675f
parameterized-trigger: 787.v665fcf2a_830b_
pipeline-build-step: 540.vb_e8849e1a_b_d8
pipeline-github: 2.8-159.09e4403bc62f
pipeline-graph-analysis: 216.vfd8b_ece330ca_
pipeline-graph-view: 237.v8959b_75f3c47
pipeline-groovy-lib: 704.vc58b_8890a_384
pipeline-input-step: 491.vb_07d21da_1a_fb_
pipeline-milestone-step: 119.vdfdc43fc3b_9a_
pipeline-model-api: 2.2198.v41dd8ef6dd56
pipeline-model-definition: 2.2198.v41dd8ef6dd56
pipeline-model-extensions: 2.2198.v41dd8ef6dd56
pipeline-rest-api: 2.34
pipeline-stage-step: 312.v8cd10304c27a_
pipeline-stage-tags-metadata: 2.2198.v41dd8ef6dd56
pipeline-stage-view: 2.34
plain-credentials: 179.vc5cb_98f6db_38
plugin-util-api: 4.1.0
prism-api: 1.29.0-13
pubsub-light: 1.18
scm-api: 689.v237b_6d3a_ef7f
script-security: 1335.vf07d9ce377a_e
snakeyaml-api: 2.2-111.vc6598e30cc65
sse-gateway: 1.26
ssh-agent: 367.vf9076cd4ee21
ssh-credentials: 337.v395d2403ccd4
ssh-slaves: 2.948.vb_8050d697fec
sshd: 3.322.v159e91f6a_550
structs: 337.v1b_04ea_4df7c8
support-core: 1427.v083f1d9372a_f
theme-manager: 215.vc1ff18d67920
timestamper: 1.26
token-macro: 400.v35420b_922dcb_
trilead-api: 2.142.v748523a_76693
variant: 60.v7290fc0eb_b_cd
warnings-ng: 11.2.2
windows-azure-storage: 419.v4046cd70d2e3
workflow-aggregator: 596.v8c21c963d92d
workflow-api: 1291.v51fd2a_625da_7
workflow-basic-steps: 1049.v257a_e6b_30fb_d
workflow-cps: 3894.vd0f0248b_a_fc4
workflow-durable-task-step: 1336.v768003e07199
workflow-job: 1400.v7fd111b_ec82f
workflow-multibranch: 773.vc4fe1378f1d5
workflow-scm-step: 427.v4ca_6512e7df1
workflow-step-api: 657.v03b_e8115821b_
workflow-support: 896.v175a_a_9c5b_78f

• Datadog configured with metrics, logs and trace collection

[dduportal]

• This issue is a "post-incident" to track the resolution and cleanup but also for further audits. The whole discussion/analysis took place the elements chat room: https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$eViX1W4HTUBrR4CisFgZHJkVuAOjSBicxFZmRcsFbBw?via=g4v.dev&via=gitter.im&via=matrix.org
• The problem was reported 3 hours before the incident in the datadog-plugin repository by someone else: https://github.com/jenkinsci/datadog-plugin/issues/411

=> The short-term corrective action has been to remove the datadog plugin and its configuration (see https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$FCc3cXmUQSzujZrovaZbvRsVSKq1FEyk-BsXgs4I4pg?via=g4v.dev&via=gitter.im&via=matrix.org):

plugin removed and un-configured (puppet agent stopped and disabled on the VM, the file $JENKINS_HOME/casc.d/datadog.yaml has been renamed to datadog.yaml.disabled)

• Service is back at 15h25 UTC (no more error stacks) and builds are flowing

[dduportal]

Subsequent problem reported by @MarkEWaite:

Pipeline stage view may have been broken by a recent plugin update ... Pipeline graph view seems to be affected as well

After further analysis, it appears that in fact none of the plugin are broken but the build.xml of these builds do not have the proper content. Administrators of ci.jenkins.io can also see the warning about "unreadable data" in the build data:

=> This is expected as the (temporary) removal of the datadog plugin also removes the impacted classes and features.

[dduportal]

After some tests (and an accidental re-install of datadog 6.0.3):

• Datadog 6.0.3 does fix the error as expected
• Alas the initial error did corrupt the build.xml files from previous build and the data does not seem recoverable:
  • New builds are working great
  • Older builds still have their status, build logs and date/time but the details of their pipeline execution (flow nodes, stages, etc.) seems broken by datadog.

VM is now back to business, let's roll