search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
100 stars 53 forks source link

OTel logs may cause `NullPointerException` / `NoClassDefFoundError` at startup #622

Closed cyrille-leclerc closed 1 year ago

cyrille-leclerc commented 1 year ago

Jenkins and plugins versions report

jenkins 2.387.1 opentelemetry.jpi 2.12.0

What Operating System are you using (both controller, and any agents involved in the problem)?

-

Reproduction steps

start jenkins

Expected Results

clean startup

Actual Results

java.lang.NoClassDefFoundError: Could not initialize class io.opentelemetry.context.LazyStorage
    at io.opentelemetry.context.ContextStorage.get(ContextStorage.java:72)
    at io.opentelemetry.context.Context.current(Context.java:91)
    at io.jenkins.plugins.opentelemetry.init.OtelJulHandler.publish(OtelJulHandler.java:96)
[2023-04-03 18:44:29][INFO  ][jenkins.InitReactorRunner$1 onAttained] System config adapted
[2023-04-03 18:44:34][INFO  ][io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider initializeOtlp] OpenTelemetry SDK initialized: SDK [config: otel.traces.exporter=otlp, otel.metrics.exporter=otlp, otel.exporter.otlp.endpoint=http://the-service-hostname:4317/, resource: service.name=jenkins, service.namespace=jenkins, service.version=2.387.1]
[2023-04-03 18:44:34][WARNING][jenkins.model.Jenkins$5 runTask] JenkinsOpenTelemetryPluginConfiguration.initializeOpenTelemetry failed perhaps due to plugin dependency issues
java.lang.Error: java.lang.reflect.InvocationTargetException
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:115)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1164)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
    ... 9 more
Caused by: java.lang.ExceptionInInitializerError
    at io.opentelemetry.context.ContextStorage.get(ContextStorage.java:72)
    at io.opentelemetry.context.Context.current(Context.java:91)
    at io.jenkins.plugins.opentelemetry.init.OtelJulHandler.publish(OtelJulHandler.java:96)
    at java.logging/java.util.logging.Logger.log(Logger.java:979)
    at org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
    at org.slf4j.jul.JDK14LoggerAdapter.handleNormalizedLoggingCall(JDK14LoggerAdapter.java:140)
    at org.slf4j.helpers.AbstractLogger.handle_1ArgsCall(AbstractLogger.java:386)
    at org.slf4j.helpers.AbstractLogger.debug(AbstractLogger.java:149)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadAsResource(WebAppClassLoader.java:570)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:502)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at jenkins.model.TransientActionFactory.factoriesFor(TransientActionFactory.java:135)
    at hudson.model.Actionable.getAction(Actionable.java:331)
    at io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener.afterSdkInitialized(MonitoringComputerListener.java:49)
    at io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider.lambda$initialize$4(OpenTelemetrySdkProvider.java:136)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:502)
    at io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider.initialize(OpenTelemetrySdkProvider.java:135)
    at io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration.initializeOpenTelemetry(JenkinsOpenTelemetryPluginConfiguration.java:222)
    ... 14 more
Caused by: java.lang.NullPointerException
    at io.opentelemetry.context.Context.current(Context.java:91)
    at io.jenkins.plugins.opentelemetry.init.OtelJulHandler.publish(OtelJulHandler.java:96)
    at java.logging/java.util.logging.Logger.log(Logger.java:979)
    at org.slf4j.jul.JDK14LoggerAdapter.innerNormalizedLoggingCallHandler(JDK14LoggerAdapter.java:156)
    at org.slf4j.jul.JDK14LoggerAdapter.handleNormalizedLoggingCall(JDK14LoggerAdapter.java:140)
    at org.slf4j.helpers.AbstractLogger.handle2ArgsCall(AbstractLogger.java:393)
    at org.slf4j.helpers.AbstractLogger.debug(AbstractLogger.java:155)
    at org.eclipse.jetty.webapp.WebAppClassLoader.getResources(WebAppClassLoader.java:368)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1196)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
    at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
    at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
    at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
    at io.opentelemetry.context.LazyStorage.createStorage(LazyStorage.java:107)
    at io.opentelemetry.context.LazyStorage.<clinit>(LazyStorage.java:80)
    ... 39 more

Anything else?

No response

acompton-slack commented 1 year ago

Here is a more complete list of the stack traces I receive. jenkins_stacktrace.log

acompton-slack commented 1 year ago

Environment report:

Jenkins: 2.387.1
OS: Linux - 5.19.0-1022-aws
Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
adoptopenjdk:1.5
analysis-model-api:11.1.0
ansicolor:1.0.2
ant:481.v7b_09e538fcca
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
artifact-manager-s3:754.vb_5d21c758eb_3
audit-trail:333.vb_e1b_b_0f1238c
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.5.1
aws-credentials:191.vcb_f183ce58b_9
aws-global-configuration:108.v47b_fd43dfec6
aws-java-sdk-ec2:1.12.406-374.v4cdf53953691
aws-java-sdk-minimal:1.12.406-374.v4cdf53953691
badge:1.9.1
basic-branch-build-strategies:71.vc1421f89888e
beer:42.v776b_04d96de3
blueocean:1.27.3
blueocean-bitbucket-pipeline:1.27.3
blueocean-commons:1.27.3
blueocean-config:1.27.3
blueocean-core-js:1.27.3
blueocean-dashboard:1.27.3
blueocean-display-url:2.4.1
blueocean-events:1.27.3
blueocean-git-pipeline:1.27.3
blueocean-github-pipeline:1.27.3
blueocean-i18n:1.27.3
blueocean-jwt:1.27.3
blueocean-personalization:1.27.3
blueocean-pipeline-api-impl:1.27.3
blueocean-pipeline-editor:1.27.3
blueocean-pipeline-scm-api:1.27.3
blueocean-rest:1.27.3
blueocean-rest-impl:1.27.3
blueocean-web:1.27.3
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.2-2
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-blocker-plugin:1.7.8
build-discarder:139.v05696a_7fe240
build-name-setter:2.2.0
build-pipeline-plugin:1.5.8
build-timeout:1.28
build-user-vars-plugin:1.9
build-with-parameters:76.v9382db_f78962
buildresult-trigger:0.18
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:2.0.0
chucknorris:1.4
cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1
cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8
cloudbees-folder:6.815.v0dd5a_cb_40e0e
code-coverage-api:4.1.0
collapsing-console-sections:1.8.0
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
conditional-buildstep:1.4.2
config-file-provider:3.11.1
configuration-as-code:1625.v27444588cc3d
configurationslicing:430.v966357576543
copyartifact:686.v6fd37018d7c2
credentials:1224.vc23ca_a_9a_2cb_0
credentials-binding:523.vd859a_4b_122e6
dashboard-view:2.472.v9ff2a_e6a_c529
data-tables-api:1.13.3-3
delivery-pipeline-plugin:1.4.2
description-setter:1.10
display-url-api:2.3.7
docker-commons:419.v8e3cd84ef49c
docker-workflow:563.vd5d2e5c4007f
dtkit-api:3.0.2
durable-task:504.vb10d1ae5ba2f
echarts-api:5.4.0-3
elastic-axis:409.v304cee6e7b_55
email-ext:2.95
envinject:2.901.v0038b_6471582
envinject-api:1.199.v3ce31253ed13
environment-dashboard:1.1.10
external-monitor-job:203.v683c09d993b_9
favorite:2.4.1
font-awesome-api:6.3.0-2
forensics-api:2.1.0
git:5.0.0
git-client:4.2.0
git-parameter:0.9.18
git-server:99.va_0826a_b_cdfa_d
git-userContent:1.4
gitea:1.4.5
github:1.37.0
github-api:1.303-417.ve35d9dd78549
github-branch-source:1703.vd5a_2b_29c6cdc
github-scm-filter-aged-refs:0.2.0
github-scm-trait-commit-skip:0.4.0
greenballs:1.15.1
groovy:453.vcdb_a_c5c99890
groovy-postbuild:2.5
h2-api:1.4.199
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
heavy-job:1.1
htmlpublisher:1.31
implied-labels:0.11
instance-identity:142.v04572ca_5b_265
ionicons-api:45.vf54fca_5d2154
jackson2-api:2.14.2-319.v37853346a_229
jacoco:3.3.3
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-8
jaxb:2.3.8-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.27.3
jersey2-api:2.39.1-1
jira:3.9
jira-steps:2.0.165.v8846cf59f3db
jira-trigger:1.0.2
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.83
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.4-1
jsch:0.1.55.61.va_e9ee26616e7
junit:1189.v1b_e593637fa_e
junit-attachments:131.vef2117770555
junit-realtime-test-reporter:102.v3f020960a_646
kubernetes:3900.va_dce992317b_4
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
label-verifier:1.4
ldap:671.v2a_9192a_7419d
leastload:3.0.0
locale:262.ved03281fa_64f
lockable-resources:1131.vb_7c3d377e723
log-parser:2.3.0
mailer:448.v5b_97805e3767
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
maven-plugin:3.21
metrics:4.2.13-420.vea_2f17932dd6
mina-sshd-api-common:2.9.2-62.v199162f0a_2f8
mina-sshd-api-core:2.9.2-62.v199162f0a_2f8
momentjs:1.1.1
monitoring:1.92.0
multiple-scms:0.8
naginator:1.18.2
next-build-number:1.8
nodelabelparameter:1.11.0
notification:1.15
okhttp-api:4.10.0-132.v7a_7b_91cef39c
opentelemetry:2.12.0
pam-auth:1.10
parallel-test-executor:383.vde60119c849a
parameterized-trigger:2.45
permissive-script-security:0.7
pipeline-build-step:487.va_823138eee8b_
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2125.vddb_a_44a_d605e
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2125.vddb_a_44a_d605e
pipeline-model-extensions:2.2125.vddb_a_44a_d605e
pipeline-rest-api:2.32
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2125.vddb_a_44a_d605e
pipeline-stage-view:2.32
pipeline-utility-steps:2.15.1
plain-credentials:143.v1b_df8b_d3b_e48
platformlabeler:1990.v82417ccb_eee8
plugin-util-api:3.2.0
pollscm:1.3.1
popper-api:1.16.1-3
popper2-api:2.11.6-2
postbuild-task:1.9
postbuildscript:3.2.0-460.va_fda_0fa_26720
preSCMbuildstep:44.v6ef4fd97f56e
prism-api:1.29.0-4
prometheus:2.1.2
promoted-builds:892.vd6219fc0a_efb
pubsub-light:1.17
rebuild:1.34
resource-disposer:0.21
reverse-proxy-auth-plugin:1.7.4
run-condition:1.5
saml:4.385.v4dea_91565e9d
scm-api:631.v9143df5b_e4a_a
scoring-load-balancer:59.vf791549fa_989
script-security:1229.v4880b_b_e905a_6
slack:631.v40deea_40323b
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.26
ssh-agent:327.v230ecd01f86f
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.275.v9e17c10f2571
startup-trigger-plugin:2.9.3
strict-crumb-issuer:2.1.0
structs:324.va_f5d6774f3a_d
text-finder:1.23
throttle-concurrents:2.12
timestamper:1.23
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
uno-choice:2.6.4
variant:59.vf075fe829ccb
versioncolumn:95.v82f3985cd6e1
view-job-filters:364.v48a_33389553d
warnings-ng:10.1.0
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:1010.vf7a_b_98e847c1
workflow-cps:3653.v07ea_433c90b_4
workflow-durable-task-step:1241.v1a_63e465f943
workflow-job:1289.vd1c337fd5354
workflow-multibranch:733.v109046189126
workflow-scm-step:408.v7d5b_135a_b_d49
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44
xml-job-to-job-dsl:0.1.14-SNAPSHOT (private-00af21b1-runner)
xshell:129.v99c1dcfa_d3d2
xunit:3.1.2
acompton-slack commented 1 year ago

As for the operating system, we are using a repack of the Jenkins Docker image: https://hub.docker.com/layers/jenkins/jenkins/2.387.1-lts-jdk11/images/sha256-4b5c14cabb880c1080ac6d07bcefb379fbf31f821556073205ac28f14ed55cf2?context=explore. This error occurs before any agents can connect.

acompton-slack commented 1 year ago

I can report that the snapshot plugin from https://github.com/jenkinsci/opentelemetry-plugin/pull/623 fixed my issue perfectly.

cyrille-leclerc commented 1 year ago

Fixed by 2.14.0