search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
98 stars 50 forks source link

Pipeline log console screen is broken when the Elasticsearch retrieval fails on a timeout #421

Closed cyrille-leclerc closed 11 months ago

cyrille-leclerc commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.332.2 OS: Mac OS X - 11.3 --- ace-editor:1.1 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 blueocean:1.25.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest:1.25.3 blueocean-rest-impl:1.25.3 blueocean-web:1.25.3 bootstrap4-api:4.6.0-4 bootstrap5-api:5.1.3-6 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-timeout:1.20 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.3 cloudbees-bitbucket-branch-source:765.v5a_2d6a_23c01d cloudbees-disk-usage-simple:0.10 cloudbees-folder:6.714.v79e858ef76a_2 command-launcher:1.6 credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.6 durable-task:496.va67c6f9eefa7 echarts-api:5.3.2-1 email-ext:2.88 extended-read-permission:3.2 favorite:2.4.1 font-awesome-api:6.0.0-1 git:4.11.1 git-client:3.11.0 git-server:1.10 github:1.34.3 github-api:1.303-400.v35c2d8258028 github-branch-source:1598.v91207e9f9b_4a_ gradle:1.38 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 jackson2-api:2.13.2.20220328-273.v11d70a_b_a_1a_52 javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.3 jjwt-api:0.11.2-71.v2722b_b_06a_2a_f jnr-posix-api:3.1.7-3 jquery3-api:3.6.0-3 jsch:0.1.55.2 junit:1.60 ldap:2.9 lockable-resources:2.14 mailer:408.vd726a_1130320 matrix-auth:2.6.9 matrix-project:758.v7a_ea_491852f3 metrics:4.1.6.2 momentjs:1.1.1 okhttp-api:4.9.3-105.vb96869f8ac3a opentelemetry:2.3.1-SNAPSHOT (private-17d7ac53-cyrilleleclerc) pam-auth:1.7 pipeline-build-step:2.18 pipeline-github-lib:36.v4c01db_ca_ed16 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-input-step:448.v37cea_9a_10a_70 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2077.vc78ec45162f1 pipeline-model-definition:2.2077.vc78ec45162f1 pipeline-model-extensions:2.2077.vc78ec45162f1 pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2077.vc78ec45162f1 pipeline-stage-view:2.24 plain-credentials:1.8 plugin-util-api:2.16.0 popper-api:1.16.1-3 popper2-api:2.11.5-1 pubsub-light:1.16 resource-disposer:0.19 scm-api:602.v6a_81757a_31d2 script-security:1158.v7c1b_73a_69a_08 snakeyaml-api:1.30.1 sse-gateway:1.25 ssh-credentials:277.v95c2fec1c047 ssh-slaves:1.814.vc82988f54b_10 sshd:3.228.v4c9f9e652c86 structs:318.va_f3ccb_729b_71 support-core:1162.vb_b_e5198c6b_22 timestamper:1.17 token-macro:293.v283932a_0a_b_49 trilead-api:1.57.v6e90e07157e1 variant:1.4 workflow-aggregator:2.7 workflow-api:1144.v61c3180fa_03f workflow-basic-steps:948.v2c72a_091b_b_68 workflow-cps:2689.v434009a_31b_f1 workflow-cps-global-lib:570.v21311f4951f8 workflow-durable-task-step:1130.v8fd69d0b_8857 workflow-job:1180.v04c4e75dce43 workflow-multibranch:712.vc169a_1387405 workflow-scm-step:399.v9b_8f4da_65061 workflow-step-api:625.vd896b_f445a_f8 workflow-support:819.v37d707a_71d9b_ ws-cleanup:0.42 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

-

Reproduction steps

Expected Results

Actual Results

2022-05-01 21:47:44.717+0000 [id=398]   WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving http://localhost:8080/job/ecommerce-antifraud/job/main/156/logText/progressiveHtml
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
    at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
    at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
Caused: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-1 [ACTIVE]
    at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:903)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:299)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:287)
    at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:146)
    at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1526)
    at io.jenkins.plugins.opentelemetry.backend.elastic.ElasticsearchBuildLogsLineIterator.loadNextFormattedLogLines(ElasticsearchBuildLogsLineIterator.java:203)
    at io.jenkins.plugins.opentelemetry.backend.elastic.ElasticsearchBuildLogsLineIterator.getCurrentIterator(ElasticsearchBuildLogsLineIterator.java:108)
Caused: java.lang.RuntimeException
    at io.jenkins.plugins.opentelemetry.backend.elastic.ElasticsearchBuildLogsLineIterator.getCurrentIterator(ElasticsearchBuildLogsLineIterator.java:122)
    at io.jenkins.plugins.opentelemetry.backend.elastic.ElasticsearchBuildLogsLineIterator.hasNext(ElasticsearchBuildLogsLineIterator.java:156)
    at io.jenkins.plugins.opentelemetry.job.log.util.LineIteratorInputStream.readLine(LineIteratorInputStream.java:71)
    at io.jenkins.plugins.opentelemetry.job.log.util.LineIteratorInputStream.read(LineIteratorInputStream.java:49)
    at java.base/java.io.InputStream.read(InputStream.java:271)
    at org.kohsuke.stapler.framework.io.LargeText$BufferSession.read(LargeText.java:569)
    at org.kohsuke.stapler.framework.io.LargeText$ByteBuf.<init>(LargeText.java:384)
    at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:237)
    at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:180)
    at io.jenkins.plugins.opentelemetry.job.log.OverallLog.writeHtmlTo(OverallLog.java:169)
    at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:152)
    at io.jenkins.plugins.opentelemetry.job.log.OverallLog.writeLogTo(OverallLog.java:96)
    at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:279)
    at io.jenkins.plugins.opentelemetry.job.log.OverallLog.doProgressText(OverallLog.java:206)
    at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:95)
    at io.jenkins.plugins.opentelemetry.job.log.OverallLog.doProgressiveHtml(OverallLog.java:53)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:398)
Caused: java.lang.reflect.InvocationTargetException
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:402)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:410)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:208)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:141)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:224)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at io.jenkins.plugins.opentelemetry.servlet.OpenTelemetryServletFilter._doFilter(OpenTelemetryServletFilter.java:240)
    at io.jenkins.plugins.opentelemetry.servlet.OpenTelemetryServletFilter.doFilter(OpenTelemetryServletFilter.java:56)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:102)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Thread.java:829)

Anything else?

No response

kuisathaverat commented 1 year ago

I would say that is expected if the backend is unavailable or it takes too much time to retrieve the logs Jenkins will fail, I do not see a way to retrieve the logs if the backend does not respond.