search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
98 stars 51 forks source link

Pipeline console logs repeat "View logs in Elastic Observability" #665

Closed chriscarpenter12 closed 9 months ago

chriscarpenter12 commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.361.1 OS: Linux - 4.18.0-305.19.1.el8_4.x86_64 Java: 11.0.18 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.18.0 ace-editor:1.1 ant:1.11 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 authentication-tokens:1.4 basic-branch-build-strategies:71.vc1421f89888e bitbucket:223.vd12f2bca5430 blueocean:1.26.0 blueocean-autofavorite:1.2.4 blueocean-bitbucket-pipeline:1.26.0 blueocean-commons:1.26.0 blueocean-config:1.26.0 blueocean-core-js:1.26.0 blueocean-dashboard:1.26.0 blueocean-display-url:2.4.1 blueocean-events:1.26.0 blueocean-git-pipeline:1.26.0 blueocean-github-pipeline:1.26.0 blueocean-i18n:1.26.0 blueocean-jwt:1.26.0 blueocean-personalization:1.26.0 blueocean-pipeline-api-impl:1.26.0 blueocean-pipeline-editor:1.26.0 blueocean-pipeline-scm-api:1.26.0 blueocean-rest:1.26.0 blueocean-rest-impl:1.26.0 blueocean-web:1.26.0 bootstrap5-api:5.2.0-1 bouncycastle-api:2.26 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.4 cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1 cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.740.ve4f4ffa_dea_54 command-launcher:1.2 commons-httpclient3-api:3.1-3 commons-lang3-api:3.12.0.0 commons-text-api:1.9-9.v39a_53e2e0343 conditional-buildstep:1.4.1 config-file-provider:3.10.0 configuration-as-code:1512.vb_79d418d5fc8 configuration-as-code-groovy:1.1 copyartifact:1.48 credentials:1143.vb_e8b_b_ceee347 credentials-binding:523.vd859a_4b_122e6 cucumber-reports:5.7.5 dependency-check-jenkins-plugin:5.2.1 display-url-api:2.3.6 docker-commons:1.18 durable-task:496.va67c6f9eefa7 echarts-api:5.4.0-1 email-ext:2.92 favorite:2.4.1 font-awesome-api:6.1.2-1 generic-webhook-trigger:1.86.2 git:4.12.1 git-changelog:3.29 git-client:3.11.1 git-server:1.10 github:1.34.5 github-api:1.303-400.v35c2d8258028 github-branch-source:1696.v3a_7603564d04 google-oauth-plugin:1.0.6 groovy:2.4 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.25 http_request:1.16 ignore-committer-strategy:1.0.4 instance-identity:3.1 jackson2-api:2.13.4.20221013-295.v8e29ea_354141 jakarta-activation-api:2.0.1-1 jakarta-mail-api:2.0.1-1 javadoc:1.6 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-6 jaxb:2.3.7-1 jdk-tool:1.0 jenkins-design-language:1.26.0 jira:3.7.1 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.78.3 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 kubernetes:1.31.0 kubernetes-client-api:5.10.1-171.vaa0774fb8c20 kubernetes-credentials:0.9.0 lockable-resources:2.11 mailer:435.v79ef3972b_5c7 mapdb-api:1.0.9.0 mask-passwords:3.3 matrix-auth:2.6.8 matrix-project:772.v494f19991984 maven-plugin:3.16 mercurial:2.16.2 metrics:4.0.2.8.1 multibranch-action-triggers:1.8.6 oauth-credentials:0.4 okhttp-api:4.9.3-108.v0feda04578cf openshift-client:1.0.37 openshift-login:1.0.27 openshift-sync:1.0.53 opentelemetry:2.11.0 pam-auth:1.6 parameterized-trigger:2.45 pipeline-build-step:2.18 pipeline-graph-analysis:1.11 pipeline-groovy-lib:591.v3a_7f422b_d058 pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2097.v33db_b_de764b_e pipeline-model-definition:2.2097.v33db_b_de764b_e pipeline-model-extensions:2.2097.v33db_b_de764b_e pipeline-rest-api:2.15 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e pipeline-utility-steps:2.15.0 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.17.0 popper2-api:2.11.5-2 prometheus:2.0.10 pubsub-light:1.16 remote-file:1.23 run-condition:1.3 scm-api:631.v9143df5b_e4a_a script-security:1175.v4b_d517d6db_f0 snakeyaml-api:1.30.2-76.vc104f7ce9870 sonar:2.14 sse-gateway:1.24 ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 ssh-steps:2.0.65.vd26b_5b_9b_de4d sshd:3.236.ved5e1b_cb_50b_2 structs:324.va_f5d6774f3a_d subversion:2.15.5 token-macro:321.vd7cc1f2a_52c8 trilead-api:1.67.vc3938a_35172f uno-choice:2.6.4 variant:1.4 workflow-aggregator:581.v0c46fa_697ffd workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:969.vc4ec3e4854b_f workflow-cps:2759.v87459c4eea_ca_ workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1164.v2334ddcf48d0 workflow-job:1203.v7b_7023424efe workflow-multibranch:716.vc692a_e52371b_ workflow-remote-loader:1.5 workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:833.va_1c71061486b_ ```

What Operating System are you using (both controller, and any agents involved in the problem)?

OpenShift using the RedHat provided Jenkins container.

Reproduction steps

Enable the option to Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins

image

Run a build and view console output.

image

Expected Results

It shows a link to view the logs in Elastic Observability along with showing the log output

Actual Results

It just shows the repeated line to view logs in elastic

Anything else?

The link to the trace id in Elastic shows the correct log data in the log stream.

image

Adding the advanced option for otel.logs.mirror_to_disk=true does correctly show the logs in the Jenkins console, but then there's no link at the top of the page to see the trace id for the log stream in Elastic.

chriscarpenter12 commented 1 year ago

I just updated to a newer Jenkins instance and the latest version of this plugin and I'm still seeing the same issue.

Environment ```text Jenkins: 2.387.3 OS: Linux - 4.18.0-305.19.1.el8_4.x86_64 Java: 11.0.19 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.18.0 ace-editor:1.1 ant:487.vd79d090d4ea_e antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 authentication-tokens:1.53.v1c90fd9191a_b_ basic-branch-build-strategies:71.vc1421f89888e bitbucket:223.vd12f2bca5430 blueocean:1.27.4 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.4 blueocean-commons:1.27.4 blueocean-config:1.27.4 blueocean-core-js:1.27.4 blueocean-dashboard:1.27.4 blueocean-display-url:2.4.2 blueocean-events:1.27.4 blueocean-git-pipeline:1.27.4 blueocean-github-pipeline:1.27.4 blueocean-i18n:1.27.4 blueocean-jwt:1.27.4 blueocean-personalization:1.27.4 blueocean-pipeline-api-impl:1.27.4 blueocean-pipeline-editor:1.27.4 blueocean-pipeline-scm-api:1.27.4 blueocean-rest:1.27.4 blueocean-rest-impl:1.27.4 blueocean-web:1.27.4 bootstrap5-api:5.2.2-3 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloudbees-bitbucket-branch-source:809.vc1d904b_30426 cloudbees-disk-usage-simple:182.v62ca_0c992a_f3 cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:100.v2f6722292ee8 commons-httpclient3-api:3.1-3 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ conditional-buildstep:1.4.2 config-file-provider:938.ve2b_8a_591c596 configuration-as-code:1647.ve39ca_b_829b_42 configuration-as-code-groovy:1.1 copyartifact:705.v5295cffec284 credentials:1254.vb_96f366e7b_a_d credentials-binding:604.vb_64480b_c56ca_ cucumber-reports:5.7.5 data-tables-api:1.13.3-3 dependency-check-jenkins-plugin:5.4.0 display-url-api:2.3.7 docker-commons:419.v8e3cd84ef49c durable-task:506.v1b_3e14b_6f5da_ echarts-api:5.4.0-3 email-ext:2.99 favorite:2.4.1 font-awesome-api:6.3.0-2 generic-webhook-trigger:1.86.3 git:5.1.0 git-changelog:3.30 git-client:4.3.0 git-server:99.va_0826a_b_cdfa_d github:1.37.0 github-api:1.303-417.ve35d9dd78549 github-branch-source:1728.v859147241f49 google-oauth-plugin:1.0.8 groovy:453.vcdb_a_c5c99890 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 http_request:1.16 ignore-committer-strategy:1.0.4 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:233.vdc1a_ec702cff javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:66.vd8fa_64ee91b_d jenkins-design-language:1.27.4 jersey2-api:2.39.1-1 jira:3.9 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.83 jquery3-api:3.6.4-1 jsch:0.2.8-65.v052c39de79b_2 junit:1202.v79a_986785076 kubernetes:3923.v294a_d4250b_91 kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 lockable-resources:1156.v5e9f897ece02 mailer:457.v3f72cb_e015e5 mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:150.vf80d33113e80 matrix-auth:3.1.7 matrix-project:789.v57a_725b_63c79 maven-plugin:3.22 mercurial:1260.vdfb_723cdcc81 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ momentjs:1.1.1 multibranch-action-triggers:1.8.6 oauth-credentials:0.645.ve666a_c332668 okhttp-api:4.10.0-136.v1d8ce1b_1db_72 openshift-client:1.1.0.413.v3023d27e8434 openshift-login:1.1.0.227.v27e08dfb_1a_20 openshift-sync:1.1.0.790.v2051fca_5ed8d opentelemetry:2.13.0 pam-auth:1.10 parameterized-trigger:2.45 pipeline-build-step:491.v1fec530da_858 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:468.va_5db_051498a_4 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2131.vb_9788088fdb_5 pipeline-model-definition:2.2131.vb_9788088fdb_5 pipeline-model-extensions:2.2131.vb_9788088fdb_5 pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2131.vb_9788088fdb_5 pipeline-stage-view:2.32 pipeline-utility-steps:2.15.4 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.2.0 prometheus:2.2.2 pubsub-light:1.17 remote-file:1.23 run-condition:1.5 scm-api:672.v64378a_b_20c60 script-security:1251.vfe552ed55f8d snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sonar:2.15 sse-gateway:1.26 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec ssh-steps:2.0.65.vd26b_5b_9b_de4d sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d subversion:2.17.2 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 uno-choice:2.6.5 variant:59.vf075fe829ccb workflow-aggregator:581.v0c46fa_697ffd workflow-api:1213.v646def1087f9 workflow-basic-steps:1017.vb_45b_302f0cea_ workflow-cps:3659.v582dc37621d8 workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1246.v5524618ea_097 workflow-job:1295.v395eb_7400005 workflow-multibranch:746.v05814d19c001 workflow-remote-loader:1.5 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ```

image

And I'm not seeing any errors in the Jenkins logs. I added otel.logs.exporter=logging to advanced configuration properties hoping to get more output too. I just see the build logs in the agent stdout

image

cyrille-leclerc commented 1 year ago

We are sorry for the inconvenience @chriscarpenter12 . It's unfortunately complicated for me to help you as I no longer use Elasticsearch. Could anyone using Elastic have a look at this issue? cc @kuisathaverat

kuisathaverat commented 1 year ago

I saw this behaviour before in the Jenkins Console output (independently of the Opentelemetry plugin), the /logText/progressiveHtml request start=0, and the plugin returned that, the logs from start=0, this makes the UI to duplicate the content in the page. In the Opentelemetry plugin with output to Elasticsearch only it is more consistent because there are no logs. What we could do is return the message only on the first request to start=0, it is kind of a workaround for a core problem.

kuisathaverat commented 1 year ago

I just discovered it is related to https://github.com/jenkinsci/opentelemetry-plugin/issues/683 it happens only when you want to store a copy of the logs locally, sending the logs only to Elasticsearch does not occur. The option to store the logs locally is not working as before 2.10.