search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
100 stars 53 forks source link

Logs not shown in BlueOcean with otel.logs.mirror_to_disk=true #563

Closed pgchb closed 1 year ago

pgchb commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.361.2 OS: Linux - 5.4.219-126.411.amzn2.x86_64 --- Parameterized-Remote-Trigger:3.1.6.3 ace-editor:1.1 ansicolor:1.0.2 antisamy-markup-formatter:155.v795fb_8702324 anything-goes-formatter:1.0 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 authentication-tokens:1.4 badge:1.9.1 basic-branch-build-strategies:71.vc1421f89888e blueocean:1.25.8 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.8 blueocean-commons:1.25.8 blueocean-config:1.25.8 blueocean-core-js:1.25.8 blueocean-dashboard:1.25.8 blueocean-display-url:2.4.1 blueocean-events:1.25.8 blueocean-git-pipeline:1.25.8 blueocean-github-pipeline:1.25.8 blueocean-i18n:1.25.8 blueocean-jwt:1.25.8 blueocean-personalization:1.25.8 blueocean-pipeline-api-impl:1.25.8 blueocean-pipeline-editor:1.25.8 blueocean-pipeline-scm-api:1.25.8 blueocean-rest:1.25.8 blueocean-rest-impl:1.25.8 blueocean-web:1.25.8 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1051.v9985666b_f6cc caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.0 cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.758.vfd75d09eea_a_1 command-launcher:1.2 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 configuration-as-code:1569.vb_72405b_80249 copyartifact:1.47 credentials:1189.vf61b_a_5e2f62e credentials-binding:523.vd859a_4b_122e6 data-tables-api:1.12.1-4 display-url-api:2.3.6 docker-commons:1.21 docker-workflow:528.v7c193a_0b_e67c dtkit-api:3.0.2 durable-task:501.ve5d4fc08b0be echarts-api:5.4.0-1 email-ext:2.92 extended-read-permission:3.2 favorite:2.4.1 font-awesome-api:6.2.0-3 git:4.13.0 git-client:3.13.0 github:1.36.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1696.v3a_7603564d04 groovy-postbuild:2.5 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 hashicorp-vault-plugin:359.v2da_3b_45f17d5 htmlpublisher:1.31 http_request:1.16 instance-identity:116.vf8f487400980 ionicons-api:31.v4757b_6987003 jackson2-api:2.13.4.20221013-295.v8e29ea_354141 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-5 jaxb:2.3.7-1 jdk-tool:1.0 jenkins-design-language:1.25.8 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.81 jobConfigHistory:1176.v1b_4290db_41a_5 jquery-detached:1.2.1 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1160.vf1f01a_a_ea_b_7f kubernetes:3734.v562b_b_a_627ea_c kubernetes-cli:1.10.3 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 ldap:2.12 locale:204.v2a_f305fe7e9d lockable-resources:1069.v726298f53f8c mailer:438.v02c7f0a_12fa_4 matrix-auth:2.6.11 matrix-project:785.v06b_7f47b_c631 metrics:4.2.10-405.v60a_9cc74e923 momentjs:1.1.1 monitoring:1.91.0 okhttp-api:4.9.3-108.v0feda04578cf opentelemetry:2.10.0 permissive-script-security:0.7 pipeline-build-step:2.18 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:621.vb_44ce045b_582 pipeline-input-step:456.vd8a_957db_5b_e9 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-rest-api:2.27 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.27 pipeline-utility-steps:2.14.0 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.18.0 popper2-api:2.11.6-2 prometheus:2.0.10 pubsub-light:1.17 robot:3.3.0 role-strategy:3.2.0 scm-api:621.vda_a_b_055e58f7 script-security:1218.v39ca_7f7ed0a_c simple-theme-plugin:136.v23a_15f86c53d snakeyaml-api:1.33-90.v80dcb_3814d35 sse-gateway:1.26 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 sshd:3.236.ved5e1b_cb_50b_2 structs:324.va_f5d6774f3a_d timestamper:1.21 token-macro:321.vd7cc1f2a_52c8 trilead-api:1.67.vc3938a_35172f variant:59.vf075fe829ccb windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:3536.vb_8a_6628079d5 workflow-durable-task-step:1217.v38306d8fa_b_5c workflow-job:1254.v3f64639b_11dd workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c xunit:3.1.2 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux (Pipeline runs on master)

Reproduction steps

  1. Configure OpenTelemetry plugin to "Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins"
  2. Enable otel.logs.mirror_to_disk=true
  3. Execute a pipeline with single echo step: echo "Hello\nWorld"

Expected Results

Output of echo step is shown in BlueOcean view and the Pipeline steps view of the pipeline HelloWorldOk

Actual Results

HelloWorldError

Anything else?

For the flowNode of the step (e.g. used by BlueOcean to retrieve logs):

LogAction logAction = flowNode.getAction(LogAction.class)
AnnotatedLargeText<? extends FlowNode> logText = logAction.getLogText()

logText.length() is 0 logText.writeLogTo(0, writer) writes no data

cyrille-leclerc commented 2 years ago

@AndriiChuzhynov did you test BlueOcean with otel.logs.mirror_to_disk=true ?

AndriiChuzhynov commented 2 years ago

Hi, no I didn't test with BlueOcean. Just checked and can confirm that it's not working :( Checking how we can fix it.

AndriiChuzhynov commented 1 year ago

Had a chance to work a bit on this problem. Looks like log-index is not creating with the mirror_to_disk option enabled. I'll continue investigate why

AndriiChuzhynov commented 1 year ago

After some time trying to make it works with the decorator, I've decided to go without a decorator. Looks like the decorator is not meant to work with replaced default LogStorage. Probably it may work well if FileLogStorage is default and Otel is applied as a decorator. But in this case, we'll lose the benefit of not sending logs back to the master without the mirroring option.

I've tried to go with merged streams when both streams are called. In this case, we have all log data in BlueOcean.

I've created draft PR #579 to check if it's aligned with plugin standards @cyrille-leclerc what do you think?

cyrille-leclerc commented 1 year ago

In progress with

cyrille-leclerc commented 1 year ago

Can you please test https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.12.0-rc1

pgchb commented 1 year ago

It works without any issues on my side with 2.12.0-rc1. It fixes as well the issue of #564 to retrieve only specific step logs, as long as otel.logs.mirror_to_disk is set to true.

Thank you (and all other involved persons) for providing this fix!

cyrille-leclerc commented 1 year ago

Thanks @AndriiChuzhynov !