Jenkins Overview dashboard in Elastic Observability is missing data

Nezz commented 2 years ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.319.3 OS: Linux - 4.4.0-1128-aws --- ace-editor:1.1 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df aws-java-sdk:1.12.163-315.v2b_716ec8e4df badge:1.9.1 basic-branch-build-strategies:1.3.2 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest-impl:1.25.3 blueocean-rest:1.25.3 blueocean-web:1.25.3 blueocean:1.25.3 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.7.0 build-timeout:1.20 build-user-vars-plugin:1.8 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-bitbucket-branch-source:757.vddedc5f2589a_ cloudbees-folder:6.17 cloudbees-jenkins-advisor:3.3.2 cobertura:1.17 code-coverage-api:2.0.4 command-launcher:1.6 conditional-buildstep:1.4.2 config-file-provider:3.9.0 copyartifact:1.46.3 credentials-binding:1.27.1 credentials:2.6.1 data-tables-api:1.11.4-4 description-setter:1.10 disable-github-multibranch-status:1.2 display-url-api:2.3.5 docker-commons:1.19 docker-workflow:1.28 dtkit-api:3.0.0 durable-task:493.v195aefbb0ff2 ec2:1.68 echarts-api:5.3.2-1 extended-choice-parameter:0.82 extended-read-permission:3.2 external-monitor-job:191.v363d0d1efdf8 favorite:2.4.1 font-awesome-api:6.0.0-1 forensics-api:1.12.0 ghprb:1.42.2 git-client:3.11.0 git-parameter:0.9.16 git-server:1.10 git:4.11.0 github-api:1.301-378.v9807bd746da5 github-branch-source:2.11.4 github-oauth:0.37 github-scm-trait-notification-context:1.1 github:1.34.3 global-build-stats:1.5 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.29 jackson2-api:2.13.2.20220328-273.v11d70a_b_a_1a_52 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.3 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-3 jobConfigHistory:2.31-rc1107.2354f08725a_8 jquery-detached:1.2.1 jquery:1.12.4-1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.54 ldap:2.8 lockable-resources:2.14 log-parser:2.2 lucene-search:370.v62a5f618cd3a mailer:408.vd726a_1130320 managed-scripts:1.5.4 mapdb-api:1.0.9.0 matrix-auth:2.6.8 matrix-project:771.v574584b_39e60 maven-plugin:3.16 metrics:4.1.6.1 momentjs:1.1.1 monitoring:1.91.0 node-iterator-api:1.5.1 nodelabelparameter:1.10.3.1 nunit:0.27 okhttp-api:4.9.3-105.vb96869f8ac3a opentelemetry:2.5.1 pam-auth:1.7 parameterized-scheduler:1.0 parameterized-trigger:2.44 pipeline-aws:1.43 pipeline-build-step:2.18 pipeline-github-lib:36.v4c01db_ca_ed16 pipeline-github:2.8-138.d766e30bb08b pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.23 pipeline-utility-steps:2.12.0 plain-credentials:1.8 plugin-usage-plugin:2.2 plugin-util-api:2.16.0 popper-api:1.16.1-2 popper2-api:2.11.2-1 pubsub-light:1.16 rebuild:1.33 resource-disposer:0.18 robot:3.1.0 role-strategy:3.2.0 run-condition:1.5 saml:2.1.1-275.va_5718591a_999 scm-api:602.v6a_81757a_31d2 script-security:1138.v8e727069a_025 slack:602.v0da_f7458945d snakeyaml-api:1.29.1 sse-gateway:1.25 ssh-agent:1.24.1 ssh-credentials:1.19 ssh-slaves:1.33.0 sshd:3.1.0 structs:308.v852b473a2b8c support-core:2.80 test-results-analyzer:0.3.5 thinBackup:1.10 throttle-concurrents:2.7 timestamper:1.17 token-macro:285.vff7645a_56ff0 trilead-api:1.0.13 uno-choice:2.6.1 variant:1.4 versioncolumn:2.2 windows-slaves:1.8 workflow-aggregator:2.7 workflow-api:1143.v2d42f1e9dea_5 workflow-basic-steps:2.24 workflow-cps-global-lib:564.ve62a_4eb_b_e039 workflow-cps:2660.vb_c0412dc4e6d workflow-durable-task-step:1121.va_65b_d2701486 workflow-job:1145.v7f2433caa07f workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:813.vb_d7c3d2984a_0 ws-cleanup:0.41 xunit:3.0.6 ```

Note: The provided script did not work for me. I had to remove the sort step and then it worked.

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu Elastic 8.1.3

Reproduction steps

We installed the OpenTelemetry plugin and enabled Elastic Observability. We did not enable "Store pipeline logs in Elastic Observability 8.0+ - Tech Preview".

Noteworthy active configuration properties

otel.exporter.otlp.endpoint=https://<redacted>.aws.found.io
otel.metrics.exporter=otlp
otel.traces.exporter=otlp

Expected Results

All metrics should function on the dashboard.

Actual Results

Several metrics are not working.

Example underlying errors:

Anything else?

No response

Nezz commented 2 years ago

These logs might be relevant:


May 04, 2022 10:36:13 AM INFO io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider initialize
OpenTelemetry SDK initialized: SDK [config: otel.traces.exporter=otlp, otel.metrics.exporter=otlp, otel.exporter.otlp.endpoint=https://<redacted>.aws.found.io, resource: service.name=jenkins, service.namespace=jenkins, service.version=2.319.3]

May 04, 2022 10:39:08 AM WARNING io.opentelemetry.sdk.internal.ThrottlingLogger doLog
Failed to export spans. Server responded with HTTP status code 200. Error message: OK

May 04, 2022 10:39:13 AM WARNING io.opentelemetry.sdk.internal.ThrottlingLogger doLog
Failed to export metrics. Server responded with HTTP status code 200. Error message: OK

swilliams-dev commented 2 years ago

I too am getting this issue on the latest demo stack, seems to be related to: https://github.com/jenkinsci/opentelemetry-plugin/issues/378

swilliams-dev commented 2 years ago

Updating to 2.5.1 fixed it for me

cyrille-leclerc commented 2 years ago

Hello @swilliams-dev and @Nezz You are likely to have been impacted by #424 and installing 2.5.1 fixed the problem thanks to the restart. We are working at fixing the cause with:

426

Sorry for the inconvenience

cyrille-leclerc commented 2 years ago

Can you please test https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.6.0-rc1 ? The plugin HPI file is attached to the release notes.

jenkinsci / opentelemetry-plugin