search

jenkinsci / junit-sql-storage-plugin

https://plugins.jenkins.io/junit-sql-storage/
MIT License
6 stars 14 forks source link

Link OpenTelemetry traces to parent span when publishing test results #414

Open timja opened 2 months ago

timja commented 2 months ago

Requires https://github.com/jenkinsci/junit-plugin/pull/622

Amends https://github.com/jenkinsci/junit-sql-storage-plugin/pull/413 to set it up so spans have the correct parent so publishing the test results is tracked properly

Testing done

Freestyle:

image

Tested on a remote agent for Pipeline:

image

Submitter checklist

timja commented 2 months ago

cc @cyrille-leclerc / @kuisathaverat is this a sensible approach to make this work properly?

cyrille-leclerc commented 2 months ago

FYI we are simplifying the usage of OpenTelemetry in plugins improving the APIs of the Jenkins OpenTelemetry Plugin:

I would like to experiment this integration on a plugin to be sure we define the right APIs

cyrille-leclerc commented 2 months ago

cc @cyrille-leclerc / @kuisathaverat is this a sensible approach to make this work properly?

Please see my comment above, there may be a problem in the Jenkins OTel Plugin

cyrille-leclerc commented 2 months ago

FYI I have a change in progress in the Jenkins OTel Plugin to properly propagate the trace context to the Junit SQL Storage plugin. I the evolution of the Jenkisn OTel Plugin is successful, then this PR in the Junit SQL Storage Plugin will not be needed.

Here is an example that also includes the instrumentation of the JDBC datasource

image
cyrille-leclerc commented 2 months ago

This PR is no longer needed. The problem is now solved in the Jenkins OTel Plugin by:

timja commented 2 months ago

On the latest version of the open telemetry plugin I can't seem to get the sql batch insert to show up at all not on master or this branch.

Are you running on the controller or a separate agent? Can you share more your configuration please?

cyrille-leclerc commented 1 month ago

Great ctch, I cleaned up my test environments and I still get the trace context propagation successful with the following:

junit-sql-storage:324.v90e2a_a_a_a_0dd7
opentelemetry:3.1320.v2eededb_d909e
Jenkins plugins version ``` Jenkins: 2.452.3 OS: Mac OS X - 14.5 Java: 21.0.3 - Homebrew (OpenJDK 64-Bit Server VM) --- ace-editor:1.1 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 asm-api:9.7-33.v4d23ef79fcc8 authentication-tokens:1.119.v50285141b_7e1 blueocean:1.27.13 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.13 blueocean-commons:1.27.13 blueocean-config:1.27.13 blueocean-core-js:1.27.13 blueocean-dashboard:1.27.13 blueocean-display-url:2.4.3 blueocean-events:1.27.13 blueocean-git-pipeline:1.27.13 blueocean-github-pipeline:1.27.13 blueocean-i18n:1.27.13 blueocean-jwt:1.27.13 blueocean-personalization:1.27.13 blueocean-pipeline-api-impl:1.27.13 blueocean-pipeline-editor:1.27.13 blueocean-pipeline-scm-api:1.27.13 blueocean-rest:1.27.13 blueocean-rest-impl:1.27.13 blueocean-web:1.27.13 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_ branch-api:2.1169.va_f810c56e895 build-timeout:1.33 byte-buddy-api:1.14.18-35.v791203db_06c0 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.0 cloudbees-bitbucket-branch-source:888.v8e6d479a_1730 cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_ cloudbees-folder:6.928.v7c780211d66e command-launcher:107.v773860566e2e commons-lang3-api:3.14.0-76.vda_5591261cfe commons-text-api:1.12.0-119.v73ef73f2345d config-file-provider:973.vb_a_80ecb_9a_4d0 configuration-as-code:1810.v9b_c30a_249a_4c credentials:1371.vfee6b_095f0a_3 credentials-binding:681.vf91669a_32e45 data-tables-api:2.0.8-1 database:999999-SNAPSHOT (private-ac02fa9f-cyrilleleclerc) database-postgresql:100.v2418e0a_c6909 display-url-api:2.204.vf6fddd8a_8b_e9 durable-task:555.v6802fe0f0b_82 echarts-api:5.5.0-1 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:1814.v404722f34263 extended-read-permission:53.v6499940139e5 favorite:2.218.vd60382506538 flyway-api:9.22.3-75.vfdfb_f75a_a_9b_e font-awesome-api:6.5.2-1 git:5.2.2 git-client:5.0.0 git-server:126.v0d945d8d2b_39 github:1.39.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1789.v5b_0c0cea_18c3 gradle:2.12 gson-api:2.11.0-41.v019fcf6125dc handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.35 instance-identity:185.v303dc7c645f9 ionicons-api:74.v93d5eb_813d5f jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javax-activation-api:1.2.0-7 javax-mail-api:1.6.2-10 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.13 jjwt-api:0.11.5-112.ve82dfb_224b_a_d jnr-posix-api:3.1.19-2 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1265.v65b_14fa_f12f0 junit-sql-storage:324.v90e2a_a_a_a_0dd7 ldap:725.v3cb_b_711b_1a_ef lockable-resources:1255.vf48745da_35d0 mailer:472.vf7c289a_4b_420 matrix-auth:3.2.2 matrix-project:832.va_66e270d2946 metrics:4.2.21-451.vd51df8df52ec mina-sshd-api-common:2.13.1-117.v2f1a_b_66ff91d mina-sshd-api-core:2.13.1-117.v2f1a_b_66ff91d momentjs:1.1.1 okhttp-api:4.11.0-172.vda_da_1feeb_c6e opentelemetry:3.1320.v2eededb_d909e opentelemetry-api:1.39.0-999999-SNAPSHOT (private-508a0f74-cyrilleleclerc) pam-auth:1.11 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-github-lib:61.v629f2cc41d83 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-groovy-lib:727.ve832a_9244dfa_ pipeline-input-step:495.ve9c153f6067b_ pipeline-maven:1421.v610fa_b_e2d60e pipeline-maven-api:1421.v610fa_b_e2d60e pipeline-maven-database:1421.v610fa_b_e2d60e pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2205.vc9522a_9d5711 pipeline-model-definition:2.2205.vc9522a_9d5711 pipeline-model-extensions:2.2205.vc9522a_9d5711 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2205.vc9522a_9d5711 pipeline-stage-view:2.34 plain-credentials:183.va_de8f1dd5a_2b_ plugin-util-api:4.1.0 popper-api:1.16.1-3 popper2-api:2.11.6-5 postgresql-api:42.7.2-40.v76d376d65c77 prism-api:1.29.0-15 pubsub-light:1.18 resource-disposer:0.23 scm-api:690.vfc8b_54395023 script-security:1341.va_2819b_414686 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.27 ssh-credentials:337.v395d2403ccd4 ssh-slaves:2.973.v0fa_8c0dea_f9f sshd:3.330.vc866a_8389b_58 structs:338.v848422169819 support-core:1461.v3ed77a_a_8e1c4 timestamper:1.27 token-macro:400.v35420b_922dcb_ trilead-api:2.147.vb_73cc728a_32e variant:60.v7290fc0eb_b_cd workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1316.v33eb_726c50b_a_ workflow-basic-steps:1058.vcb_fc1e3a_21a_9 workflow-cps:3908.vd6b_b_5a_a_54010 workflow-cps-global-lib:612.v55f2f80781ef workflow-durable-task-step:1360.v82d13453da_a_f workflow-job:1400.v7fd111b_ec82f workflow-multibranch:783.787.v50539468395f workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:678.v3ee58b_469476 workflow-support:920.v59f71ce16f04 ws-cleanup:0.46 ```
image
timja commented 1 month ago

@cyrille-leclerc how are you testing it? do you have a remote agent?

I just tried with this setup:

(Inbound agent secret will need replacing)

version: '3.1'

services:
  db:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: postgres
    ports:
      - 5432:5432

  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - "4317:4317"
      - "16686:16686"

  jenkins:
    build:
      context: .
    ports:
      - "8080:8080"
      - "50000:50000"
      - "5005:5005"
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
      OTEL_SERVICE_NAME: jenkins
      OTEL_METRICS_EXPORTER: none
      OTEL_LOGS_EXPORTER: none
      OTEL_JAVAAGENT_LOGGING: simple
      JAVA_OPTS: "-Djenkins.install.runSetupWizard=false -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

  jenkins-agent:
    image: jenkins/inbound-agent
    environment:
      JENKINS_URL: http://jenkins:8080
      JENKINS_SECRET: <replace-this>
      JENKINS_AGENT_NAME: agent
      JENKINS_AGENT_WORKDIR: /home/jenkins/agent

I was on the master branch of this repository and ran it with:

docker compose up -d --build

(note: I've made a couple of changes to the docker-compose files in the repo to test this fully)

and I don't get the sql trace:

image
timja commented 1 month ago

(you can use the master docker-compose if you add these vars:)

+    environment:
+      # Since v0.105 the OTEL Collector components default hostname to 'localhost'.
+      # However, that does not work inside a Docker container, so we listen on all IPs.
+      # see https://github.com/jaegertracing/jaeger/issues/5737
+      COLLECTOR_OTLP_GRPC_HOST_PORT: 0.0.0.0:4317
+      COLLECTOR_OTLP_HTTP_HOST_PORT: 0.0.0.0:4318

Since it'll be fixed in the next version I wasn't going to bother checking it in for now although I lost some time trying to figure out what was happening

cyrille-leclerc commented 1 month ago

Sorry for the delay. Problem reproduced. We didn't think of plugins writing code that runs on the jenkins build agent and that would want to instrument this code. We have to look at this.

cyrille-leclerc commented 1 week ago

Please test the latest Jenkins OpenTelemetry Plugin with the configuration properties:

otel.instrumentation.jenkins.agent.enabled=true
otel.instrumentation.jenkins.remoting.enabled=true
image

Ignore the "INSERT my_shopping_cart.caseResults" and "PoolingDataSource.getConnection" that are an experiment of instrumenting the Jenkins Database Plugin, it's not released yet.