Stop creating empty spans with /job/:jobFullName/:runNumber/logText/progressiveHtml

stmlange commented 1 year ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.387.3 OS: Linux - 5.4.0-91-generic Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.18.0 ace-editor:1.1 analysis-model-api:11.2.0 ant:487.vd79d090d4ea_e antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 authentication-tokens:1.53.v1c90fd9191a_b_ blueocean:1.27.4 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.4 blueocean-commons:1.27.4 blueocean-config:1.27.4 blueocean-core-js:1.27.4 blueocean-dashboard:1.27.4 blueocean-display-url:2.4.2 blueocean-events:1.27.4 blueocean-git-pipeline:1.27.4 blueocean-github-pipeline:1.27.4 blueocean-i18n:1.27.4 blueocean-jira:1.27.4 blueocean-jwt:1.27.4 blueocean-personalization:1.27.4 blueocean-pipeline-api-impl:1.27.4 blueocean-pipeline-editor:1.27.4 blueocean-pipeline-scm-api:1.27.4 blueocean-rest:1.27.4 blueocean-rest-impl:1.27.4 blueocean-web:1.27.4 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.2-3 bouncycastle-api:2.27 branch-api:2.1092.vda_3c2a_a_f0c11 build-blocker-plugin:1.7.8 build-name-setter:2.2.0 build-timeout:1.30 build-timestamp:1.0.3 build-with-parameters:76.v9382db_f78962 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1 cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:100.v2f6722292ee8 commons-httpclient3-api:3.1-3 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ compact-columns:1.169.v60b_7d496b_860 conditional-buildstep:1.4.2 console-column-plugin:197.vcf5a_ec1d7b_47 copyartifact:698.v393f578eb_ddc credentials:1254.vb_96f366e7b_a_d credentials-binding:604.vb_64480b_c56ca_ data-tables-api:1.13.3-3 display-url-api:2.3.7 docker-commons:419.v8e3cd84ef49c docker-java-api:3.2.13-68.va_875df25a_b_45 docker-plugin:1.3.0 docker-workflow:563.vd5d2e5c4007f durable-task:507.v050055d0cb_dd echarts-api:5.4.0-3 email-ext:2.97 envinject:2.901.v0038b_6471582 envinject-api:1.199.v3ce31253ed13 extended-read-permission:3.2 external-monitor-job:203.v683c09d993b_9 fail-the-build-plugin:5.v153b_2c826ef0 favorite:2.4.1 flexible-publish:0.16.1 font-awesome-api:6.3.0-2 forensics-api:2.1.0 git:5.0.2 git-client:4.2.0 git-server:99.va_0826a_b_cdfa_d github:1.37.0 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1703.vd5a_2b_29c6cdc gitlab-api:5.2.0-86.v1ed41a_9cf486 gitlab-branch-source:659.va_685a_51fda_db_ gitlab-plugin:1.7.13 gradle:2.7 grayballs:0.12-SNAPSHOT (private-09/01/2020 18:58-matej) handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 jackson2-api:2.15.0-334.v317a_165f9b_7c jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:233.vdc1a_ec702cff javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:66.vd8fa_64ee91b_d jenkins-design-language:1.27.4 jersey2-api:2.39.1-1 jira:3.9 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.83 jquery-detached:1.2.1 jquery3-api:3.6.4-1 jsch:0.2.8-65.v052c39de79b_2 junit:1202.v79a_986785076 ldap:676.vfa_64cf6b_b_002 lockable-resources:1156.v5e9f897ece02 mailer:448.v5b_97805e3767 mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.1.7 matrix-combinations-parameter:1.3.2 matrix-project:789.v57a_725b_63c79 matrix-reloaded:1.1.3 maven-plugin:3.22 mercurial:1260.vdfb_723cdcc81 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ momentjs:1.1.1 multi-slave-config-plugin:1.2.0 multiple-scms:0.8 naginator:1.18.2 okhttp-api:4.10.0-132.v7a_7b_91cef39c opentelemetry:2.13.0 ownership:0.13.0 pam-auth:1.10 parameterized-trigger:2.45 permissive-script-security:0.7 pipeline-build-step:491.v1fec530da_858 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:468.va_5db_051498a_4 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2131.vb_9788088fdb_5 pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:2.2131.vb_9788088fdb_5 pipeline-model-extensions:2.2131.vb_9788088fdb_5 pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2131.vb_9788088fdb_5 pipeline-stage-view:2.32 plain-credentials:143.v1b_df8b_d3b_e48 plot:2.1.12 plugin-util-api:3.2.1 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-4 pubsub-light:1.17 rebuild:320.v5a_0933a_e7d61 resource-disposer:0.22 role-strategy:633.v836e5b_3e80a_5 run-condition:1.5 scm-api:667.v8b_6e07cdc7f2 scm-sync-configuration:0.0.11-SNAPSHOT (private-a8c53d6f-pjohnston) script-security:1244.ve463715a_f89c snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sse-gateway:1.26 ssh-agent:333.v878b_53c89511 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d subversion:2.17.2 throttle-concurrents:2.13 timestamper:1.25 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb violation-comments-to-gitlab:2.59 violations:0.7.11 warnings-ng:10.1.0 windows-slaves:1.8.1 workflow-aggregator:596.v8c21c963d92d workflow-api:1213.v646def1087f9 workflow-basic-steps:1017.vb_45b_302f0cea_ workflow-cps:3659.v582dc37621d8 workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1246.v5524618ea_097 workflow-job:1295.v395eb_7400005 workflow-multibranch:746.v05814d19c001 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

Setup OTEL Collector with jaeger backend
Watch that empty spans only containing a /job/:jobFullName/:runNumber/logText/progressiveHtml are created in jaeger

Expected Results

Spans like /job/:jobFullName/:runNumber/logText/progressiveHtml that do not contain any other information are not created.

Actual Results

Spans like /job/:jobFullName/:runNumber/logText/progressiveHtml are created

Anything else?

No response

cyrille-leclerc commented 1 year ago

These spans are created because the Jenkins GUI is retrieving logs from the browser side. What problem is caused by tracing this HTTP request?

stmlange commented 1 year ago

It's not a problem, but I don't really see the purpose or benefit of such traces. For me they are essentially wasting disk space and polluting jaeger with unrelevant traces.

cyrille-leclerc commented 1 year ago

Thanks @stmlange , this trace is useful when troubleshooting problems with Jenkins pipeline logs persisted in an external storage like Elasticsearch and, in such cases, there are more spans to the trace. Could you by any chance use the OpenTelemetry Collector Filter Processor to delete these traces?

stmlange commented 1 year ago

Hello, I finally had the chance to play around with the filter-processor. I must admit that there seems to be a lot of such spans created so here is the current version I ended up with, that filters almost all spans:

processors:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
  filter/noisy_traces_url:
    error_mode: ignore
    traces:
      span:
        - IsMatch(resource.attributes["http.route"], "/administrativeMonitor/.*")
        - 'resource.attributes["http.route"] == "/ajaxBuildQueue"'
        - 'resource.attributes["http.route"] == "/ajaxExecutors"'
        - 'resource.attributes["http.route"] == "/asynchPeople"'
        - IsMatch(resource.attributes["http.route"], "/blue/.*")
        - IsMatch(resource.attributes["http.route"], "/configSubmit/.*")
        - IsMatch(resource.attributes["http.route"], "/configure/.*")
        - IsMatch(resource.attributes["http.route"], "/computer/.*")
        - IsMatch(resource.attributes["http.route"], "/fingerprintCheck/.*")
        - IsMatch(resource.attributes["http.route"], "/job/.*")
        - IsMatch(resource.attributes["http.route"], "/label/.*")
        - IsMatch(resource.attributes["http.route"], "/lockable-resources/.*")
        - IsMatch(resource.attributes["http.route"], "/login/.*")
        - 'resource.attributes["http.route"] == "/manage/\*"'
        - IsMatch(resource.attributes["http.route"], "/manage/.*")
        - IsMatch(resource.attributes["http.route"], "/me/.*")
        - IsMatch(resource.attributes["http.route"], "/script/.*")
        - IsMatch(resource.attributes["http.route"], "/tcpSlaveAgentListener/.*")
        - IsMatch(resource.attributes["http.route"], "/user/:user/.*")
        - 'resource.attributes["http.route"] == "/view/all/builds"'
service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - batch
        - filter/noisy_traces_url
      exporters:
        - otlp

Despite using a IsMatch(resource.attributes["http.route"], "/job/.*") I still see sometimes traces with http.route=/job/:jobFullName/:runNumber/execution/:execution/*. Is there perhaps a flaw in the rules above? Otherwise feel free to close this ticket.

stmlange commented 1 year ago

Just as follow up: The filter tracing is IMHO too cumbersome to setup! Running any Vulnerability Scanner like the Burp Suite causes yet another set of zillion pointless traces with who knoes what of a http.route. Filtering for IPs is pointless, since they can change without notice.

Hence setting up filter rules for Vulnerability Scanner would be insanity (unless someone wants to encode 5000+ filtering conditions). Hence I think the initial request to have a more principled solution is still valid. In short we only want to have traces created by actual jobs. E.g. from the job I have the fancy button of viewing the trace. Some users might need the ability to trace every web-request, but I still do not think this is an every-day use-case.

cyrille-leclerc commented 1 year ago

I'm sorry for the pain. If you are interested in disabling all the HTTP traces, you can do so using the configuration property otel.instrumentation.jenkins.web.enabled=false Would that solve your problem or do you want to keep some HTTP traces?

stmlange commented 1 year ago

Hello, thanks for pointing me to otel.instrumentation.jenkins.web.enabled=false. That is exactly what I wanted. Apologies that I didn't realize that there is already an option available, so thanks again for pointing it out to me :-)

Clear case of RTFM ;-)

Closing now.

cyrille-leclerc commented 1 year ago

I'm so sorry for the wasted time @stmlange . I understood from your first message that you wanted to only skip the HTTP traces for verbose requests with little meaning like logs rendering. I didn't understand you were interesting is disabling all logs.

stmlange commented 1 year ago

All good, thank you so much for all your help! Initially I didn't realize to what extend the Jaeger OTEL Plugin creates traces. So users of the jaeger instance I happen to maintain just mentioned the trace of /job/:jobFullName/:runNumber/logText/progressiveHtml to me. Only after further investigation it turned out that each and every web request creates a trace. I think your responses had been on point and indeed just to filter out a single trace the filtertracer would be the right choice. However I only realized after experimenting with the filtering that we want a different option (which you also provided to me).

So thanks again for the plugin and extremely fast responses in pointing me to the right options/tools :-)

cyrille-leclerc commented 1 year ago

Thanks @stmlange and please don't forget to share feedback with us so we can improve Jenkins observability. We are maintaining this plugin outside of our day job so we have little bandwidth to collect user feedback but we are eager to continue to improve it.

jenkinsci / opentelemetry-plugin