jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
100 stars 53 forks source link

JCasC is not loaded correctly #453

Closed kuisathaverat closed 2 years ago

kuisathaverat commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.332.3 OS: Linux - 5.10.104-linuxkit --- ace-editor:1.1 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 blueocean:1.25.5 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.5 blueocean-commons:1.25.5 blueocean-config:1.25.5 blueocean-core-js:1.25.5 blueocean-dashboard:1.25.5 blueocean-display-url:2.4.1 blueocean-events:1.25.5 blueocean-git-pipeline:1.25.5 blueocean-github-pipeline:1.25.5 blueocean-i18n:1.25.5 blueocean-jwt:1.25.5 blueocean-personalization:1.25.5 blueocean-pipeline-api-impl:1.25.5 blueocean-pipeline-editor:1.25.5 blueocean-pipeline-scm-api:1.25.5 blueocean-rest:1.25.5 blueocean-rest-impl:1.25.5 blueocean-web:1.25.5 bootstrap4-api:4.6.0-5 bootstrap5-api:5.1.3-7 bouncycastle-api:2.25 branch-api:2.1046.v0ca_37783ecc5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.4 cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_ cloudbees-folder:6.722.v8165b_a_cf25e9 command-launcher:1.2 configuration-as-code:1429.v09b_044a_c93de credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.5 docker-commons:1.19 docker-workflow:1.28 durable-task:496.va67c6f9eefa7 echarts-api:5.3.2-2 favorite:2.4.1 filesystem_scm:2.1 font-awesome-api:6.1.1-1 git:4.11.3 git-client:3.11.0 git-server:1.11 github:1.34.3 github-api:1.303-400.v35c2d8258028 github-branch-source:1637.vd833b_7ca_7654 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 jackson2-api:2.13.3-285.vc03c0256d517 javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-6 jaxb:2.3.6-1 jdk-tool:1.5 jenkins-design-language:1.25.5 jjwt-api:0.11.2-71.v2722b_b_06a_2a_f job-dsl:1.79 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1.59 mailer:414.vcc4c33714601 matrix-project:771.v574584b_39e60 okhttp-api:4.9.3-105.vb96869f8ac3a opentelemetry:2.7.1-rc3-rc698.c8ebe060f313 pipeline-build-step:2.18 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:591.v3a_7f422b_d058 pipeline-input-step:448.v37cea_9a_10a_70 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2064.v5eef7d0982b_e pipeline-model-definition:2.2064.v5eef7d0982b_e pipeline-model-extensions:2.2064.v5eef7d0982b_e pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2064.v5eef7d0982b_e plain-credentials:1.8 plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.5-2 pubsub-light:1.16 scm-api:608.vfa_f971c5a_a_e9 script-security:1145.1148.vf6d17a_a_a_eef6 snakeyaml-api:1.30.1 sse-gateway:1.25 ssh-credentials:277.v95c2fec1c047 sshd:3.237.v883d165a_c1d3 structs:318.va_f3ccb_729b_71 swarm:3.32 token-macro:293.v283932a_0a_b_49 trilead-api:1.57.v6e90e07157e1 variant:1.4 workflow-aggregator:581.v0c46fa_697ffd workflow-api:1153.vb_912c0e47fb_a_ workflow-basic-steps:941.vdfe1b_a_132c64 workflow-cps:2660.vb_c0412dc4e6d workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1139.v252a_e12e8463 workflow-job:1174.1176.va_29023983d67 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:818.v4eb_969241b_c7 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

tested on Docker compose and k8s

Reproduction steps

  1. Checkout the PR https://github.com/jenkinsci/opentelemetry-plugin/pull/452
  2. Edit the file demo/config/plugins.txt changing opentelemetry by opentelemetry::https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/opentelemetry/2.7.1-rc3-rc698.c8ebe060f313/opentelemetry-2.7.1-rc3-rc698.c8ebe060f313.hpi
  3. Execute make -C demos build
  4. Execute make -C demos start-all
  5. build one of the job and check the logs an exception is show
  6. Go to Manage Jenkins and save the system configuration without changes
  7. Go to the job logs, there is no exception

Expected Results

A user should see the logs of a job without having to save the configuration manually.

Actual Results

The user has to save manually the configuration without changes to allow see the logs.

Anything else?

No response

cyrille-leclerc commented 2 years ago
jenkins.yml with OTel initialization log categories enabled ```yaml jenkins: systemMessage: "Configured automatically with JCasC plugin" crumbIssuer: standard: excludeClientIPFromCrumb: false labelString: flyweight log: recorders: - loggers: - level: "FINER" name: "io.jenkins.plugins.opentelemetry.job.log.OverallLog" - level: "FINER" name: "io.jenkins.plugins.opentelemetry.backend.elastic" - level: "FINER" name: "io.jenkins.plugins.opentelemetry.job.log.util" - level: "FINE" name: "io.jenkins.plugins.opentelemetry" - level: "FINE" name: "io.jenkins.plugins.opentelemetry.init" - level: "ALL" name: "io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider" name: "Otel" mode: EXCLUSIVE numExecutors: 5 remotingSecurity: enabled: false scmCheckoutRetryCount: 2 nodes: - permanent: labelString: "local" launcher: jnlp: workDirSettings: disabled: true failIfWorkDirIsMissing: false internalDir: "remoting" workDirPath: "/tmp" name: "local" numExecutors: 2 remoteFS: "/tmp" retentionStrategy: "always" credentials: system: domainCredentials: - credentials: - usernamePassword: description: "elasticsearch-logs" id: "elasticsearch-logs" password: "changeme" scope: GLOBAL username: "admin" unclassified: openTelemetry: authentication: "noAuthentication" endpoint: "http://otel-collector:4317" exportOtelConfigurationAsEnvironmentVariables: true # Note that OpenTelemetry credentials, if configured, will be exposed as environment variables observabilityBackends: - elastic: displayKibanaDashboardLink: true elasticLogsBackend: elasticLogsBackendWithJenkinsVisualization: elasticsearchCredentialsId: "elasticsearch-logs" elasticsearchUrl: "http://elasticsearch:9200" kibanaBaseUrl: "http://localhost:5601" name: "Elastic Observability" jobs: - file: "/var/jenkins_home/config/ansible.dsl" - file: "/var/jenkins_home/config/gradle.dsl" - file: "/var/jenkins_home/config/maven.dsl" - file: "/var/jenkins_home/config/otel-cli.dsl" - file: "/var/jenkins_home/config/otel-cli-make.dsl" ```

io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration#initializeOpenTelemetry is invoked by JCasC when io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration#configure doesn't seem to be invoked

Jun 01, 2022 9:20:40 PM FINE io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration
setEndpoint(http://otel-collector:4317)
Jun 01, 2022 9:20:47 PM FINE io.jenkins.plugins.opentelemetry.backend.ObservabilityBackend
Configure OpenTelemetry SDK to export logs
Jun 01, 2022 9:20:47 PM FINE io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration
Initialize Jenkins OpenTelemetry Plugin...
Jun 01, 2022 9:20:52 PM INFO io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider initializeOtlp
OpenTelemetry SDK initialized: SDK [config: otel.traces.exporter=otlp, otel.metrics.exporter=otlp, otel.logs.exporter=otlp, otel.exporter.otlp.endpoint=http://otel-collector:4317, resource: service.name=jenkins, service.namespace=jenkins, service.version=2.332.3]
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider
Initialize Otel SDK on components: io.jenkins.plugins.opentelemetry.computer.MonitoringCloudListener, io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener, io.jenkins.plugins.opentelemetry.init.GitHubClientMonitoring, io.jenkins.plugins.opentelemetry.init.JvmMonitoringInitializer, io.jenkins.plugins.opentelemetry.init.SCMEventMonitoringInitializer, io.jenkins.plugins.opentelemetry.init.ServletFilterInitializer, io.jenkins.plugins.opentelemetry.job.MonitoringBuildStepListener, io.jenkins.plugins.opentelemetry.job.MonitoringPipelineListener, io.jenkins.plugins.opentelemetry.job.MonitoringRunListener, io.jenkins.plugins.opentelemetry.job.OtelTraceService, io.jenkins.plugins.opentelemetry.job.log.OtelLogStorageFactory, io.jenkins.plugins.opentelemetry.queue.MonitoringQueueListener, io.jenkins.plugins.opentelemetry.security.AuditingSecurityListener
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.computer.MonitoringCloudListener
Start monitoring Jenkins cloud agent provisioning...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener
Resources for Jenkins Controller computer hudson.model.Hudson$MasterComputer@48c43f0: OpenTelemetryAttributesAction{attributes=host.ip-STRING - 172.18.0.6, jenkins.computer.name-STRING - #controller#, host.name-STRING - 610187df5209}
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener
Start monitoring Jenkins agents management...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.init.GitHubClientMonitoring
Start monitoring Jenkins GitHub client...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.init.JvmMonitoringInitializer
Start monitoring Jenkins JVM...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.init.SCMEventMonitoringInitializer
Start monitoring Jenkins SCM events...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.init.ServletFilterInitializer
Jenkins Web instrumentation enabled
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.job.MonitoringRunListener
Start monitoring Jenkins build executions...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.queue.MonitoringQueueListener
Start monitoring Jenkins queue...
Jun 01, 2022 9:20:52 PM FINE io.jenkins.plugins.opentelemetry.security.AuditingSecurityListener
Start monitoring Jenkins authentication events...
Jun 01, 2022 9:21:43 PM FINE io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener
preOnline(hudson.slaves.SlaveComputer@1c4b76b2): OpenTelemetryAttributesAction{attributes=host.ip-STRING - 192.168.1.46, jenkins.computer.name-STRING - local, host.name-STRING - MacBook-Pro.localdomain}
...
cyrille-leclerc commented 2 years ago

FYI https://github.com/jenkinsci/opentelemetry-plugin/pull/454 may be the solution. Not causing regressions, need to test with https://github.com/jenkinsci/opentelemetry-plugin/tree/opentelemetry-2.7.1-rc2/demos