otlpHeader is null - Githubissues

ashish6913 commented 2 months ago

Jenkins and plugins versions report

Environment

```text Paste the output here ```

I'm running Jenkins, Jenkins agent, Prometheus, Grafana on Docker. I've added the required plugin ,installed the otelcol-contrib(0.70.0) & node_exporter(1.5.0) executable.

Used this doc to setup Jenkins, Prometheus and Grafana.

What Operating System are you using (both controller, and any agents involved in the problem)?

Docker

Reproduction steps

Run any Jenkins pipeline with following option: onMonit(debug: true)

Expected Results

I expect the pipeline to run successfully with data being available in Grafana to visualize.

Actual Results

[on-monit] Detected agent info: ComputerInfo{os='linux', isAmd64=true} [on-monit] Looking for node_exporter implementation... [on-monit] Delegated exec node_exporter (fixed order delegation to other process factories) [on-monit] Exec node_exporter (binary node_exporter present on remote machine) [on-monit] Looking for otel-contrib implementation... [on-monit] Delegated exec otelcol-contrib (fixed order delegation to other process factories) [on-monit] Exec otelcol-contrib (binary otelcol-contrib present on remote machine) [on-monit] node_exporter will listen on 9100 $ node_exporter --web.disable-exporter-metrics --web.listen-address=:9100 ts=2024-07-08T18:52:42.497Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" ts=2024-07-08T18:52:42.497Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" ts=2024-07-08T18:52:42.498Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) ts=2024-07-08T18:52:42.498Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ ts=2024-07-08T18:52:42.499Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ ts=2024-07-08T18:52:42.499Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:110 level=info msg="Enabled collectors" ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=arp ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=bcache ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=bonding ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=btrfs ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=conntrack ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=cpu ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=cpufreq ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=diskstats ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=dmi ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=edac ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=entropy ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=fibrechannel ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=filefd ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=filesystem ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=hwmon ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=infiniband ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=ipvs ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=loadavg ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=mdadm ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=meminfo ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=netclass ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=netdev ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=netstat ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=nfs ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=nfsd ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=nvme ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=os ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=powersupplyclass ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=pressure ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=rapl ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=schedstat ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=selinux ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=sockstat ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=softnet ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=stat ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=tapestats ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=textfile ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=thermal_zone ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=time ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=timex ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=udp_queues ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=uname ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=vmstat ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=xfs ts=2024-07-08T18:52:42.499Z caller=node_exporter.go:117 level=info collector=zfs ts=2024-07-08T18:52:42.500Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 ts=2024-07-08T18:52:42.500Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 Could not start node_exporter + otel-contrib processes node_exporter + otel-contrib stopped [Pipeline] // onMonit [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 16bdb677-8d03-4605-972d-a24fcef54359 java.lang.NullPointerException: Cannot invoke "String.indexOf(String)" because "otlpHeader" is null at io.jenkins.plugins.onmonit.ONTemplating.getJobContext(ONTemplating.java:101) at io.jenkins.plugins.onmonit.ONMonitoringStepExecution.initRemoteProcesses(ONMonitoringStepExecution.java:224) at io.jenkins.plugins.onmonit.ONMonitoringStepExecution.start(ONMonitoringStepExecution.java:74) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:323) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) at org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:105) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.recursiveWrappers(ModelInterpreter.groovy:648) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.recursiveWrappers(ModelInterpreter.groovy:647) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(ModelInterpreter.groovy:621) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(ModelInterpreter.groovy:620) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:80) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:445) at cps.transform(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85) at jdk.internal.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:136) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50) at java.base/java.lang.Thread.run(Unknown Source) Finished: FAILURE

Anything else?

No response

Are you interested in contributing a fix?

No response

christophe-kamphaus-jemmic commented 2 months ago

Currently the plugin assumes that the environment variables OTEL_EXPORTER_OTLP_ENDPOINT (eg. http://otel.example.com:4317) and OTEL_EXPORTER_OTLP_HEADERS (eg. Authorization=Bearer abcd) are available. These can be defined using either the withEnv step before onMonit or by configuring the opentelemetry-plugin with the global parameters Authentication and Export OpenTelemetry configuration as environment variables enabled.

If you'd like to improve otel-collector exporter config related to endpoint and authorization, then pull requests are welcome. References:

ashish6913 commented 2 months ago

Thanks for the response. However, I've currently installed otelcol-contrib on the jenkins-agent and so far my understanding is that i'm not supposed to have the otecol-contrib running in the agent but just have it in the path.

Also, i'm adding onMonit() in the options block like following:

options { skipStagesAfterUnstable() onMonit() }

Since, the otelcol is not supposed to be running, if i provide the url of jenkins agent to opentelemetry plugin config, it logs out that it failed to connect which would be right because its not running, the otel binary is just sitting there. Having said that, please clarify how to use onMonit() with both Opentelemetry plugin and withEnv. Thanks

jenkinsci / opentelemetry-agent-metrics-plugin

otlpHeader is null #43