search

jenkinsci / datadog-plugin

A Jenkins plugin used to forward metrics, events, and service checks to an account at Datadog, automatically.
https://plugins.jenkins.io/datadog/
MIT License
30 stars 49 forks source link

StackOverflowError after enabling log collection #389

Open m0n5t3r opened 5 months ago

m0n5t3r commented 5 months ago

We've been getting stack overflows in a pipeline that uses stashedFile / the file parameter plugin; it appears to be the same issue as https://github.com/jenkinsci/file-parameters-plugin/issues/182 (where they say the problem is in this plugin)

To Reproduce Steps to reproduce the behavior:

  1. Enable datadog log collection in Manage Jenkins -> System
  2. Use something along the lines of stashedFile name: 'aFile', description: 'a file' as a parameter in a pipeline
  3. Try to run it
  4. The job fails with no console output, and the jenkins log shows a really long trace that starts like this, and just repeats the same list forever, bouncing between the file parameter plugin and the datadog plugin:
trace

```syslog Feb 01 00:53:27 jenkins jenkins[507501]: java.lang.StackOverflowError Feb 01 00:53:27 jenkins jenkins[507501]: at org.codehaus.groovy.reflection.ParameterTypes.coerceArgumentsToClasses(ParameterTypes.java:145) Feb 01 00:53:27 jenkins jenkins[507501]: at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) Feb 01 00:53:27 jenkins jenkins[507501]: at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) Feb 01 00:53:27 jenkins jenkins[507501]: at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) Feb 01 00:53:27 jenkins jenkins[507501]: at groovy.lang.Closure.call(Closure.java:420) Feb 01 00:53:27 jenkins jenkins[507501]: at groovy.lang.Closure$WritableClosure.writeTo(Closure.java:860) Feb 01 00:53:27 jenkins jenkins[507501]: at groovy.lang.Closure$WritableClosure.toString(Closure.java:986) Feb 01 00:53:27 jenkins jenkins[507501]: at io.jenkins.plugins.opentelemetry.backend.ObservabilityBackend.getTraceVisualisationUrl(ObservabilityBackend.java:114) Feb 01 00:53:27 jenkins jenkins[507501]: at io.jenkins.plugins.opentelemetry.job.MonitoringAction.lambda$getLinks$6(MonitoringAction.java:155) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) Feb 01 00:53:27 jenkins jenkins[507501]: at io.jenkins.plugins.opentelemetry.job.MonitoringAction.getLinks(MonitoringAction.java:158) Feb 01 00:53:27 jenkins jenkins[507501]: at io.jenkins.plugins.opentelemetry.job.OtelEnvironmentContributorService.addEnvironmentVariables(OtelEnvironmentContributorService.java:58) Feb 01 00:53:27 jenkins jenkins[507501]: at io.jenkins.plugins.opentelemetry.job.OtelEnvironmentContributor.buildEnvironmentFor(OtelEnvironmentContributor.java:31) Feb 01 00:53:27 jenkins jenkins[507501]: at hudson.model.Run.getEnvironment(Run.java:2430) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:519) Feb 01 00:53:27 jenkins jenkins[507501]: at org.datadog.jenkins.plugins.datadog.model.BuildData.(BuildData.java:150) Feb 01 00:53:27 jenkins jenkins[507501]: at org.datadog.jenkins.plugins.datadog.logs.DatadogTaskListenerDecorator.(DatadogTaskListenerDecorator.java:49) Feb 01 00:53:27 jenkins jenkins[507501]: at org.datadog.jenkins.plugins.datadog.logs.DatadogTaskListenerDecorator$Factory.of(DatadogTaskListenerDecorator.java:82) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator.lambda$apply$3(TaskListenerDecorator.java:164) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) Feb 01 00:53:27 jenkins jenkins[507501]: at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator.apply(TaskListenerDecorator.java:166) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:236) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:1024) Feb 01 00:53:27 jenkins jenkins[507501]: at io.jenkins.plugins.file_parameters.StashedFileParameterValue.buildEnvironment(StashedFileParameterValue.java:70) Feb 01 00:53:27 jenkins jenkins[507501]: at hudson.model.ParametersAction.buildEnvironment(ParametersAction.java:143) Feb 01 00:53:27 jenkins jenkins[507501]: at hudson.model.Run.getEnvironment(Run.java:2434) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:519) Feb 01 00:53:27 jenkins jenkins[507501]: at org.datadog.jenkins.plugins.datadog.model.BuildData.(BuildData.java:150) Feb 01 00:53:27 jenkins jenkins[507501]: at org.datadog.jenkins.plugins.datadog.logs.DatadogTaskListenerDecorator.(DatadogTaskListenerDecorator.java:49) Feb 01 00:53:27 jenkins jenkins[507501]: at org.datadog.jenkins.plugins.datadog.logs.DatadogTaskListenerDecorator$Factory.of(DatadogTaskListenerDecorator.java:82) Feb 01 00:53:27 jenkins jenkins[507501]: at org.jenkinsci.plugins.workflow.log.TaskListenerDecorator.lambda$apply$3(TaskListenerDecorator.java:164) [...] ```

Expected behavior Job runs

Environment and Versions (please complete the following information): openjdk 17.0.9 2023-10-17 Jenkins 2.443 Datadog plugin 6.0.0 Datadog agent 6.50.3 File parameter plugin 316.va_83a_1221db_a_7

lemeurherve commented 5 months ago

We just had the same issue on ci.jenkins.io instance yesterday and today, here is the corresponding stack trace: https://gist.github.com/lemeurherve/eaf08053a133733a829c24ca0eec7117

A reboot of our instance fixed the stackoverflow error, we then updated Datadog version from 6.0.0 to 6.0.1

See also https://github.com/jenkins-infra/helpdesk/issues/3934#issuecomment-1928944474

nikita-tkachenko-datadog commented 5 months ago

Plugin v6.0.2 has been released. @lemeurherve, it should fix the issue that you're experiencing. It does not fix the originally reported issue with stashedFile, however, since the two are slightly different.

lemeurherve commented 5 months ago

Updating ci.jenkins.io to 6.0.2 already fixed another issue, cf https://github.com/jenkins-infra/helpdesk/issues/3944#issuecomment-1934793244 👏

I'll keep you updated if we encounter new stackoverflow errors or not in the following days.