search

jenkinsci / generic-webhook-trigger-plugin

Can receive any HTTP request, extract any values from JSON or XML and trigger a job with those values available as variables. Works with GitHub, GitLab, Bitbucket, Jira and many more.
https://plugins.jenkins.io/generic-webhook-trigger
404 stars 159 forks source link

Large webhook, Argument list too long (MAX_ARG_STRLEN) #219

Closed facetheheat closed 1 year ago

facetheheat commented 2 years ago

Hi there, I've got a problem with large webhook from Jira. Is it possible to save it to file?

Version report

Jenkins and plugins versions report: 1.76

Reproduction steps

Scripted pipeline

    properties([
        pipelineTriggers([
            [$class: 'GenericTrigger',
                genericVariables: [
                    [key: 'var_webhook', value: '$'],
                ],
                tokenCredentialId: 'webhook-jira-production',
                printContributedVariables: true,
                printPostContent: true,
                silentResponse: false
            ]
        ])
    ])

Results

Caused by: java.io.IOException: Cannot run program "git" (in directory "/storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira"): error=7, Argument list too long

Expected result: Build is OK and I can save webhook data to file for processing in steps

Actual result: Build failed.

facetheheat commented 2 years ago 
22:09:55  
22:09:55  
22:09:55 Obtained Jenkins/pipelines/im_Webhook_Jira.groovy from git git@gitlab/myrepo.git
22:09:55 Resume disabled by user, switching to high-performance, low-durability mode.
22:09:55 [Pipeline] Start of Pipeline
22:09:55 [Pipeline] node
22:09:55 Running on Jenkins in /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira
22:09:55 [Pipeline] {
22:09:55 [Pipeline] stage
22:09:55 [Pipeline] { (Init)
22:09:56 [Pipeline] echo
22:09:56 [info] checkout scm
22:09:56 [Pipeline] checkout
22:09:56 The recommended git tool is: git
22:09:56 using credential jenkins-ssh-keyfile
22:09:56 Cloning the remote Git repository
22:09:56 Cloning repository git@gitlab/myrepo.git

22:09:56 > git init /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira # timeout=10
22:09:56 ERROR: Error cloning remote repo 'origin'
22:09:56 hudson.plugins.git.GitException: Could not init /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1048)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:803)
22:09:56 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1227)
22:09:56 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
22:09:56 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
22:09:56 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
22:09:56 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
22:09:56 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
22:09:56 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
22:09:56 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
22:09:56 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
22:09:56 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
22:09:56 at java.base/java.lang.Thread.run(Thread.java:829)
22:09:56 Caused by: hudson.plugins.git.GitException: Error performing git command: git init /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2689)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2611)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2607)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1974)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1046)
22:09:56 ... 12 more
22:09:56 Caused by: java.io.IOException: Cannot run program "git" (in directory "/storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira"): error=7, Argument list too long
22:09:56 at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
22:09:56 at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
22:09:56 at hudson.Proc$LocalProc.<init>(Proc.java:253)
22:09:56 at hudson.Proc$LocalProc.<init>(Proc.java:222)
22:09:56 at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
22:09:56 at hudson.Launcher$ProcStarter.start(Launcher.java:508)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2674)
22:09:56 ... 16 more
22:09:56 Caused by: java.io.IOException: error=7, Argument list too long
22:09:56 at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
22:09:56 at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
22:09:56 at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
22:09:56 at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
22:09:56 ... 22 more
tomasbjerre commented 2 years ago

Are you using some value from the webhook with the git plugin? What value? What does the json look like? Can you not have a more specific jsonpath?

You can save var_webhook to a file as part of your build. Just like you would save any variable to a file. You can google that.

facetheheat commented 2 years ago

Yes, I do. But build failed before that I can process it. I found same issues and thats is jenkins/node issue:

tomasbjerre commented 2 years ago

Looks unrelated to this plugin.

I would resolve the values I need with this plugin. And have a pipeline where I do git clone. I would not use the git plugin.

facetheheat commented 2 years ago

There is an example json from jira https://pastebin.com/gvphwwdq

facetheheat commented 2 years ago

And I've remove any other plugins from job, make freejob with single build sh step. The issue reproduces

[JIRA_WEBHOOK_NO_GIT] $ /bin/sh -xe /tmp/jenkins17235265673168013619.sh
FATAL: command execution failed
java.io.IOException: error=7, Argument list too long
    at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
    at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
    at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
Caused: java.io.IOException: Cannot run program "/bin/sh" (in directory "/storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/JIRA_WEBHOOK_NO_GIT"): error=7, Argument list too long
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
    at hudson.Proc$LocalProc.<init>(Proc.java:253)
    at hudson.Proc$LocalProc.<init>(Proc.java:222)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
    at hudson.Launcher$ProcStarter.start(Launcher.java:508)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:21)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:808)
    at hudson.model.Build$BuildExecution.build(Build.java:199)
    at hudson.model.Build$BuildExecution.doRun(Build.java:164)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:516)
    at hudson.model.Run.execute(Run.java:1889)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:100)
    at hudson.model.Executor.run(Executor.java:433)
Build step 'Execute shell' marked build as failure
Finished: FAILURE
tomasbjerre commented 2 years ago

What do you do in that sh step?

facetheheat commented 2 years ago

I do it with just single "echo" as stub(without access to genericVariable). Webhook's payload printed in the build output and than build failed on sh stage.

electricalbah commented 2 years ago

Any update on this? @facetheheat did you get a workaround?

@tomasbjerre I am getting the same error "java.io.IOException: error=7, Argument list too long", if the JSON payload exceeds the maximum size allowed for an environment variable.

Is there a way to access the HTTP request object and extract the payload? or convert the payload(post content) to file instead of setting it as ENV VAR?

electricalbah commented 2 years ago

I do it with just single "echo" as stub(without access to genericVariable). Webhook's payload printed in the build output and than build failed on sh stage.

same issue over here

facetheheat commented 2 years ago

@electricalbah Yep. The workaround is the stop using this plugin :-)

electricalbah commented 2 years ago

@tomasbjerre It is very easy to reproduce just send a JSON payload over 2MB and try to access the variable, it will crash.

tomasbjerre commented 2 years ago

If I configure a pipeline like:

node {
 properties([
  pipelineTriggers([
   [$class: 'GenericTrigger',
    genericVariables: [
     [
      key: 'all',
      value: '$'
     ]
    ],

    causeString: 'Triggered on $ref',

    tokenCredentialId: 'theid',

    printContributedVariables: true,
    printPostContent: true,

    silentResponse: false
   ]
  ])
 ])

 stage("build") {
  sh '''
  echo hello
  '''
 }
}

And trigger it with the given in a previous comment.

curl -v -H "Content-Type: application/json" http://localhost:8080/jenkins/generic-webhook-trigger/invoke?token=abc123 --data @/home/bjerre/Hämtningar/gvphwwdq.txt

I can see that the code reaches: https://github.com/jenkinsci/generic-webhook-trigger-plugin/blob/70b68f6de359b1add3f7b8264eb967a9d13f1043/src/main/java/org/jenkinsci/plugins/gwt/GenericTrigger.java#L181

And the job is also triggered, but fails with:

java.io.IOException: error=7, Argumentlistan för lång
    at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
    at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
    at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
    at hudson.Proc$LocalProc.<init>(Proc.java:252)
    at hudson.Proc$LocalProc.<init>(Proc.java:221)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
    at hudson.Launcher$ProcStarter.start(Launcher.java:508)
    at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
Caused: java.io.IOException: Cannot run program "nohup" (in directory "/home/bjerre/workspace/generic-webhook-trigger-plugin/work/workspace/pipe"): error=7, Argumentlistan för lång
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
    at hudson.Proc$LocalProc.<init>(Proc.java:252)
    at hudson.Proc$LocalProc.<init>(Proc.java:221)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
    at hudson.Launcher$ProcStarter.start(Launcher.java:508)
    at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
    at WorkflowScript.run(WorkflowScript:25)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

But if I change the pipeline so that it only resolves one variable, like this:

node {
 properties([
  pipelineTriggers([
   [$class: 'GenericTrigger',
    genericVariables: [
     [
      key: 'all',
      value: '$.timestamp'
     ]
    ],

    causeString: 'Triggered on $ref',

    tokenCredentialId: 'theid',

    printContributedVariables: true,
    printPostContent: true,

    silentResponse: false
   ]
  ])
 ])

 stage("build") {
  sh '''
  echo hello
  '''
 }
}

The job succeeds.

I don't see this as a bug in the plugin, I see it as a bug in Jenkins. There seems to be an upper limit on number of parameters.

It is a common error, can be googled: https://www.google.com/search?q=%22error%3D7%2C+Argumentlist

Conclusion

Only solution, for now, is to only resolve the values you actually need. Use more specific JSONPath:s.