Question - MPLPostStep fails

RamchandV commented 4 years ago

Below is the POST checkout step i modified , if this runs all i see is the below error

MPLPostStep('always') {
  echo "Checkout in Progress"
}

MPLModule('Default Checkout', CFG)

java.util.NoSuchElementException: Cannot access last() element from an empty List

i even added some displays this is what i see , body comes in null is that expected ?

inside Post Step : always
[Pipeline] echo
Checkout in Progress
[Pipeline] echo (hide)
body : null

RamchandV commented 4 years ago

the pipeline works fine if i remove the MPLPostStep

sparshev commented 4 years ago

Hi @RamchandV ,

Could you please describe the steps to reproduce, because right now it's hard to get what did you changed in the pipeline (if you changed anything).

In the default MPL modules we using MPLPostStep during Deploy/OpenshiftDeploy.groovy module execution - and it's tested in our CI.

Thank you

RamchandV commented 4 years ago

That's the weird part i did not change anything and it still fails. I'm thinking to use your your new merge as is and see whats happening. Is there any examples other than what you have here ?

sparshev commented 4 years ago

@RamchandV ,

Unfortunately release 18.06.1 is not maintained - we changing the release process and it's better to use master - it contains all the required changes for the latest LTS jenkins. We working on the next release 1.0.0 right now in master branch and it's stable. If the latest master fails unmodified - that's for sure something wrong and we need to check that. So please check master - hopefully it will work for you.

Right now what we have the next examples:

MPL itself example - it's enforced by the CircleCI which is executing build of MPL and build of MPL in jenkins using MPL: https://github.com/griddynamics/mpl/blob/master/.circleci/config.yml
Wiki page with a number of examples: https://github.com/griddynamics/mpl/wiki - you can just clone it and see the examples folder.

Thank you

RamchandV commented 4 years ago

I did not change anything , running your code as is , this is the error i get

java.lang.StackOverflowError: Excessively nested closures/functions at com.griddynamics.devops.mpl.MPLManager.getPostStepsErrors(MPLManager.groovy:217) - look for unbounded recursion - call depth: 1025
    at com.cloudbees.groovy.cps.impl.CpsFunction.invoke(CpsFunction.java:28)
    at com.cloudbees.groovy.cps.impl.CpsCallableInvocation.invoke(CpsCallableInvocation.java:57)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:98)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78)
    at jdk.internal.reflect.GeneratedMethodAccessor426.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:405)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:317)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:281)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Finished: FAILURE

JenkinsFile Config

@Library('mpl@test') _

MPLPipeline {
  agent_label = 'linux'
  modules.Checkout = null
  modules.Build = null
  //modules.Deploy = null
  modules.Test = null
}

sparshev commented 4 years ago

@RamchandV , I asked you before to describe the steps to reproduce - probably you missed the important things:

What the jenkins version you're using?
What the plugins you're using?
Why it's @Library('mpl@test') instead of @Library('mpl@master')?

First of all - try to build the MPL itself, how it's built by CircleCI, than you can go to modifications (use some test branch etc.).

RamchandV commented 4 years ago

@sparshev Sorry below is a detailed description. One more thing i noticed when you see my first description the body the call gets is null is that expected?

def call(String name, Closure body) {
  MPLManager.instance.postStep(name, body)
}

Configuration Jenkins - 2.204.4 Pipeline: Groovy - 2.78 ( workflow-cps) Pipeline: Job - 2.34 (workflow-job) Pipeline: Shared Groovy Libraries - 2.15 (workflow-cps-global-lib )

my branch test is your current master and i'm using it unchanged . changes made only in the JenkinsFile to enable deploy stage.

How i enabled the project

Downloaded your master
made the downloaded code as the test branch in my repo
updated the jenkinsfile to only run the deploy stage
It errored out when i can the job with the below error.

OpenShift Deploy Decomission poststep
Error when executing always post condition:
java.util.NoSuchElementException: Cannot access first() element from an empty List
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.first(DefaultGroovyMethods.java:8982)
    at org.codehaus.groovy.runtime.dgm$223.doMethodInvoke(Unknown Source)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
    at com.griddynamics.devops.mpl.MPLManager.getPostStepsErrors(MPLManager.groovy:218)
    at com.griddynamics.devops.mpl.MPLManager.getPostStepsErrors(MPLManager.groovy:221)
    at MPLPostStepsRun.call(MPLPostStepsRun.groovy:35)
    at MPLPipeline.call(MPLPipeline.groovy:76)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:140)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:758)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:398)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:396)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:757)
    at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030)
    at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015)
    at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:747)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executePostBuild(ModelInterpreter.groovy:725)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78)
    at jdk.internal.reflect.GeneratedMethodAccessor227.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:405)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:317)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:281)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

sparshev commented 4 years ago

Ok, @RamchandV - I just tested the latest MPL master in all possible ways with the LTS jenkins and quite fresh plugins:

Jenkins - 2.204.2
Pipeline: Groovy - 2.80 ( workflow-cps)
Pipeline: Job - 2.36 (workflow-job)
Pipeline: Shared Groovy Libraries - 2.15 (workflow-cps-global-lib )

And it's working well - both with default and your way modified Jenkinsfile (although I used master instead of linux): tmp

About the exception you see: MPLPostStepsRun executing during post - and providing getPostStepsErrors a name for sure (at least in line #76 in the default MPLPipeline which we are both using). So it just can't go into MPLManager.groovy#218 - because there is if that checks the name is null or not. In your case it's null for some reason - so please double check that your MPLPipeline on line 76 provides 'always' name into the MPLPostStepsRun call.

And again - please try to build the MPL itself (with no modifications anywhere) to check your jenkins is ok. But most probably you have a wrongly modified version of some MPL logic - that's why you see this issue.

One more thing i noticed when you see my first description the body the call gets is null is that expected?

What do you mean? It's getting 2 values: 'always' name and {...} - body closure

RamchandV commented 4 years ago

I had to make the below change to make it work , it did not make sense for me , let me know if you think otherwise

Just added the decorator this is in MPLManager.groovy

  @NonCPS
  public List getPostStepsErrors(String name = null) {
    if( name == null ) {
      def block = Helper.getMPLBlocks().first()
      name = "${block.module}(${block.id})"
    }
    postStepsErrors[name] ?: []
  }

sparshev commented 4 years ago

@RamchandV , did you made the default mpl build work on your jenkins? It's working fine? Could I close this question ticket?

Regarding the change - there is no need in @NonCPS, because in the function MPLManager.getPostStepsErrors no persistent variables that could not be serialized. Otherwise you will see the serialization exception - please post it as the new issue if you see some.

Thank you

RamchandV commented 4 years ago

I was not able to make the default pipeline run , only after i made this changes it started to work for me.

sparshev commented 4 years ago

@RamchandV , you mean "you saw the exception ends up with java.util.NoSuchElementException: Cannot access first() element from an empty List and when you set @NonCPS for MPLManager.getPostStepsErrors - it was solved"?

If you trying to make this statement - it for sure makes no sense for me right now (I already described what kind of issue could cause such exception). Probably I missing something, so:

If you can setup a clean jenkins, test the default mpl build and reproduce the bug - that will be great.
Or if you can provide some clear complete information about the environment, states of your current mpl/jenkins before and after the fix you made for me to reproduce the bug - that will also work.

But for now - it looks for me like not reproducible issue on your side. I can't reproduce the issue and confirm your statement about @NonCPS.

RamchandV commented 4 years ago

yes adding @NonCPS fixed mine , i was also thinking jenkins ENV is different. Let me see if i can setup a clean Jenkins and test this

Cheava commented 4 years ago

@sparshev @NonCPS is also work for me. My nested library demo is based on mpl.wiki's repo. My NestedPipeline.groovy is edited by me like below

def call(body) {
  echo "Init the MPL library"
  MPLInit()
  echo "Call the MPL library"
  MPLPipeline(body)
}

And my app project's jenkinsfile is like below

@Library('mpl-nested@master') _

NestedPipeline {
    git = [
        url:'XXXXXX',
        branch: 'master',
        credentials: "XXXXX"
    ]
    }

sparshev commented 4 years ago

@Cheava , the question right now - how to reproduce the issue.

About your logic: MPLPipeline(body) is a step, that supposed to be started from another step. Just copy the logic of MPLPipeline to your NestedPipeline - because pipeline will be changed by you anyway. Otherwise it's easier to use MPLPipeline directly.

Probably it's the way to reproduce the issue, need to check it.

sparshev commented 4 years ago

Hi @Cheava , I tried to reproduce the issue in the ticket using the same call MPLPipeline in the NestedPipeline step - but that's worked fine on the latest lts jenkins with the latest plugins...

So hopefully someone can provide some strict steps to reproduce - but right now this ticket looks like unable to reproduce for me.

amenaafreen commented 4 years ago

I have a separate question, let's say we want to give the users the flexibility of builds like suppose I have gradle builds then it is not always ./gradlew clean build right? What is te way of providing them with a generalized approach? Also like if they want to run their tests in specific environment profiles how can this be achieved by MPL library? Please let me know @sparshev Waiting to hear from you soon.

amenaafreen commented 4 years ago

@Library('mpl@release') _

// Use default master pipeline
MPLPipeline {
  // Pipeline configuration override here
  // Example: (check available options in the pipeline definition)
  agent_label = 'LS'                     // Set agent label
  modules.Build.tool_version = 'Maven 2' // Change tool for build stage
  modules.Test = null                    // Disable Test stage
}

In the above example as we are overriding the Maven version. Now similarly if we want to override the command ./gradlew clean build with some other command and provide some system parameters to gradle build, how exactly do we do that? Could you please help me out

yamsivakumarreddy commented 4 years ago

Hi, I started using the libraries, But If I want some variables from one stage to another stage How can it be done?

rolando-lorenzo commented 4 years ago

Hi All. What was the solution for hudson.remoting.ProxyException: com.griddynamics.devops.mpl.MPLModuleException: java.util.NoSuchElementException: Cannot access last() element from an empty List I got the same when I added a call MPLPostStep in my module ?

So I am using a nested library and calling official library like this:

MPLInit.groovy def call() { library('mpl@release') MPLModulesPath('com/mycompany/cjflib') }

Should I use mpl@master instead?

sparshev commented 4 years ago

@rolando-lorenzo right, master branch is the preffered one and most stable with latest features and working on the latest jenkins well.

rolando-lorenzo commented 4 years ago

@sparshev thank you :)

Napo2k commented 4 years ago

I have just been able to reproduce this issue, with the following pipeline

@Library('mpl') _  // this is master of this repository

def mpl = MPLPipelineConfig([:])

pipeline {
    agent {
        node {
            label 'ondemand-lin'
        }
    }
    options {
        skipDefaultCheckout(true)
    }
    stages {
        stage( 'Setup Poststeps' ) {
            steps {
                MPLPostStep('success') {
                    MPLModule('Test', [:])   // I just wanted to add a module
                }
            }
        }
    }
    post {
        always {
            MPLPostStepsRun('always')
        }
        success {
            MPLPostStepsRun('success')
        }
        failure {
            MPLPostStepsRun('failure')
        }
    }
}

Jenkins: 2.190.3 Plugins: workflow-cps: '2.80' workflow-cps-global-lib: '2.16' workflow-job: '2.39'

Output:

[Pipeline] {
[Pipeline] stage
[Pipeline] { (Setup Poststeps)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
java.util.NoSuchElementException: Cannot access first() element from an empty List
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.first(DefaultGroovyMethods.java:8982)
    at org.codehaus.groovy.runtime.dgm$223.doMethodInvoke(Unknown Source)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
    at com.griddynamics.devops.mpl.MPLManager.postStep(MPLManager.groovy:136)
    at MPLPostStep.call(MPLPostStep.groovy:39)
    at WorkflowScript.run(WorkflowScript:17)

sparshev commented 4 years ago

Hi @Napo2k , sure will check that tomorrow morning. Could you please describe - it's necessary to use agent ondemand-lin there? I mean you can reproduce the same just on master? And what's in the Test module?

Napo2k commented 4 years ago

The Test module is simply https://github.com/griddynamics/mpl/blob/master/resources/com/griddynamics/devops/mpl/modules/Test/Test.groovy I did this test with the master branch for the base MPL repository. I do not think ondemand-lin will make a difference, but will test on master too.

Napo2k commented 4 years ago

Exact same issue when running on the master agent

sparshev commented 4 years ago

@Napo2k I did a couple of tests with your proposed Jenkinsfile and looks like the actual issue is not related to MPLModule() inside MPLPostStep() - it's just about the MPLPostStep() that specified directly in pipeline without the MPLModule() parent - means MPLPostStep() right now is working only inside the module, but can't be used directly in pipeline.

The minimal reproducible logic is:

@Library('mpl@master') _

def mpl = MPLPipelineConfig([:])

pipeline {
    agent {
        node { label 'master' }
    }
    options {
        skipDefaultCheckout(true)
    }
    stages {
        stage( 'Setup Poststeps' ) {
            steps {
                echo 'step 1'
                MPLPostStep('success') { } // Will fail here
                echo 'step 2 - not executed'
            }
        }
    }
}

For sure this is a bug - will check how to properly fix it.

jeffoverflow commented 4 years ago

I had the same problem with an older version of Jenkins (2.235.1), and after updating Jenkins (2.235.3) the issue was resolved

griddynamics / mpl

Question - MPLPostStep fails #58