search

opendevstack / ods-jenkins-shared-library

Shared Jenkins library which all ODS projects & components use - provisioning, SonarQube code scanning, Nexus publishing, OpenShift template based deployments and repository orchestration
Apache License 2.0
75 stars 57 forks source link

RM: Failures in test component (NOT failed tests) in QA/Prod do not generate DIL but make the build stop / fail #830

Open albertpuente opened 2 years ago

albertpuente commented 2 years ago

Describe the bug If a test component fails in production the pipeline fails and no DIL is generated. Reported with @clemensutschig

https://github.com/opendevstack/ods-jenkins-shared-library/blob/f2470541f2e9cd8d247f5924ce0f5246b2069644/src/org/ods/orchestration/util/MROPipelineUtil.groovy#L399 does not set failfast to false - which means if an error occurs it will be thrown immediately .. (ref: https://github.com/opendevstack/ods-jenkins-shared-library/blob/master/src/org/ods/orchestration/util/MROPipelineUtil.groovy#L127-L134)

To Reproduce Do a full deployment to WIP -> D -> QA -> PROD where a test component fails either in QA or PROD. For instance, if sonarqube within the test case fails in PROD, the pipeline fails and no DIL is generated.

Expected behavior The pipeline should create and upload documents such as DIL.

Screenshots If applicable, add screenshots to help explain your problem.

Affected version (please complete the following information):

Log Output (ensure to remove any confidential information like tokens, project names, etc.

Failing SQ in component

14:46:13.500 ERROR: Error during SonarQube Scanner execution
java.lang.IllegalStateException: No files nor directories matching 'build/classes'
    at org.sonar.java.AbstractJavaClasspath.getFilesFromProperty(AbstractJavaClasspath.java:92)
    at org.sonar.java.JavaClasspath.init(JavaClasspath.java:54)
    at org.sonar.java.AbstractJavaClasspath.getElements(AbstractJavaClasspath.java:280)
....
[Pipeline] }
[Pipeline] // withSonarQubeEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] ech)[SonarQube Analysis-spock] **** ENDED stage 'SonarQube Analysis' for component 'spock' branch 'master' **** (took 19023 ms)
[Pipeline] }
[Pipeline] // wrap
[Pipeline] stag)
[Pipeline] { (odsPipeline error)
[Pipeline] ech)WARN: [spock] ***** Finished ODS Pipeline for spock (with error) ***** (took 96705 ms)
[Pipeline] ech)WARN: Error: hudson.AbortException: script returned exit code 1
[Pipeline] ech)DEBUG: [buildstatus-464fb7ca-FAILED] Setting Bitbucket build status to 'FAILED' on commit '464fb7cad4bc535b4e1a01f4fd6dda4aac60ca71' / 'https://..../job/edpst-cd/job/edpst-cd-releasemanager-master/40/'
[Pipeline] withCredential)Masking supported pattern matches of $USERNAME or $TOKEN
[Pipeline] )
[Pipeline] sh (Set bitbucket build status via API)+ curl --fail -sS --request POST --header 'Authorization: Bearer ****' --header 'Content-Type: application/json' --data '{"state":"FAILED","key":"464fb7ca","name":"464fb7ca","url":"https://..../job/edpst-cd/job/edpst-cd-releasemanager-master/40/"}' https://....s/1.0/commits/464fb7cad4bc535b4e1a01f4fd6dda4aac60ca71
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] ech)DEBUG: [buildstatus-464fb7ca-FAILED]  (took 507 ms)
[Pipeline] }
[Pipeline] // stage
[Pipeline] ech)DEBUG: Collecting (applicable) testresults from location: 'build/test-results/test'
[Pipeline] sh (Moving test results to system location: build/test-results/test)+ mkdir -p build/test-results/test build/test-results/test
+ cp -rf 'build/test-results/test/*' build/test-results/test
+ true
cp: cannot stat 'build/test-results/test/*': No such file or directory
[Pipeline] sh (Counting test results in build/test-results/test)+ ls -la 'build/test-results/test/*.xml'
+ wc -l
ls: cannot access build/test-results/test/*.xml: No such file or directory
[Pipeline] ech)DEBUG: Found 0 test files in 'build/test-results/test'
[Pipeline] ech)DEBUG: No xUnit results for stashing
[Pipeline] ech)DEBUG: [spock] ODS Component Pipeline 'spock-40'
ODS Build Artifacts 'spock': 
{
   ​"builds": {
       ​
   ​},
   ​"deployments": {
       ​
   ​},
   ​"failedStage": "odsPipeline error",
   ​"testResultsFolder": "build/test-results/test",
   ​"testResults": "0"
} (took 98246 ms)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // load
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] ech)DEBUG: Collected ODS build artifacts for repo 'spock': [builds:[:], deployments:[:], testResultsFolder:build/test-results/test, testResults:0, CREATED_BY_BUILD:Beta/40]
[Pipeline] }
[Pipeline] // dir
[Pipeline] ech)WARN: Error: aborting due to previous errors in repo 'spock'.
[Pipeline] }
Failed in branch spock
[Pipeline] // parallel
[Pipeline] }
Failed in branch Test
[Pipeline] // parallel
[Pipeline] ech)
WARN: Error occured within the orchestration pipeline: Error: aborting due to previous errors in repo 'spock'.
[Pipeline] ech)
DEBUG: [jira-update-release-EDPST-132]  (took 1858 ms)
[Pipeline] ech)
[Test] **** ENDED orchestration stage **** (took 105397 ms)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] ech)
[ods-mro-pipeline] **** ENDED orchestration pipeline **** (took 223616 ms)
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Error: aborting due to previous errors in repo 'spock'.
    at org.ods.orchestration.util.MROPipelineUtil.executeODSComponent(MROPipelineUtil.groovy:122)
    at ___cps.transform___(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
    at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
    at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.constructorCall(DefaultInvoker.java:25)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:97)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
    at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:107)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
    at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
    at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Finished: FAILURE

Additional context n.a

metmajer commented 2 years ago

@albertpuente The EDP only creates entries in the DIL for failing test cases in Q/P, not for random errors.

clemensutschig commented 2 years ago

@metmajer - are you folks sure (and this was tested e2e? :D)

If you take a look into the spock quickstarter . if gradle fails: https://github.com/opendevstack/ods-quickstarters/blob/master/e2e-spock-geb/Jenkinsfile.template#L33 we use 'error' - and that will fail downstream because of the above . From a compliance pow: ... this should result in an installation discrepency :))

metmajer commented 2 years ago

@clemensutschig discrepancies are created from Jira Bugs assigned to the currently deployed version. Jira Bugs are created by EDP automatically, but only for failing tests. In case of a random pipeline failure, docs will not be generated. Remember?

clemensutschig commented 2 years ago

@metmajer I do remember - but a "generic" failure in a "test" component - should create a bug & discrepency ... Not super urgent - but for the backlog

We should create a bug in this case - and create a discrepency from it .. (to really fit into the flow)