search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

Testing "failing" after passing due to groovy.lang.MissingPropertyException RUNTEST_CMD #7155

Closed pshipton closed 4 years ago

pshipton commented 5 years ago

The tests are passing, but then the job has the following problem, and gets failed status. Many examples, I'm not going to include them all https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_sanity.functional_x86-64_linux_Nightly/44/ https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_extended.system_x86-64_mac_Nightly/43/ https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.system_x86-32_windows_Nightly/143/ https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_xl_OMR/267 https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_x86-32_windows_Nightly/144

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
groovy.lang.MissingPropertyException: No such property: RUNTEST_CMD for class: groovy.lang.Binding
    at groovy.lang.Binding.getVariable(Binding.java:63)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:270)
    at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:291)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:295)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:271)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:271)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
    at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
    at Script1.runTest(Script1.groovy:422)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
    at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
    at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
    at sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
pshipton commented 5 years ago

@llxia @smlambert @AdamBrousseau @jdekonin

pshipton commented 5 years ago

I also see this in some parent Build_Test jobs

Also:   groovy.lang.MissingPropertyException: No such property: GITHUB_REPO for class: groovy.lang.Binding
Also:   groovy.lang.MissingPropertyException: No such property: GITHUB_REPO for class: groovy.lang.Binding
Also:   groovy.lang.MissingPropertyException: No such property: GITHUB_REPO for class: groovy.lang.Binding
groovy.lang.MissingPropertyException: No such property: GITHUB_REPO for class: groovy.lang.Binding
    at groovy.lang.Binding.getVariable(Binding.java:63)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:270)
    at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:291)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:295)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:271)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
    at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
    at Script2.build_with_slack(Script2.groovy:333)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:74)
    at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
    at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
    at sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Finished: FAILURE
smlambert commented 5 years ago

Must be an OpenJ9 customizations, RUNTEST_CMD is not one of the parameters that we need/use in the Test script.

**yep, after posting this, see we have it, thanks for noting it below Adam.

AdamBrousseau commented 5 years ago

I think GITHUB_REPO is an unrelated issue. Not sure how it's failing but it's this function https://github.com/eclipse/openj9/blob/master/buildenv/jenkins/common/pipeline-functions.groovy#L285

RUNTEST_CMD is something on the test side https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/buildenv/jenkins/JenkinsfileBase#L386

pshipton commented 5 years ago

OpenJ9 changes from the previous nightly build https://github.com/eclipse/openj9/compare/c9d84df...b0e9913

pshipton commented 5 years ago

OpenJ9 changes between OMR acceptance builds, when the problem appeared. I don't think it's caused by any OpenJ9 changes. https://github.com/eclipse/openj9/compare/b0e9913...0449f02

llxia commented 5 years ago

The same test script passed in some builds and failed in others. It looks like a machine/jenkins issue to me. Did we have machine/jenkins update recently?

image

https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK13_x86-64_linux/109/

I am running Grinders on failed machines and will update once I have the results.

llxia commented 5 years ago

Grinder test passed on the previously failed machine: https://ci.eclipse.org/openj9/view/Test/job/Test-Grinder/516/

reran Test_openjdk13_j9_sanity.functional_x86-64_linux_Nightly on ub16x64j96: https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_sanity.functional_x86-64_linux_Nightly/45/

pshipton commented 5 years ago

jenkins seems a bit wacked, maybe we should restart it. https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_sanity.functional_x86-64_linux_Nightly/45 started running at 10am, but it's part of https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-JDK13/59/ which is done.

I also noticed some workers reported offline in a few jobs, although everything seems fine now. Perhaps it's all caused by machine/network issues that occurred overnight.

Waiting to resume part of Pipeline-Build-Test-JDK13 #59 - Nightly: ‘worker-1’ is offline
Waiting to resume part of Pipeline-Build-Test-JDK13 #59 - Nightly: ‘worker-1’ is offline
Waiting to resume part of Pipeline-Build-Test-JDK13 #59 - Nightly: ‘worker-1’ is offline
pshipton commented 4 years ago

Happened again last night, 21+ "failures", although the testing passed.

Example https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_extended.functional_x86-64_linux_xl_Nightly/72

AdamBrousseau commented 4 years ago

Still an issue?

pshipton commented 4 years ago

Still an issue?

Nope.