search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

PR Groovy pipeline scripts are slow #6274

Open fjeremic opened 5 years ago

fjeremic commented 5 years ago

I've been noticing a slowdown in the amount of time it takes to launch PR builds since the switch to pipeline generated jobs. For example a build triggered via Jenkins test extended all jdk8,jdk11 produces the build seen in [1]. Inside each "Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy" step seems to take roughly 2:30 min. for each job that is being generated.

For this particular case there are many jobs, so we end up waiting roughly one hour between requesting the tests via a PR comment and any build jobs being launched. Was wondering if something can be done to improve the performance of these pipelines so that we reach the build step faster.

[1] https://ci.eclipse.org/openj9/job/PullRequest-OpenJ9/632/console

DanHeidinga commented 5 years ago

fyi @AdamBrousseau

AdamBrousseau commented 5 years ago

When we first implemented job DSL generation it was very quick. It seems to be getting slower and slower. We have the option to temporarily disable this while we investigate (#6275). Last night's build took ~50min to generate the 24 pipeline jobs. There is more time spent in each of those 24 to generate the compile job and then the test jobs. The generation runs on a slave so it is having to communicate back to master. A quick test on a nightly run on master instead of a slave took 40 seconds to generate the same 24 pipelines. I'm going to do a bit more digging and look at our options before I change anything (including delivering #6275).

smlambert commented 5 years ago

As there is no compelling reason to be generating jobs each time, should we consider not doing it?

AdamBrousseau commented 5 years ago

While I agree that generating the jobs is unnecessary 99% of the time, the benefits outweigh the drawbacks imo, assuming we can get the times under control. As far as I know the only drawback is the time it is now taking. Having everything automated is the goal.

I have a PR #6276 that generates on the master instead of the worker and the perf is much better. https://ci.eclipse.org/openj9/job/PullRequest-OpenJ9/639/console Running all 22 in parallel takes 21 seconds.

11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@2
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@3
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@4
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@5
[Pipeline] node
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@6
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@7
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@8
[Pipeline] node
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@9
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@10
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@11
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@12
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@13
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@14
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@15
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@16
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@17
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@18
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@19
[Pipeline] node
[Pipeline] node
[Pipeline] node
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@20
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@21
11:40:06  Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/PullRequest-OpenJ9/workspace@22
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
11:40:07  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_linux_xl'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_mac'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_windows'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_s390x_linux'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64le_linux'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_s390x_linux'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_ppc64le_linux'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_linux_xl'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26  Added items:
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64_aix'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_linux'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_linux'}
11:40:26  Unreferenced items:
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Unreferenced items:
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_mac'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_s390x_linux'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_windows'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK12_ppc64_aix'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-32_windows'}
11:40:26  Unreferenced items:
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_linux'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_linux_xl'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Added items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_windows'}
11:40:26  Unreferenced items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:26  Existing items:
11:40:26      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:27  Added items:
11:40:27      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
11:40:27  Unreferenced items:
11:40:27      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
11:40:27  Added items:
11:40:27      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_mac'}
11:40:27  Unreferenced items:
11:40:27      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
[Pipeline] }
[Pipeline] // node
[Pipeline] stage
[Pipeline] { (Pipeline_Build_Test_JDK8_ppc64le_linux)
[Pipeline] build (Building Pipeline_Build_Test_JDK8_ppc64le_linux)
11:40:34  Scheduling project: Pipeline_Build_Test_JDK8_ppc64le_linux
AdamBrousseau commented 5 years ago

Can confirm similar numbers on internal Jenkins. 33 Seconds for 23 jobs.

00:03:18.246  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace
[Pipeline] node
[Pipeline] node
00:03:18.340  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@2
00:03:18.341  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@3
[Pipeline] node
00:03:18.343  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@4
[Pipeline] node
[Pipeline] node
00:03:18.493  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@5
00:03:18.598  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@6
[Pipeline] node
00:03:18.634  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@7
[Pipeline] node
00:03:18.659  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@8
[Pipeline] node
[Pipeline] node
00:03:18.665  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@9
[Pipeline] node
[Pipeline] node
[Pipeline] node
00:03:18.693  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@10
[Pipeline] node
00:03:18.700  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@11
00:03:18.701  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@12
00:03:18.716  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@13
[Pipeline] node
[Pipeline] node
00:03:18.727  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@14
00:03:18.728  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@15
[Pipeline] node
[Pipeline] node
[Pipeline] node
00:03:18.779  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@16
00:03:18.780  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@17
[Pipeline] node
00:03:18.794  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@18
00:03:18.795  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@19
[Pipeline] node
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
00:03:19.739  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@20
[Pipeline] unstash
00:03:19.751  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@21
00:03:19.751  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@22
[Pipeline] unstash
[Pipeline] unstash
00:03:19.785  Running on Jenkins in /var/jenkins_home/jobs/Pipeline-Build-Test-All/workspace@23
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
[Pipeline] jobDsl
00:03:20.223  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
00:03:20.224  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
00:03:20.224  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
00:03:20.241  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:21.764  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:21.863  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:21.891  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:22.494  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:22.570  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:22.643  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:22.683  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
[Pipeline] {
00:03:22.772  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] unstash
[Pipeline] jobDsl
00:03:23.095  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:23.129  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:23.196  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
[Pipeline] jobDsl
00:03:23.309  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
00:03:23.347  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:23.421  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:23.572  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:23.693  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
00:03:23.784  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
[Pipeline] jobDsl
[Pipeline] jobDsl
00:03:23.926  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
00:03:23.951  Processing DSL script buildenv/jenkins/jobs/pipelines/Pipeline_Template.groovy
00:03:43.551  Added items:
00:03:43.551      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_linux'}
00:03:43.551  Unreferenced items:
00:03:43.551      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:43.553  Added items:
00:03:43.553      GeneratedJob{name='Pipeline_Build_Test_JDK12_ppc64le_linux'}
00:03:43.553  Unreferenced items:
00:03:43.553      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:44.224  Added items:
00:03:44.224      GeneratedJob{name='Pipeline_Build_Test_JDK12_s390x_linux'}
00:03:44.224  Unreferenced items:
00:03:44.224      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:44.681  Added items:
00:03:44.681      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_linux'}
00:03:44.681  Unreferenced items:
00:03:44.681      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:48.062  Added items:
00:03:48.063      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_linux'}
00:03:48.063  Unreferenced items:
00:03:48.063      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:48.251  Added items:
00:03:48.251      GeneratedJob{name='Pipeline_Build_Test_JDK8_s390x_linux'}
00:03:48.251  Unreferenced items:
00:03:48.251      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:48.525  Added items:
00:03:48.525      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_linux_xl'}
00:03:48.525  Unreferenced items:
00:03:48.525      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:49.315  Added items:
00:03:49.315      GeneratedJob{name='Pipeline_Build_Test_JDK11_s390x_linux'}
00:03:49.315  Unreferenced items:
00:03:49.315      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:49.674  Added items:
00:03:49.674      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_linux_xl'}
00:03:49.674  Unreferenced items:
00:03:49.674      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:49.995  Added items:
00:03:49.995      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_windows'}
00:03:49.995  Unreferenced items:
00:03:49.995      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.171  Existing items:
00:03:50.172      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.179  Added items:
00:03:50.179      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64_aix'}
00:03:50.179  Unreferenced items:
00:03:50.179      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.302  Added items:
00:03:50.302      GeneratedJob{name='Pipeline_Build_Test_JDK11_s390x_zos'}
00:03:50.302  Unreferenced items:
00:03:50.302      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.548  Added items:
00:03:50.548      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_windows'}
00:03:50.548  Unreferenced items:
00:03:50.548      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.569  Added items:
00:03:50.569      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_mac'}
00:03:50.569  Unreferenced items:
00:03:50.569      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.993  Added items:
00:03:51.001      GeneratedJob{name='Pipeline_Build_Test_JDK12_ppc64_aix'}
00:03:51.001  Unreferenced items:
00:03:51.001      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:50.997  Added items:
00:03:51.002      GeneratedJob{name='Pipeline_Build_Test_JDK11_x86-64_mac'}
00:03:51.002  Unreferenced items:
00:03:51.002      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:51.003  Added items:
00:03:51.003      GeneratedJob{name='Pipeline_Build_Test_JDK11_ppc64le_linux'}
00:03:51.004  Unreferenced items:
00:03:51.004      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:51.016  Added items:
00:03:51.016      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-64_mac'}
00:03:51.016  Unreferenced items:
00:03:51.016      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:51.205  Added items:
00:03:51.205      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_windows'}
00:03:51.205  Unreferenced items:
00:03:51.205      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:51.215  Added items:
00:03:51.215      GeneratedJob{name='Pipeline_Build_Test_JDK12_x86-64_linux_xl'}
00:03:51.215  Unreferenced items:
00:03:51.215      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:51.249  Added items:
00:03:51.250      GeneratedJob{name='Pipeline_Build_Test_JDK8_x86-32_windows'}
00:03:51.250  Unreferenced items:
00:03:51.250      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
00:03:51.288  Added items:
00:03:51.288      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64le_linux'}
00:03:51.288  Unreferenced items:
00:03:51.288      GeneratedJob{name='Pipeline_Build_Test_JDK8_ppc64_aix'}
[Pipeline] }
[Pipeline] // node
[Pipeline] stage
[Pipeline] { (Pipeline_Build_Test_JDK11_x86-64_linux)
[Pipeline] build (Building Pipeline_Build_Test_JDK11_x86-64_linux)
00:04:05.441  Scheduling project: Pipeline_Build_Test_JDK11_x86-64_linux
AdamBrousseau commented 5 years ago

Will have to do a similar PR for the test gen job.

fjeremic commented 5 years ago

Thanks for looking into this so quickly. I'd say if the whole generation step takes under a minute or two that is very reasonable.

AdamBrousseau commented 5 years ago

For the test autogen job I simply configured it to run on master instead of worker and the time trend went from >2 minutes to <25 seconds.

pshipton commented 5 years ago

I think we can close this now? Fixed by #6276

AdamBrousseau commented 4 years ago

This seems to be a problem again. At least for my pr build. Nightly looks fine. I will have to look into why this tends to get slower over time. We could ask Eclipse for some executors on master so we can run it there instead (We had to move it back to the worker agents when we moved to the new infra).