Make use of openj9-jenkins repo to reduce the load on master

[pshipton]

Jenkins jobs start by running on master, and do a git clone of the openj9 repo in order to get the jenkins code required before they start running on a worker and then a Node. Although we may be using a reference repo, there are a couple of problems.

• the reference repo is on master which is under Eclipse Foundation control, updating it is problematic
• even with a reference repo, the workload can still be reduced by using a smaller repository

A new repo has been created https://github.com/eclipse/openj9-jenkins. We can move the necessary code to this repo.

IMO we should be moving only the minimum, although I think Adam wants to move the entire buildenv/jenkins directory and change the way that PR builds work.

buildenv/jenkins/docker-slaves/jenkinsfile-build-container.groovy buildenv/jenkins/jobs/pipelines/Pipeline-Build-Test-All.groovy buildenv/jenkins/jobs/infrastructure/copyrightCheck.groovy buildenv/jenkins/jobs/infrastructure/lineEndingsCheck.groovy buildenv/jenkins/jobs/infrastructure/signedOffByCheck.groovy

[pshipton]

@jdekonin @AdamBrousseau

[pshipton]

Many failures fetching from git on master. https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK8_ppc64le_linux_xl/242 https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK8_s390x_linux/1383/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK8_x86-32_windows/1298/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK8_x86-64_linux/1571/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK8_x86-64_linux_xl/1218/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK8_x86-64_windows/1471/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK11_ppc64_aix/1319/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK11_ppc64le_linux_xl/234/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK11_s390x_linux_xl/232/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK11_x86-64_linux_cm/880/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK11_x86-64_linux_xl/1229/ https://ci.eclipse.org/openj9/job/Pipeline_Build_Test_JDK11_x86-64_mac/1262/

[AdamBrousseau]

I believe this is a machine issue not a network issue

Resolving deltas: 100% (89628/89628), done.
fatal: fsync error on '.git/objects/pack/tmp_pack_NOhjze': I/O error
fatal: index-pack failed

Resolving deltas:  52% (46682/89628)
fatal: cannot pread pack file: Out of memory
fatal: index-pack failed

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:758)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:149)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:299)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
No credentials specified
Wiping out workspace first.
ERROR: Checkout failed
java.nio.file.FileSystemException: /var/jenkins_home/workspace/Pipeline_Build_Test_JDK8_ppc64le_linux_xl@script/.git/objects/info: Cannot allocate memory
    at sun.nio.fs.UnixException.translateToIOException(Unknown Source)
    at sun.nio.fs.UnixException.asIOException(Unknown Source)
Caused: java.nio.file.DirectoryIteratorException

Resolving deltas: 100% (89628/89628), done.
fatal: sha1 file '.git/objects/pack/tmp_idx_JtlsVm' write error: Bad file descriptor
fatal: index-pack failed

Will a separate repo help us in this case? Maybe, since the new repo will be smaller. Although we already do sparse checkout so I'm not sure what we will save.

[pshipton]

It's not the checkout which is the problem, it's the git fetch.

[AdamBrousseau]

Before the separate repo idea was proposed I proposed a separate set of "releng pr builds". Lets compare the solutions.

Releng builds

• Code stays in the same repo :+1:
• Regular builds are able to use lightweight checkout which should resolve all the master git issues. Whether that resolves all the master issues remains to be seen.
• Regular builds always pull master.
• Releng builds would be fired using a different keyword (eg. "Releng test sanity....) :-1: in order for pipeline changes to be tested in a PR build. Basically the PR builds we have today would remain as releng builds, and the new regular builds would be more simple and lightweight for Jenkins load.
• Changes to defaults.yml variable file would have to be tested via a releng build.
• Depends would still work. If you had pipeline changes with dependent changes the PR would need to be initialised from the releng PR.
• Separate 2nd level pipelines (with suffix)

Separate repo for Pipeline code

• Second repo :-1:
• PR builds can use the same keywords :+1: because they would be coming from different repos so the jobs can be configured differently to handle the different build types.
• Pipeline changes needed to handle the code split. :-1:
• Dependent PRs should still work. Might need some changes here to support the new repo.
• PRs from the new repo would be the same as they are today (non-lightweight checkout).
• PRs from the openj9 repo would be lightweight checkout.
• deaults.yml variable file will be located in the new repo.
• Separate 2nd level pipelines (with suffix)

I think they're very similar solutions, in terms of solving this issue. I do like the releng builds better. It does introduce a new trigger keyword but I think it's a more simple story overall. There will need to be code changes to handle the repo split. Shouldn't be too bad but might be a bit of a rabbit hole.

Update: The best attempt at a solution is the get the majority of the builds using lightweight checkout. So the path of least resistance/change to get that tested should be what we try. Based on discussion with Pete, if we can trim PBTA.groovy, PBTAny.groovy and BAny.groovy such that we can use lightweight checkout everywhere. This may be the easiest path to get us testing the theory. The side effect will be that we will not be able to test changes to these 3 files via a PR build. Unless we come up with some other build.

[pshipton]

https://github.com/eclipse/openj9/pull/8568 has resolved the problem, no need to use a different repo.