Update deployment to support post-JEP-200 Jenkins versions

nuclearsandwich commented 5 years ago

The changes here seek to resolve issues with current Jenkins LTS.

PRs to buildfarm_deployment_config and ros_buildfarm are on the way once things settle down. It's important to note that #160 is still outstanding and will not be resolved by this PR. Manual changes to existing buildfarms for people who want to upgrade will get drafted here. When we migrate build.ros2.org and build.ros.org I'll make a discord announcement as well but if anyone wants to try the "pre-release" state of the instructions watch this space and I'll of course be grateful for the feedback.

While I originally started trying to update plugins minimally it quickly became overwhelming so I've moved to a strategy of update everything and check what breaks.

Summary of updates:

Forked upstream puppet-jenkins module to cherry pick changes we need for the newer swarm client plugin versions (Which themselves were needed to move off the deprecated JNLP version)
Update plugin versions to latest available.
Update of scriptApproval.xml with some additional signatures
Write a script that jenkins admins can run to sync with the plugin versions in rosplugins.pp without re-running puppet.

nuclearsandwich commented 5 years ago

This branch is now in production on build.ros2.org. The upgrade guide is being drafted in https://gist.github.com/nuclearsandwich/cfc272e7efd82384accd0cf595656dea

nuclearsandwich commented 5 years ago

@gavanderhoorn I'm not sure what the status of your buildfarm deployment is but if you'd like to review any of the changes your feedback is always very useful.

gavanderhoorn commented 5 years ago

I can't promise anything, but I'll see if I can attempt a migration of one of our deployments. They're not all busy so if your changes break one of them it won't be the end of the world ;)

asierfernandez commented 5 years ago

We've set up a clean instance of the buildfarm to test those PR. We fixed #38 in our private forked repo.

Once Jenkins was initialized we've noticed that there is no connection between the master and the agents and we had to enable JNLP connection Jenkin's UI -> Manage Jenkins -> Configure Global Security -> TCP port for JNLP agents , and select *Random*

After that, when trying to execute K_rel_trigger-jobs, it returned us following error: org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such field found: field java.lang.String size

# BEGIN SECTION: Run Dockerfile - trigger jobs
11:51:56 + rm -fr /home/jenkins-agent/workspace/Krel_trigger-jobs/debian_repo_cache
11:51:56 + mkdir -p /home/jenkins-agent/workspace/Krel_trigger-jobs/debian_repo_cache
11:51:56 + docker run --rm --cidfile=/home/jenkins-agent/workspace/Krel_trigger-jobs/docker_trigger_jobs/docker.cid --net=host -v /home/jenkins-agent/workspace/Krel_trigger-jobs/ros_buildfarm:/tmp/ros_buildfarm:ro -v /home/jenkins-agent/.buildfarm:/home/buildfarm/.buildfarm:ro -v /home/jenkins-agent/workspace/Krel_trigger-jobs/trigger_jobs:/tmp/trigger_jobs -v /home/jenkins-agent/workspace/Krel_trigger-jobs/debian_repo_cache:/tmp/debian_repo_cache release_trigger_jobs
11:51:56 The build file contains the following targets:
11:51:56   - ubuntu xenial source
11:51:56 Writing groovy script '/tmp/trigger_jobs/trigger_jobs.groovy' to trigger 56 jobs
11:51:57 + echo # END SECTION
11:51:57 # END SECTION
11:51:57 ERROR: Build step failed with exception
11:51:57 org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such field found: field java.lang.String size
11:51:57    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.unclassifiedField(SandboxInterceptor.java:419)
11:51:57    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:403)
11:51:57    at org.kohsuke.groovy.sandbox.impl.Checker$6.call(Checker.java:288)
11:51:57    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:292)
11:51:57    at org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:255)
11:51:57    at org.kohsuke.groovy.sandbox.impl.Checker$checkedGetProperty$0.callStatic(Unknown Source)
11:51:57    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
11:51:57    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
11:51:57    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:230)
11:51:57    at Script1.run(Script1.groovy:118)
11:51:57    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.run(GroovySandbox.java:141)
11:51:57    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.evaluate(SecureGroovyScript.java:333)
11:51:57    at hudson.plugins.groovy.SystemGroovy.run(SystemGroovy.java:95)
11:51:57    at hudson.plugins.groovy.SystemGroovy.perform(SystemGroovy.java:59)
11:51:57    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
11:51:57    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
11:51:57    at hudson.model.Build$BuildExecution.build(Build.java:206)
11:51:57    at hudson.model.Build$BuildExecution.doRun(Build.java:163)
11:51:57    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
11:51:57    at hudson.model.Run.execute(Run.java:1819)
11:51:57    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:51:57    at hudson.model.ResourceController.execute(ResourceController.java:97)
11:51:57    at hudson.model.Executor.run(Executor.java:429)
11:51:57 Build step 'Execute system Groovy script' marked build as failure

First I thought the script would be asking for permissions in Manage jenkins > In-process Script Approval, but there was No pending signature approvals.

nuclearsandwich commented 5 years ago

@gavanderhoorn ooh yeah, there's an issue with the script security plugin that I still need to ticket upstream. In the meantime, https://github.com/ros-infrastructure/ros_buildfarm/pull/587 is the PR for updates to the ros_buildfarm library targeting the latest Jenkins LTS. Aside from putting parens on java.lang.String size in a few places so it is properly treated as a method rather than a field it's mostly updates to the plugin versions used.

nuclearsandwich commented 5 years ago

Thanks to your report @gavanderhoorn I realized that I'd completely omitted that branch from the migration guide. This has been remedied. Thanks so much for doing the testing!

gavanderhoorn commented 5 years ago

It's actually @asierfernandez who tested :)

nuclearsandwich commented 5 years ago

It's actually @asierfernandez who tested :)

My hat is off to the pair of you then. :bowing_man:

nuclearsandwich commented 5 years ago

Preserving the development branch for a week to allow others to migrate to master.

ros-infrastructure / buildfarm_deployment

Update deployment to support post-JEP-200 Jenkins versions #207