search

ros-infrastructure / cookbook-ros-buildfarm

Apache License 2.0
2 stars 2 forks source link

Update xunit plugin for 2.277 compatibility. #89

Closed nuclearsandwich closed 3 years ago

nuclearsandwich commented 3 years ago

The XStream changes in 2.277.1 are not compatible with the xunit plugin prior to 2.4.0. We were on 2.3.9.

In addition to xunit I had to update a number of other plugins in its dependency chain.

After this update, there is no longer a need to modify the XML configuration of jobs using the xunit results reporting facilities.

nuclearsandwich commented 3 years ago

During the authoring of this PR I ended up getting Jenkins in a hard-locked state because of plugin dependency conflicts which were severe enough to halt Jenkins startup. (Why the xunit plugin is in the dependency path of the GitHubSecurityRealm I shudder to think). In order to recover I ended up trying out the https://github.com/jenkinsci/plugin-installation-manager-tool to reinstall the earlier version of xunit and junit and get back on track.

It's still a bit rough to use but it works without a running Jenkins instance and so may be either an alternative to the jenkins_plugin cookbook resource or a future implementation vehicle for it.

nuclearsandwich commented 3 years ago

Build Status is a dev job which ran successfully after this change.

nuclearsandwich commented 3 years ago

The agent CI failure is the known flake starting up the jenkins-agent service which I really need to investigate...

ros-discourse commented 3 years ago

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/upgrade-to-jenkins-2-277-3-causing-job-failures-across-ros-build-farms/20013/4

gavanderhoorn commented 3 years ago

During the authoring of this PR I ended up getting Jenkins in a hard-locked state because of plugin dependency conflicts which were severe enough to halt Jenkins startup. (Why the xunit plugin is in the dependency path of the GitHubSecurityRealm I shudder to think). In order to recover I ended up trying out the https://github.com/jenkinsci/plugin-installation-manager-tool to reinstall the earlier version of xunit and junit and get back on track.

I'm rather surprised about the fall-out of all these minor (patch even?) updates.

Not pointing at you of course, @nuclearsandwich.

But from the version nrs mentioned, there appear to be a couple of minor/patch level updates which manage to completely take down an existing installation.

nuclearsandwich commented 3 years ago

But from the version nrs mentioned, there appear to be a couple of minor/patch level updates which manage to completely take down an existing installation.

Do you have more info? The hard-locked state I got into was due to a partial upgrade as I was resolving dependencies by hand by inspecting META_INF/MANIFEST files and missed the bootstrap4-api dep.

When I finally landed on the complete patch in this PR. I was able to update our production machines without any issues during a single chef update.

Our standard operating procedure for plugin upgrades is to do a live upgrade on one of our Jenkins hosts, then use a script like https://github.com/ros-infrastructure/buildfarm_deployment/blob/master/scripts/jenkins_plugins.py to fetch the live plugin setup so that we're always in a good state. We did not / could not do that here because the Jenkins update manager will only ever upgrade to the latest version with dependency resolution.

I have another round of plugin updates to check for security reasons. I think I will try using the plugin-installation-manager utility if a jump to the latest release for a plugin is assessed to be too risky.