search

ros-infrastructure / buildfarm_deployment

Apache License 2.0
30 stars 39 forks source link

heads-up: Jenkins 2.107.1 breaks jobs on upgrade #193

Closed gavanderhoorn closed 5 years ago

gavanderhoorn commented 6 years ago

Just ran into this on a deployment: the latest version of Jenkins in the stable repository is (as of 2018-03-18) 2.107.1 (changelog, migration guide). After upgrading from 2.89.x, all my builds started failing due to -- what appears to be -- missing SSH credentials in all jobs. They all failed at the 'Send build artifacts over SSH' step (known issue: see Plugins affected by fix for JEP-200).

After logging in and checking the administrative issues panel many (all?) jobs are now listed as having "old data" that cannot be migrated/deserialised/updated because of changes to the security policies of a nr of components. This seems to be related to JEP-200. See for some more info/context.

As I didn't want to deal with this now I figured I'd downgrade to 2.89 and sort things out later. That doesn't work however, as part of the upgrade process all job configuration files are upgraded to include a XML 1.1 preamble, which makes those files not loadable in 2.89.

I thought I'd post here to give other admins a heads-up about this. I'm not sure whether JEP-200 will cause more trouble, but it seems like quite an invasive change (going from blacklist to whitelist).

gavanderhoorn commented 6 years ago

It could be that new deployments won't run into trouble, as they wouldn't have any "old data".

There do appear to be quite some plugins however that are not yet compatible with JEP-200, so that could prevent a successful deployment of a buildfarm instance on those recent Jenkins versions. See jenkins.io: Plugins affected by fix for JEP-200 for an overview.

gavanderhoorn commented 6 years ago

All Groovy postbuild steps were also gone from jobs.

gavanderhoorn commented 6 years ago

Downgrading was possible in the end by running:

for f in `grep -i "xml version='1.1'" -r . -l`; do sed -i "s/xml version='1.1'/xml version='1.0'/g" $f; done

in /var/lib/jenkins.

After restarting Jenkins jobs are running normally again, although it now complains that I'm using an out-of-date version (which is true).

nuclearsandwich commented 6 years ago

Thanks for the heads up. While we work on the upgrade path we should pin the previous LTS release in buildfarm_deployment_config so new farms are working by default.

gavanderhoorn commented 6 years ago

(note to self) for now I've used apt-mark hold jenkins to keep it at a known working version.

nuclearsandwich commented 5 years ago

@gavanderhoorn if I'm not mistaken https://github.com/ros-infrastructure/buildfarm_deployment/pull/207 resolves this issue, correct?

gavanderhoorn commented 5 years ago

Technically it doesn't ;) But it's not longer an issue, true.

Closing.