[JENKINS-48899] Jenkins downgrades detached plugins on upgrade

[timja]

Seems to be caused by JENKINS-48365, similar to JENKINS-48604 but not the same.

Steps to reproduce:

• Install Jenkins 2.73
• Upgrade to 2.89
• Install/Upgrade command-launcher from the Plugin Manager (to version 1.2)
• Upgrade to 2.101
• command-launcher has been downgraded to 1.0

The problem seems to be that given that upgrade detection was also fixed by JENKINS-48365 in the upgrade from 2.89 to 2.101 the core thinks it is upgrading from 2.73 so it "blindly" install command-launcher without checking if it is installed, overwriting the installed version with the bundled, older one.

Last step logs:

webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
ene 11, 2018 6:55:08 PM Main deleteWinstoneTempContents
ADVERTENCIA: Failed to delete the temporary Winstone file /var/folders/ry/pkk2p_d110lgzpwn80f_tr080000gn/T/winstone/jenkins-war-2.101.war
ene 11, 2018 6:55:08 PM org.eclipse.jetty.util.log.Log initialized
INFORMACIÓN: Logging initialized @464ms to org.eclipse.jetty.util.log.JavaUtilLog
ene 11, 2018 6:55:08 PM winstone.Logger logInternal
INFORMACIÓN: Beginning extraction from war file
ene 11, 2018 6:55:11 PM org.eclipse.jetty.server.handler.ContextHandler setContextPath
ADVERTENCIA: Empty contextPath
ene 11, 2018 6:55:11 PM org.eclipse.jetty.server.Server doStart
INFORMACIÓN: jetty-9.4.z-SNAPSHOT
ene 11, 2018 6:55:12 PM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFORMACIÓN: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
ene 11, 2018 6:55:12 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFORMACIÓN: DefaultSessionIdManager workerName=node0
ene 11, 2018 6:55:12 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFORMACIÓN: No SessionScavenger set, using defaults
ene 11, 2018 6:55:12 PM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFORMACIÓN: Scavenging every 600000ms
Jenkins home directory: /Users/andresrc/Documents/Testing/CJP-8205/oss/home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
ene 11, 2018 6:55:14 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFORMACIÓN: Started w.@74cec793{/,file:///Users/andresrc/Documents/Testing/CJP-8205/oss/home/war/,AVAILABLE}{/Users/andresrc/Documents/Testing/CJP-8205/oss/home/war}
ene 11, 2018 6:55:14 PM org.eclipse.jetty.server.AbstractConnector doStart
INFORMACIÓN: Started ServerConnector@754777cd{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
ene 11, 2018 6:55:14 PM org.eclipse.jetty.server.Server doStart
INFORMACIÓN: Started @6726ms
ene 11, 2018 6:55:14 PM winstone.Logger logInternal
INFORMACIÓN: Winstone Servlet Engine v4.0 running: controlPort=disabled
ene 11, 2018 6:55:15 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Started initialization
ene 11, 2018 6:55:15 PM hudson.PluginManager loadDetachedPlugins
INFORMACIÓN: Upgrading Jenkins. The last running version was 2.73. This Jenkins is version 2.101.
ene 11, 2018 6:55:15 PM hudson.PluginManager loadDetachedPlugins
INFORMACIÓN: Upgraded Jenkins from version 2.73 to version 2.101. Loaded detached plugins (and dependencies): [command-launcher.hpi]
ene 11, 2018 6:55:15 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Listed all plugins
ene 11, 2018 6:55:21 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Prepared all plugins
ene 11, 2018 6:55:21 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Started all plugins
ene 11, 2018 6:55:23 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Augmented all extensions
ene 11, 2018 6:55:23 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Loaded all jobs
ene 11, 2018 6:55:23 PM hudson.model.AsyncPeriodicWork$1 run
INFORMACIÓN: Started Download metadata
ene 11, 2018 6:55:23 PM hudson.model.AsyncPeriodicWork$1 run
INFORMACIÓN: Finished Download metadata. 17 ms
ene 11, 2018 6:55:24 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFORMACIÓN: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@58eb85b5: display name [Root WebApplicationContext]; startup date [Thu Jan 11 18:55:24 CET 2018]; root of context hierarchy
ene 11, 2018 6:55:24 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFORMACIÓN: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@58eb85b5]: org.springframework.beans.factory.support.DefaultListableBeanFactory@2f3a3b51
ene 11, 2018 6:55:24 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFORMACIÓN: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2f3a3b51: defining beans [authenticationManager]; root of factory hierarchy
ene 11, 2018 6:55:24 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFORMACIÓN: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@4fa96319: display name [Root WebApplicationContext]; startup date [Thu Jan 11 18:55:24 CET 2018]; root of context hierarchy
ene 11, 2018 6:55:24 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFORMACIÓN: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@4fa96319]: org.springframework.beans.factory.support.DefaultListableBeanFactory@6a908666
ene 11, 2018 6:55:24 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFORMACIÓN: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@6a908666: defining beans [filter,legacy]; root of factory hierarchy
ene 11, 2018 6:55:24 PM jenkins.InitReactorRunner$1 onAttained
INFORMACIÓN: Completed initialization
ene 11, 2018 6:55:24 PM hudson.UDPBroadcastThread run
INFORMACIÓN: Cannot listen to UDP port 33.848, skipping: java.net.SocketException: Can't assign requested address
ene 11, 2018 6:55:24 PM hudson.WebAppMain$3 run
INFORMACIÓN: Jenkins is fully up and running

---

Originally reported by andresrc, imported from: Jenkins downgrades detached plugins on upgrade

• assignee: dnusbaum
• status: Resolved
• priority: Critical
• resolution: Fixed
• resolved: 2018-01-22T13:03:52+00:00
• imported: 2022/01/10

[timja]

dnusbaum:

INFORMACIÓN: Upgrading Jenkins. The last running version was 2.73. This Jenkins is version 2.101. ene 11, 2018 6:55:15 PM hudson.PluginManager loadDetachedPlugins

I am somewhat surprised that 2.89 was not detected as the last running version, but it looks like only the following events save the currently running version, so if none of these happened then it is possible that Jenkins never saves 2.89 as a running version:

• Restarting Edit: See later comment
• Downgrading
• Going through the setup wizard
• Installing detached plugins after the fix for JENKINS-48365

There is code that is supposed to check that detached plugins are not downgraded, and I'm not sure why that didn't work. I will try to reproduce locally.

[timja]

dnusbaum:

Ah, never mind, it is because the linked code only returns early if it knows the plugin should be installed, so it falls through to the next if statement which returns true because the last version is wrong, so it gets installed.

I think it makes sense to save the running version somewhere that happens more consistently than in the 4 places mentioned previously, and it also makes sense to invert the first check in LoadDetachedPlugins to return false early if the plugin is already installed and is newer than the currently running version.

[timja]

andresrc:

Exactly, 2.89 was never saved as a running version. My understanding is that saving the running version was also fixed in JENKINS-48365, the problem is that the current code needs to take into account upgrades from versions with the bug in saving the running version.

[timja]

andresrc:

Yes, the problem is the second if when having the incorrect lastExecVersion.

[timja]

dnusbaum:

My understanding is that saving the running version was also fixed in JENKINS-48365

No, saving the running version was unchanged by JENKINS-48365, it just changed the condition used to decide if an upgrade is occurring (Switched to comparing versions directly instead of checking for InstallState.UPGRADE).

EDIT: Since one of the events that saves the running version is detached plugins being installed when an upgrade is detected, I guess JENKINS-48365 did fix the version number saving issue as it causes upgrades to be detected reliably.

I think that inverting the check in LoadDetachedPlugins will fix the problem for all instances regardless of what version has been detected, but I will make sure to add tests to verify.

[timja]

andresrc:

Yes, but given this, by fixing when an upgrade was occurring you also started saving the running version when that actually happened.

[timja]

dnusbaum:

Yeah good point, I realized that after I wrote my previous comment and edited. That means that I just need to fix these lines and add some tests.

[timja]

dnusbaum:

I realized that of the ways to save the running version:

1. Restarting
2. Downgrading
3. Going through the setup wizard
4. Installing detached plugins after the fix for JENKINS-48365

The first one doesn't really matter, because the restart state is only triggered if the running version is detected to be the same as before, so saving it again is pointless.

[timja]

scm_issue_link:

Code changed in jenkins
User: Devin Nusbaum
Path:
core/src/main/java/hudson/PluginManager.java
test/src/test/java/jenkins/install/LoadDetachedPluginsTest.java
test/src/test/resources/jenkins/install/LoadDetachedPluginsTest/upgradeFromJenkins2WithNewerPlugin.zip
http://jenkins-ci.org/commit/jenkins/d6298979581da67336f077cda9fd218eb790bdb3
Log:
JENKINS-48899 Do not downgrade detached plugins when upgrading Jenkins (#3229)

• Reproduce JENKINS-48899

• Do not downgrade installed detached plugins

[timja]

oleg_nenashev:

It has been fixed and released in 2.103

[timja]

scm_issue_link:

Code changed in jenkins
User: Devin Nusbaum
Path:
core/src/main/java/hudson/PluginManager.java
test/src/test/java/jenkins/install/LoadDetachedPluginsTest.java
test/src/test/resources/jenkins/install/LoadDetachedPluginsTest/upgradeFromJenkins2WithNewerPlugin.zip
http://jenkins-ci.org/commit/jenkins/7b6c86dc581e573a91802639647630ae40d8e208
Log:
JENKINS-48899 Do not downgrade detached plugins when upgrading Jenkins (#3229)

• Reproduce JENKINS-48899

• Do not downgrade installed detached plugins

(cherry picked from commit d6298979581da67336f077cda9fd218eb790bdb3)