[JENKINS-28352] Jenkins restart hangs

[timja]

I am starting jenkins with

java -DsessionTimeout=1440 -jar jenkins.war

After that If I try to restart jenkins by issuing restart request from browser then Jenkins hangs with error message

WARNING: Failed to restart Hudson
java.io.IOException: Failed to exec '' No such file or directory
at hudson.lifecycle.UnixLifecycle.restart(UnixLifecycle.java:84)
at jenkins.model.Jenkins$22.run(Jenkins.java:3357)

---

Originally reported by rajesh_kumar, imported from: Jenkins restart hangs

• status: Resolved
• priority: Minor
• resolution: Cannot Reproduce
• resolved: 2022-05-04T20:39:59+00:00
• imported: 2022/01/10

[timja]

danielbeck:

Too little information about your environment to investigate further (e.g. Java version, OS and version, whether there are other warnings/errors in the log, etc.)

Please follow the instructions on https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

[timja]

rajesh_kumar:

Added more details already..

[timja]

kairatra:

Reproduced with Jenkins 2.7.2 and JVM 1.8.0_101 on SunOS 5.10 (Solaris 10). No error or warning in the log.

[timja]

jimklimov:

Gave this a shot today on OpenIndiana Hipster updated yesterday with jenkins-2.63 (current as of today), and failed to reproduce the issue.

$ java -version
openjdk version "1.8.0_131-internal"
OpenJDK Runtime Environment (build 1.8.0_131-internal-oi-1.8.131-b00)
OpenJDK 64-Bit Server VM (build 25.131-b00, mixed mode)

$ cat /etc/release 
     OpenIndiana Hipster 2017.04 (powered by illumos)
OpenIndiana Project, part of The Illumos Foundation (C) 2010-2017
Use is subject to license terms.
   Assembled 01 May 2017

$ wget http://mirrors.jenkins.io/war/latest/jenkins.war
$ mkdir /tmp/jjj
$ LIBZFS4J_ABI=openzfs JENKINS_HOME=/tmp/jjj java -DsessionTimeout=1440 -jar jenkins.war --httpPort=9090

I did not find a direct way to cause a restart from inside jenkins beside the graceful one I know - by downloading a plugin and installing it "after a restart" triggering one with the checkbox

...
Jun 02, 2017 1:32:02 PM hudson.model.UpdateCenter$DownloadJob run
INFO: Starting the installation of Build Monitor View on behalf of admin
Jun 02, 2017 1:32:02 PM hudson.model.UpdateCenter$InstallationJob _run
INFO: Skipping duplicate install of: Build Monitor View@1.11+build.201701152243
Jun 02, 2017 1:32:02 PM hudson.model.UpdateCenter$DownloadJob run
INFO: Installation successful: Build Monitor View

Jun 02, 2017 1:32:15 PM hudson.model.UpdateCenter doSafeRestart
INFO: Scheduling Jenkins reboot
Jun 02, 2017 1:32:15 PM jenkins.model.Jenkins$26 run
INFO: Restart in 10 seconds

Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins$26 run
SEVERE: Restarting VM as requested by admin
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins cleanUp
INFO: Stopping Jenkins
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins$21 onAttained
INFO: Started termination
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins$21 onAttained
INFO: Completed termination
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins _cleanUpDisconnectComputers
INFO: Starting node disconnection
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins _cleanUpShutdownPluginManager
INFO: Stopping plugin manager
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins _cleanUpPersistQueue
INFO: Persisting build queue
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins _cleanUpAwaitDisconnects
INFO: Waiting for node disconnection completion
Jun 02, 2017 1:32:25 PM jenkins.model.Jenkins cleanUp
INFO: Jenkins stopped
Running from: /tmp/jjj/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Jun 02, 2017 1:32:26 PM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file /var/tmp/winstone/jenkins.war
Jun 02, 2017 1:32:26 PM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @510ms to org.eclipse.jetty.util.log.JavaUtilLog
Jun 02, 2017 1:32:27 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Jun 02, 2017 1:32:27 PM org.eclipse.jetty.server.handler.ContextHandler setContextPath
WARNING: Empty contextPath
Jun 02, 2017 1:32:27 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT
Jun 02, 2017 1:32:27 PM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Jun 02, 2017 1:32:27 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Jun 02, 2017 1:32:27 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Jun 02, 2017 1:32:27 PM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: Scavenging every 660000ms
Jenkins home directory: /tmp/jjj found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Jun 02, 2017 1:32:28 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@4681c175{/,file:///tmp/jjj/war/,AVAILABLE}{/tmp/jjj/war}
Jun 02, 2017 1:32:28 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@2b30a42c{HTTP/1.1,[http/1.1]}{0.0.0.0:9090}
Jun 02, 2017 1:32:28 PM org.eclipse.jetty.server.Server doStart
INFO: Started @1718ms
Jun 02, 2017 1:32:28 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
Jun 02, 2017 1:32:28 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Jun 02, 2017 1:32:29 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Jun 02, 2017 1:32:34 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Jun 02, 2017 1:32:34 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Jun 02, 2017 1:32:34 PM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
INFO: Failed to instantiate optional component hudson.plugins.build_timeout.operations.AbortAndRestartOperation$DescriptorImpl; skipping
Jun 02, 2017 1:32:51 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Jun 02, 2017 1:32:52 PM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Jun 02, 2017 1:32:52 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Jun 02, 2017 1:32:52 PM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Jun 02, 2017 1:32:52 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 13 ms
Jun 02, 2017 1:32:52 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@14ecbec1: display name [Root WebApplicationContext]; startup date [Fri Jun 02 13:32:52 CEST 2017]; root of context hierarchy
Jun 02, 2017 1:32:52 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@14ecbec1]: org.springframework.beans.factory.support.DefaultListableBeanFactory@3ad59549
Jun 02, 2017 1:32:52 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3ad59549: defining beans [authenticationManager]; root of factory hierarchy
Jun 02, 2017 1:32:53 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@2433566a: display name [Root WebApplicationContext]; startup date [Fri Jun 02 13:32:53 CEST 2017]; root of context hierarchy
Jun 02, 2017 1:32:53 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@2433566a]: org.springframework.beans.factory.support.DefaultListableBeanFactory@4e3b8d84
Jun 02, 2017 1:32:53 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4e3b8d84: defining beans [filter,legacy]; root of factory hierarchy
Jun 02, 2017 1:32:53 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
Jun 02, 2017 1:32:54 PM javax.jmdns.impl.tasks.state.DNSStateTask run
WARNING: Prober(0-0-0-0.local.).run() exception 
java.io.IOException: Network is unreachable (sendto failed)
 at java.net.PlainDatagramSocketImpl.send(Native Method)
 at java.net.DatagramSocket.send(DatagramSocket.java:693)
 at javax.jmdns.impl.JmDNSImpl.send(JmDNSImpl.java:1537)
 at javax.jmdns.impl.tasks.state.DNSStateTask.run(DNSStateTask.java:131)
 at java.util.TimerThread.mainLoop(Timer.java:555)
 at java.util.TimerThread.run(Timer.java:505)

Jun 02, 2017 1:32:54 PM javax.jmdns.impl.JmDNSImpl __recover
WARNING: RECOVERING
Jun 02, 2017 1:32:54 PM javax.jmdns.impl.tasks.state.DNSStateTask run
WARNING: Canceler(0-0-0-0.local.).run() exception 
java.io.IOException: Network is unreachable (sendto failed)
 at java.net.PlainDatagramSocketImpl.send(Native Method)
 at java.net.DatagramSocket.send(DatagramSocket.java:693)
 at javax.jmdns.impl.JmDNSImpl.send(JmDNSImpl.java:1537)
 at javax.jmdns.impl.tasks.state.DNSStateTask.run(DNSStateTask.java:131)
 at java.util.TimerThread.mainLoop(Timer.java:555)
 at java.util.TimerThread.run(Timer.java:505)

Jun 02, 2017 1:32:59 PM javax.jmdns.impl.DNSStatefulObject$DefaultImplementation waitForCanceled
WARNING: Wait for canceled timed out: DNS: 0-0-0-0.local. state: canceling 1 task: null
Jun 02, 2017 1:32:59 PM javax.jmdns.impl.JmDNSImpl __recover
WARNING: 0-0-0-0.local.recover() Could not recover we are Down!
Jun 02, 2017 1:33:11 PM org.eclipse.jetty.server.session.Session beginInvalidate
INFO: Session node0ibn4k4fmrk0v5cw1n7a2mecr0 already being invalidated

Also to note, I tried to click"prepare for shutdown" first and it seems Jenkins ignored attempts to restart that way, until I canceled the shutdown.

Finally note I am not exactly on Solaris 10 - rather on OpenIndiana... maybe there are relevant differences by now...

[timja]

danielbeck:

Solaris 10 premier support ends Jan 2018, so not sure anyone will bother to look into this.

[timja]

basil:

There are no known issues on recent versions of illumos. If this problem occurs again on a recent version of illumos, please open a new ticket.