Tomcat process may not exit on Linux

GoogleCodeExporter commented 8 years ago

What steps will reproduce the problem?
1. Stop Tomcat using the bin/shutdown.sh script.

What is the expected output? What do you see instead?

The Tomcat java process should exit, but it may not. This has been experienced 
by multiple 
customers and internally. It may not happen every time.

Please use labels and text to provide additional information.

The workaround is simply to kill the process after waiting a reasonable time, 
say 30 seconds. The 
threads normally take 5-30 seconds to stop on other platforms, so killing the 
process 
immediately may interfere with a safe shutdown.

Details from Marty:

I see this on occasion.  When I attach a debugger to the process I see the 
WorkQueueThreads still 
running.  Looking at the log (at least in this case) it seems the LifeThread is 
resurrecting the 
WorkQueueThreads before it is killed.

Process stack:
-------------------
Java HotSpot(TM) Server VM[localhost:7010]    
    Thread [DestroyJavaVM] (Running)    
    Thread [WorkQueueThread-17] (Running)    
    Thread [WorkQueueThread-8] (Running)    
    Thread [WorkQueueThread-18] (Running)    
    Thread [WorkQueueThread-12] (Running)    
    Thread [WorkQueueThread-5] (Running)    
    Thread [WorkQueueThread-10] (Running)    
    Thread [WorkQueueThread-19] (Running)    
    Thread [WorkQueueThread-2] (Running)    
    Thread [WorkQueueThread-9] (Running)    
    Thread [WorkQueueThread-4] (Running)    
    Thread [WorkQueueThread-1] (Running)    
    Thread [WorkQueueThread-16] (Running)    
    Thread [WorkQueueThread-7] (Running)    
    Thread [WorkQueueThread-3] (Running)    
    Thread [WorkQueueThread-0] (Running)    
    Thread [WorkQueueThread-15] (Running)    
    Thread [WorkQueueThread-13] (Running)    
    Thread [WorkQueueThread-14] (Running)    
    Thread [WorkQueueThread-6] (Running)    
    Thread [WorkQueueThread-11] (Running)    
    Daemon Thread [http-7453-Processor3] (Running)    
    Daemon Thread [http-7090-Processor3] (Running)    
    Daemon Thread [Timer-0] (Running)    

Log:

---
Jul 2, 2008 10:50:13 AM com.google.enterprise.connector.servlet.StartUp destroy
INFO: destroy
Jul 2, 2008 10:50:13 AM com.google.enterprise.connector.manager.Context shutdown
INFO: shutdown
Jul 2, 2008 10:50:13 AM 
com.google.enterprise.connector.scheduler.TraversalScheduler 
shutdown
INFO: Shutdown initiated...
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.scheduler.TraversalScheduler run
INFO: TraversalScheduler thread is stopping due to shutdown or not being 
initialized.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM com.google.enterprise.connector.common.WorkQueueThread 
run
INFO: Interrupted WorkQueueThread is exiting due to interrupt and kill.
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-11
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-6
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-14
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-13
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-15
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-0
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-3
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-7
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-16
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-1
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-4
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-9
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-2
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-19
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-10
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-5
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-12
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-18
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-8
Jul 2, 2008 10:50:23 AM 
com.google.enterprise.connector.common.WorkQueue$LifeThread run
WARNING: WorkQueueThread was dead and is restarted by LifeThread: 
WorkQueueThread-17
Jul 2, 2008 10:50:23 AM org.apache.coyote.http11.Http11BaseProtocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-7090
Jul 2, 2008 10:50:23 AM org.apache.coyote.http11.Http11BaseProtocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-7453

Original issue reported on code.google.com by jl1615@gmail.com on 2 Jul 2008 at 11:08

GoogleCodeExporter commented 8 years ago

Original comment by mgron...@gmail.com on 18 Jul 2008 at 9:10

GoogleCodeExporter commented 8 years ago

r899 | mgronber | 2008-08-06 14:17:01 -0700 (Wed, 06 Aug 2008) | 16 lines

I've tested on Linux with this patch and it seems to shutdown every time.

In the WorkQueue.shutdown() method, it interrupts all the WorkThreads and tries
to shutdown the LifeThread but it never really waits for the LifeThread to be
done.  On top of this, it does not remove any of the interrupted threads from
the 'threads' HashSet.  Therefore, if the LifeThread happens to be in the wait()
state when shutdown is called (which is very likely), it will check the status
of all the threads in the 'threads' HashSet.  If it gets a chance to run before
the Tomcat server is shutdown and the JVM exits, then it will restart all the
interrupted WorkThreads and the Tomcat server will not be able to exit.

This change basically kills the LifeThread and makes sure it is killed before
taking care of all the WorkThreads.  It also moves the wait down to the end of
the LifeThread run loop so the state of the 'shutdown' field is checked
instantly before any work is done.

Original comment by mgron...@gmail.com on 6 Aug 2008 at 9:31

Changed state: Fixed

GoogleCodeExporter commented 8 years ago

Original comment by jl1615@gmail.com on 12 Jan 2009 at 3:30

Added labels: Milestone-Release_1.3.0

PILLUTLAAVINASH / google-enterprise-connector-manager

Tomcat process may not exit on Linux #102