shutdown may unsuccessful by blocked IO

[GoogleCodeExporter]

Version: spymemcached-2.7.3

MemcachedClient created a non-daemon thread and blocked JVM shutdown even if 
all other thread stopped and even System.exit() were called.

Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode):

"Attach Listener" daemon prio=10 tid=0x000000005ad50000 nid=0x166e runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" prio=10 tid=0x00002aaab0029800 nid=0x348f waiting on condition 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Memcached IO over {MemcachedConnection to /192.168.147.157:11211}" prio=10 
tid=0x00002aaab005e800 nid=0x35e1 runnable [0x0000000042d16000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00000000ecf38648> (a sun.nio.ch.Util$2)
        - locked <0x00000000ecf38638> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ecf37810> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:249)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)

"Low Memory Detector" daemon prio=10 tid=0x000000005ade9000 nid=0x34e5 runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x000000005ade7000 nid=0x34e4 waiting on 
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x000000005ade1800 nid=0x34e3 waiting on 
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x000000005addf000 nid=0x34e2 runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x000000005adbb800 nid=0x34b6 in Object.wait() 
[0x0000000042811000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000ecf5c120> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00000000ecf5c120> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000000005adb9800 nid=0x34b5 in 
Object.wait() [0x0000000042710000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000ecf64090> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x00000000ecf64090> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x000000005adb5800 nid=0x34af runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005ad63000 nid=0x3494 
runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005ad65000 nid=0x3495 
runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005ad66800 nid=0x3496 
runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005ad68800 nid=0x3497 
runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005ad6a800 nid=0x3498 
runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005ad6c800 nid=0x3499 
runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005ad6e000 nid=0x349a 
runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005ad70000 nid=0x349b 
runnable

"VM Periodic Task Thread" prio=10 tid=0x000000005adf4000 nid=0x34e7 waiting on 
condition

Original issue reported on code.google.com by hkdenni...@gmail.com on 23 Dec 2011 at 3:50

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

No. It didn't, but System.exit(0) is called.

Original comment by hkdenni...@gmail.com on 24 Dec 2011 at 5:35

[GoogleCodeExporter]

There is an option to start the client with daemon threads.  Is this what you'd 
like to do?

Original comment by ingen...@cep.net on 27 Dec 2011 at 9:51

[GoogleCodeExporter]

Original comment by ingen...@gmail.com on 27 Dec 2011 at 9:52

• Changed state: NeedInfo

[GoogleCodeExporter]

I started it with default settings, which should not non-daemon thread.

It just did not shutdown safe all the time but RANDOMLY hold, it happen less 
then 5% of all the time.

Original comment by hkdenni...@gmail.com on 28 Dec 2011 at 3:39

[GoogleCodeExporter]

So, if I try to start / stop 100 times, I should see this issue 5 times?  Any 
steps to reproduce?

Original comment by ingen...@gmail.com on 28 Dec 2011 at 4:01

[GoogleCodeExporter]

I connected to one MC(on a not excellent network), then do a batch of 
replace/add, followed by a System.exit(0).

Original comment by hkdenni...@gmail.com on 28 Dec 2011 at 5:25