Web server freezing

[GoogleCodeExporter]

In certain circumstances, the Wonderland web server freezes for as long as 2 
- 3 minutes. During this time, many world operations (like login) don't 
work. 

See https://wonderland.dev.java.net/issues/show_bug.cgi?id=789 for some 
background.

Original issue reported on code.google.com by jonathan...@gmail.com on 13 Mar 2010 at 5:28

• Blocking: #5

[GoogleCodeExporter]

The stack trace below is from a frozen web server on Ubuntu Intrepid with JDK 
1.6.0_10.  I believe this is related to a bug in Grizzly, the underlying NIO 
framework:

https://grizzly.dev.java.net/issues/show_bug.cgi?id=547

Wonderland web server stack trace:

"Thread-8" daemon prio=10 tid=0x096e5000 nid=0xd2f runnable 
[0x8f925000..0x8f926030]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcher.preClose0(Native Method)
        at sun.nio.ch.SocketDispatcher.preClose(SocketDispatcher.java:41)
        at 
sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(SocketChannelImpl.java:6
84)
        - locked <0xb3e03e80> (a java.lang.Object)
        at 
java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelecta
bleCh
annel.java:201)
        at 
java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleCh
annel
.java:97)
        - locked <0xb3e03e20> (a java.lang.Object)
        at sun.nio.ch.SocketAdaptor.close(SocketAdaptor.java:352)
        at 
com.sun.grizzly.TCPSelectorHandler.closeChannel(TCPSelectorHandler.java:1354)
        at 
com.sun.grizzly.BaseSelectionKeyHandler.doAfterKeyCancel(BaseSelectionKeyHandler
.java
:229)
        at 
com.sun.grizzly.BaseSelectionKeyHandler.cancel(BaseSelectionKeyHandler.java:216)
        at 
com.sun.grizzly.http.SelectorThreadKeyHandler.cancel(SelectorThreadKeyHandler.ja
va:80
)
        at 
com.sun.enterprise.v3.services.impl.monitor.MonitorableSelectionKeyHandler.cance
l(Mon
itorableSelectionKeyHandler.java:80)
        at 
com.sun.grizzly.TCPSelectorHandler.addPendingKeyCancel(TCPSelectorHandler.java:6
13)
        at 
com.sun.grizzly.http.SelectorThreadKeyHandler.expire(SelectorThreadKeyHandler.ja
va:13
6)
        at com.sun.grizzly.TCPSelectorHandler.postSelect(TCPSelectorHandler.java:556)
        at 
com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:206)
        at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:8
86)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

Original comment by jonathan...@gmail.com on 14 Mar 2010 at 10:22

[GoogleCodeExporter]

To reproduce the issue:

- Launch a client
- Click "login"
- After the client starts downloading content, shut off the client's network 
connection

Original comment by jonathan...@gmail.com on 16 Mar 2010 at 7:51

[GoogleCodeExporter]

This issue was updated by revision r4394.

Set socket linger time to -1 in Grizzly in rev 4394. This seems to fix the 
disconnected client case.

Original comment by jonathan...@gmail.com on 16 Mar 2010 at 11:14

[GoogleCodeExporter]

After fixing the socket linger problem, the web server still freezes, although 
with a 
different stack trace (see attached). Note that all the http pool threads are 
stuck 
in flushChannel() with the same stack:

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:126)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)

Based on this information, Oleksiy from Grizzly made some suggestions for what 
to 
test next:

https://grizzly.dev.java.net/servlets/ReadMsg?list=users&msgNo=4833

Original comment by jonathan...@gmail.com on 8 Apr 2010 at 11:19

Attachments:

• jstack4.out

[GoogleCodeExporter]

The simple application below demonstrates the issue. With no changes, the web 
server 
freezes after a few threads start downloading (between 2 and 5). Following 
Oleksiy's 
suggestion and setting:

com.sun.grizzly.http.asyncwrite.enabled=true

In my.run.properties allows many clients (I tested up to 100) to connect 
without 
affecting the web server operation.

Sample code to freeze the server:

public class Main {
    private static final String BASE_URL = 
        "http://localhost:8080/wonderland-web-front/app/";

    private static final String[] URLS = new String[] {
        "wonderland-client.jar",
        "jme.jar",
        "jogl.jar",
        "jme-collada.jar",
        "jaxb-xjc.jar",
        "derby.jar"
    };

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        List<Thread> threads = new ArrayList<Thread>();

        for (int i = 0; i < 15; i++) {
            String url = BASE_URL + URLS[i % URLS.length];
            Thread t = new Thread(new DownloadThread(url, 5000));
            t.start();
            threads.add(t);
        }

        for (Thread t : threads) {
            try {
                t.join();
            } catch (InterruptedException ie) {
                Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ie);
            }
        }
    }

    static class DownloadThread implements Runnable {
        private final String url;
        private final long pause;

        public DownloadThread(String url, long pause) {
            this.url = url;
            this.pause = pause;
        }

        public void run() {
            try {
                URL u = new URL(url);

                byte[] buffer = new byte[1024 * 16];
                InputStream is = u.openStream();

                System.out.println("Start reading " + url);
                while (is.read(buffer) > 0) {
                    Thread.sleep(pause);
                }
                System.out.println("Done reading " + url);
            } catch (IOException ex) {
                Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
            } catch (InterruptedException ie) {
                Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ie);
            }
        }
    }
}

Original comment by jonathan...@gmail.com on 8 Apr 2010 at 11:27

[GoogleCodeExporter]

This issue was updated by revision r4408.

Enable async writes to avoid server stalls with slow clients

Original comment by jonathan...@gmail.com on 14 Apr 2010 at 5:24

[GoogleCodeExporter]

This has not been seen since we enabled async writes.

Original comment by jonathan...@gmail.com on 7 Sep 2011 at 6:29

• Changed state: Fixed