Reopening Grizzly Bug 1878

[glassfishrobot]

Hello,

I would like to reopen https://java.net/jira/browse/GRIZZLY-1878 because the issue is not resolved with version 2.3.29.

To summarize the problem again:

All asynchronous requests that use ChunkedOutput as entity and are canceled by the client persist in one of the DelayedExecutor DelayedQueues. This is causing a severe memory leak.

To observe the problem set e.g. a breakpoint at DelayedExecutor line 139 (in v2.3.28) and monitor the queue sizes. One of the queue sizes will grow with the number of asynchronous requests.

Affected Versions

[2.3.28, 2.3.29]

[glassfishrobot]

Reported by MarcelMerkle

[glassfishrobot]

@rlubke said: I haven't been able to reproduce the problem. Can you please provide a complete test case that I can run out of the box that demonstrates the issue?

Thanks.

[glassfishrobot]

marcelmerkle said: Hi Ryan,

you can find a repro case here: https://github.com/marcel91/grizzly-repro. Just import this project into Eclipse.

To reproduce the problem first start the Main.java class. It will start a http server on port 3003 providing one resource http://localhost:3003/example/test. Now you can either use a command line tool like curl to trigger requests or the Client.java class. The Client.java class will first trigger one request and then cancel it before the server finished it. To reproduce the problem with curl you have to cancel the command before the third "test" line was printed.

You can see the queue size growing at DelayedExecutor line 139.

Best regards, Marcel

[glassfishrobot]

@rlubke said: Thanks. Please give this build a shot: https://www.dropbox.com/s/pe0qawu2cie8z4o/grizzly-framework-2.3.30-SNAPSHOT.jar?dl=0

[glassfishrobot]

@rlubke said: While my change will force the expected behavior if there's an unbalanced FilterChainContext interaction between init and release, I wanted to look further into this test case.

Here's the init/release cycle from the test case:

operation=ACCEPT, message=null, address=null] added listener operation=ACCEPT, message=null, address=null] completed

operation=READ, message=org.glassfish.grizzly.http.HttpContent@7e9842d4, address=/127.0.0.1:51537] added listener

operation=WRITE, message=HttpResponsePacket () added listener operation=WRITE, message=null, address=/127.0.0.1:51537] completed

operation=WRITE, message=org.glassfish.grizzly.http.HttpContent@10d4a5ef, address=/127.0.0.1:51537] added listener operation=WRITE, message=null, address=/127.0.0.1:51537] completed

operation=WRITE, message=org.glassfish.grizzly.http.HttpContent@10d4a5ef, address=/127.0.0.1:51537] added listener operation=WRITE, message=null, address=/127.0.0.1:51537] completed

operation=WRITE, message=org.glassfish.grizzly.http.HttpContent@10d4a5ef, address=/127.0.0.1:51537] added listener operation=WRITE, message=null, address=/127.0.0.1:51537] completed

operation=WRITE, message=org.glassfish.grizzly.http.HttpContent@10d4a5ef, address=/127.0.0.1:51537] added listener operation=WRITE, message=null, address=/127.0.0.1:51537] completed

Notice that the READ operation results in the IdleTimeoutFilter adding a listener, but the context for the read is never completed.

I looked at the test case, I I think there's a small bug with it:

ChunkedOutput<byte[]> chunkedOutput = new ChunkedOutput<>(byte[].class, new byte[0]);        
        Response response = Response.status(200)
.header("Content-Type", "application/octet-stream")
.entity(chunkedOutput)
.build();
        async.resume(response);
        new Thread(new Runnable() {
            @Override
            public void run() {
veryExpensiveOperation();
            }

            private void veryExpensiveOperation() {
try {
    for (int i = 0; i < 3; ++i) {
        chunkedOutput.write("test\n".getBytes());
        Thread.sleep(1000);
    }
} catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
} finally {
    try {
        chunkedOutput.close();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}
            }
        }).start();

It seems to me that the async.resume(response): call should be done in the finally block of the thread. I made the change and the queue doesn't grow and works as expected without my change in place.

Can you comment on this?

[glassfishrobot]

marcelmerkle said: Your suggested change changes the behavior on the client side. While before every line was received one by one, now all three lines are received at once.

[glassfishrobot]

@rlubke said: Okay. Did you have time to test the patch jar?

[glassfishrobot]

marcelmerkle said: I just did. It seems to work.

[glassfishrobot]

marcelmerkle said: Referring to "While my change will force the expected behavior if there's an unbalanced FilterChainContext interaction between init and release":

Would it be possible to fix the root cause of this problem?

[glassfishrobot]

@rlubke said: I'm discussing it with the Jersey folks.

[glassfishrobot]

@rlubke said: Please try https://www.dropbox.com/s/9rc05jvh7xroksu/grizzly-http-server-2.3.31-SNAPSHOT.jar?dl=0. Don't include the patched framework jar in your test.

[glassfishrobot]

marcelmerkle said: The patched http-server jar seems to work as well.

[glassfishrobot]

@rlubke said: fcd3cbad193367698539ab1bdac429481bebe55c

[glassfishrobot]

This issue was imported from java.net JIRA GRIZZLY-1891

[glassfishrobot]

Marked as fixed on Tuesday, March 14th 2017, 10:30:36 am