Wildfly: IllegalStateException running Async

[lRawd]

Getting the following error in Wildfly 10.1 when I run in with <async-supported>true</async-supported>

16:33:15,634 ERROR [stderr] (default task-233) Exception in thread "default task-233" java.lang.RuntimeException: java.lang.IllegalStateException: WFLYCLWEBUT0001: Session kTZwtiijfbNrhkJVnugUEPgZWSdz_2t22yVzvG9O is invalid
16:33:15,635 ERROR [stderr] (default task-233)  at io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:898)
16:33:15,635 ERROR [stderr] (default task-233)  at io.undertow.servlet.spec.AsyncContextImpl$4.run(AsyncContextImpl.java:333)
16:33:15,635 ERROR [stderr] (default task-233)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
16:33:15,636 ERROR [stderr] (default task-233)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
16:33:15,636 ERROR [stderr] (default task-233)  at java.lang.Thread.run(Thread.java:748)
16:33:15,636 ERROR [stderr] (default task-233) Caused by: java.lang.IllegalStateException: WFLYCLWEBUT0001: Session kTZwtiijfbNrhkJVnugUEPgZWSdz_2t22yVzvG9O is invalid
16:33:15,636 ERROR [stderr] (default task-233)  at org.wildfly.clustering.web.undertow.session.DistributableSession.validate(DistributableSession.java:55)
16:33:15,636 ERROR [stderr] (default task-233)  at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:142)
16:33:15,636 ERROR [stderr] (default task-233)  at io.undertow.servlet.spec.HttpSessionImpl.getAttribute(HttpSessionImpl.java:122)
16:33:15,636 ERROR [stderr] (default task-233)  at org.directwebremoting.dwrp.BaseDwrpHandler.updateRegisteredDwrSession(BaseDwrpHandler.java:161)
16:33:15,636 ERROR [stderr] (default task-233)  at org.directwebremoting.dwrp.BaseDwrpHandler.updateCsrfState(BaseDwrpHandler.java:150)
16:33:15,637 ERROR [stderr] (default task-233)  at org.directwebremoting.dwrp.BasePollHandler$2.run(BasePollHandler.java:209)
16:33:15,637 ERROR [stderr] (default task-233)  at org.directwebremoting.impl.BaseSleeper.doClose(BaseSleeper.java:227)
16:33:15,637 ERROR [stderr] (default task-233)  at org.directwebremoting.impl.BaseSleeper.doWork(BaseSleeper.java:148)
16:33:15,637 ERROR [stderr] (default task-233)  at org.directwebremoting.server.servlet3.Servlet30Sleeper.resumeWork(Servlet30Sleeper.java:97)
16:33:15,638 ERROR [stderr] (default task-233)  at org.directwebremoting.server.servlet3.Servlet30Sleeper.access$000(Servlet30Sleeper.java:32)
16:33:15,638 ERROR [stderr] (default task-233)  at org.directwebremoting.server.servlet3.Servlet30Sleeper$2.run(Servlet30Sleeper.java:83)
16:33:15,638 ERROR [stderr] (default task-233)  at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:83)
16:33:15,638 ERROR [stderr] (default task-233)  at io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:154)
16:33:15,638 ERROR [stderr] (default task-233)  at io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:151)
16:33:15,638 ERROR [stderr] (default task-233)  at io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:896)
16:33:15,638 ERROR [stderr] (default task-233)  ... 4 more

[mikewse]

Thanks for your report! It looks like more than async could be at play here. Are you using clustering with session replication as well? We really need more to go on to find any potential bug in DWR. The best is if you can supply a minimal application that reproduces the problem. If needed, also supply any appserver configuration you do that differs from the default install.

[lRawd]

We are using the standalone-ha.xml configuration that comes with Wildfly 10.1.

Aside from datasources and security settings, the locking and transaction settings were changed in the infinispan section:

<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
    <transport lock-timeout="60000"/>
    <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
        <locking isolation="READ_COMMITTED"/>
        <transaction locking="OPTIMISTIC" mode="BATCH"/>
        <file-store/>
    </distributed-cache>
    <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">
        <file-store/>
    </distributed-cache>
</cache-container>

[mikewse]

Documentation on Wildfly clustering is very scarce but it looks like session failover is on by default. DWR doesn't support session migration/failover. You need to configure some form of sticky sessions if you are using a cluster with multiple servlet containers. My guess is that your error above is caused by cluster configuration and not by async-supported.

DWR is validated with async-supported against Tomcat and Jetty. If you believe there is in fact a problem with DWR, async and Wildfly, then please try the following:

• download the DWR demo app (dwrdemo.war) and update its web.xml with a Servlet 3 descriptor and async-supported=true
• deploy on a non-clustered Wildfly instance
• try out the Reverse Ajax samples, f ex Clock Demo

If you can reproduce an error in this scenario then please post stacktraces and other details here, and we will continue looking at it.

[lRawd]

With standalone-ha.xml, I believe clustered sessions should be enabled via infinispan. However, I am only using a single node/server in development where I see this issue occurring, so in effect sticky sessions are 'enabled'.

This issue appears after the users has selected to log out, and the server invalidates the session. Shortly after the error is logged when DWR attempts to access that session async.

I will try to replicate the issue via your demo app.

[mikewse]

What you describe about logging out / invalidating the session opens my eyes about another interpretation of your stacktrace. I'll look a bit more and get back to you!

[mikewse]

There was a weak spot in the code with regard to async invalidation of sessions which I have now checked in a fix for. Sorry about not seeing this sooner during the discussion! Please try this out and let us know if it fixes your problem.

[lRawd]

Unfortunately Wildfly still returns true in this case and the exception is still thrown. Seems like a bug in Wildfly to me. I see a note in their code that they need to start checking session.isValid, which is what is returning false in this case and throwing the IllegalStateException.

This hack seems to work:

try {
  sess.getCreationTime();
} catch (IllegalStateException ise) {
  return;
}

[mikewse]

Yeah I can see there has been some activity on https://issues.jboss.org/browse/UNDERTOW-478 that seems to be fixed in newer versions. I am suspecting though that there are more things in this area that are shaky in Wildfly/Undertow as, AFAIK, they shouldn't be giving us an invalidated session in the first place - the getSession(false) call should return null if the session isn't valid. But there are many small nuances to this field and the servlet spec certainly isn't clear on all of them. I have changed the code to instead catch the exception thrown from misbehaving containers, and branch handling from there. Let me know how it works!

[lRawd]

Thanks for your help. This issue has been resolved with your latest commit.

[mikewse]

Great, many thanks for your efforts in reporting this and trying out fixes!