AsyncListener.onError(...) not called if IOException occurs while calling HttpOutput.flush()

[peng12341]

I'm using the Spring SseEmitter to stream events to my client. If the client closes the connection and I send the next event, the following exception occurs (correctly):

org.eclipse.jetty.io.EofException: null
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:285) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:808) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:841) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:891) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:398) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.flush(OnCommittedResponseWrapper.java:514) [spring-security-web-5.1.2.RELEASE.jar:5.1.2.RELEASE]
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1100) [jackson-core-2.9.7.jar:2.9.7]
    at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:915) [jackson-databind-2.9.7.jar:2.9.7]
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287) [spring-web-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.http.converter.AbstractGenericHttpMessageConverter.writeInternal(AbstractGenericHttpMessageConverter.java:112) [spring-web-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227) [spring-web-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$HttpMessageConvertingHandler.sendInternal(ResponseBodyEmitterReturnValueHandler.java:191) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$HttpMessageConvertingHandler.send(ResponseBodyEmitterReturnValueHandler.java:184) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.sendInternal(ResponseBodyEmitter.java:189) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:183) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.SseEmitter.send(SseEmitter.java:133) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]

According to the following Spring issues the AsyncListener.onError-Method should also be called with that exception, but is not. (And therefore the SseEmitter never does its cleanup)

https://github.com/spring-projects/spring-framework/issues/21091 https://github.com/spring-projects/spring-framework/issues/20173

[gregw]

@peng12341 I don't agree. I can't see anything in the servlet spec that suggests that AsyncListener#onError should be called for a failure in an async write operation. Failures in async write operations are reported to WriteListener.onError not the AsyncListener. In fact the AsyncListener#onError interface predates the async IO API in servlets and is unrelated. Specifically it used to report failures associated with AsyncContext operations.

Does your WriteListener.onError get called?

[peng12341]

@gregw Thanks for your quick response. As far as I can see, spring does not register a WriteListener. Spring creates an AsyncContext and then registers the AsyncListener, so the write operation is associated with the AsyncContext.

Here is the relevant code in Spring that creates the AsyncContext: https://github.com/spring-projects/spring-framework/blob/155ef5fd778579c9bae3469244a3aee64f079ac5/spring-web/src/main/java/org/springframework/web/context/request/async/StandardServletAsyncWebRequest.java#L122

From the Javadoc of AsyncListener.onError:

Notifies this AsyncListener that an asynchronous operation has failed to complete. 

Isn't writing to the response from another thread an asynchronous operation?

[stale[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been closed due to it having no activity.

[hudac]

Just stumbled upon this issue and is still valid in Jetty v12.0.5.

In case of a client side disconnect (EofException) while sending SseEmitter events Jetty does not notify AsyncListener#onError as expected by Spring. This leaves DeferredResult objects unnecessarily dangling and eventually causing timeouts. Either you need to keep the AsyncContext#timeout quite short, which forces the clients to reconnect more often, or risk resource leak when having a too large timeout.

Not sure if this is an issue in Spring (tested on Spring v6.1.2 / Boot v3.2.1), however, switching back to Tomcat or Undertow solves the issue, AsyncListener#onError is immediately notified when an error occurs during an async send and all related resources are cleaned up.

[sbordet]

@gregw I reopened this for further investigation. Has anything changed or be clarified in Servlet 6 about this? Should we raise the issue with the spec?

[eberleant]

I think I'm running into the same issue in Jetty 12.0.6. From ResponseBodyEmitter docs:

Note: if the send fails with an IOException, you do not need to call completeWithError(Throwable) in order to clean up. Instead the Servlet container creates a notification that results in a dispatch where Spring MVC invokes exception resolvers and completes processing.

Here are some relevant comments on Spring Framework issues -

Spring Framework issue 30867:

As this Javadoc note says, when there is an IO error, the Servlet container is aware of the exception, and calls AsyncListener#onError. In that case we need to wait for that notification rather than complete immediately. More details on that in https://github.com/spring-projects/spring-framework/issues/20173.

Spring Framework issue 27252 - the same issue in Undertow was labelled an Undertow bug:

When emitter.send(i) is called after the client has disconnected, it fails with an IOException due to a broken pipe. This causes SseEmitter to set its sendFailed flag to true. The purpose of this flag is described by its javadoc:

After an I/O error, we don't call completeWithError directly but wait for the Servlet container to call us via AsyncListener#onError on a container thread at which point we call completeWithError. This flag is used to ignore further calls to complete or completeWithError that may come for example from an application try-catch block on the thread of the I/O error.

When you're using Undertow, the expected call to AsyncListener#onError never comes and, instead, the async request eventually times out. If you use Tomcat, the onError call does occur and your onError handler is called as expected.

This looks like an Undertow bug to me but we'll transfer the issue to the Framework team just in case there's anything that they can do.

When an IOException happens, the SseEmitter.ioErrorOnSend flag is set to true, which causes calls to SseEmitter.complete() and SseEmitter.completeWithError() to do nothing: https://github.com/spring-projects/spring-framework/blob/v6.1.3/spring-webmvc/src/main/java/org/springframework/web/servlet/mvc/method/annotation/ResponseBodyEmitter.java#L257-L266

However, maybe this is all moot, as https://github.com/spring-projects/spring-framework/issues/32629 was just closed with commit https://github.com/spring-projects/spring-framework/commit/c6b6ccdc890f980ac910f0ea01b828eb6720a15b (which removes the ioErrorOnSend flag).

[kshep92]

I'm running into the same issue on 12.0.10 #12008 In addition to my async listener not being called, the IOException that results from the client disconnecting isn't thrown to my catch block for handling it. Instead it seems that it is being caught internally by Jetty and logged, but not thrown.

[gregw]

@kshep92 I think this is perhaps a different issue to what you are seeing in #12008, as you are using an async servlet, but with blocking IO. So exceptions are reported by exceptions thrown from write and flush calls. This is an issue about if those same failure events should also be reported to an AsyncListener.

So let's keep looking at your specific example in #12008 separately unless we find they really are the same.

@sbordet Let's work on a reproducer that can be deployed on other containers as well so we can pin down exactly what the different behaviour is.... or see if there is actually a bug which only triggers in specific circumstances (this could be #12008).

[kshep92]

@gregw I agree that they are two separate issues, however, in my investigation of #12008 I found that Tomcat reported similar errors to the AsyncListener:

Output from Jetty:

Catch block caught exception: An established connection was aborted by the software in your host machine

Output from Tomcat:

Catch block caught exception: An established connection was aborted by the software in your host machine
AsyncListener.onError caught an exception: An established connection was aborted by the software in your host machine

Maybe it's worth exploring for the sake of consistency?