search

OpenLiberty / open-liberty

Open Liberty is a highly composable, fast to start, dynamic application server runtime environment
https://openliberty.io
Eclipse Public License 2.0
1.14k stars 587 forks source link

[ERROR ] Spring Boot application spits SRVE0777E (stream is closed) after every request in logs. #15647

Open Ajz316 opened 3 years ago

Ajz316 commented 3 years ago

I have deployed two Spring Boot Apps and a few RESTful services in them. Irrespective of a GET or POST I am consistently getting this error in log. When run as FAT JAR (embedded Tomcat), this issue does not occur.

[ERROR   ] SRVE0777E: Exception thrown by application class 'com.ibm.ws.http.channel.internal.outbound.HttpOutputStreamImpl.validate:223'
java.io.IOException: Stream is closed
    at com.ibm.ws.http.channel.internal.outbound.HttpOutputStreamImpl.validate(HttpOutputStreamImpl.java:223)
    at [internal classes]
    at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:277)
    at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219)
    at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:123)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:686)
    at [internal classes]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
    at [internal classes]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at [internal classes]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at [internal classes]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at [internal classes]
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at [internal classes]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at [internal classes]

[INFO    ] FFDC1015I: An FFDC Incident has been created: "java.io.IOException: Stream is closed com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest() 830" at ffdc_21.01.25_11.30.28.0.log
[INFO    ] FFDC1015I: An FFDC Incident has been created: "com.ibm.ws.webcontainer.webapp.WebAppErrorReport: java.io.IOException: Stream is closed com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter 82" at ffdc_21.01.25_11.30.28.1.log
[INFO    ] FFDC1015I: An FFDC Incident has been created: "com.ibm.ws.webcontainer.webapp.WebAppErrorReport: java.io.IOException: Stream is closed com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter 144" at ffdc_21.01.25_11.30.28.2.log
2021-01-25 11:30:28.658 ERROR 16704 --- [ecutor-thread-1] o.s.b.w.servlet.support.ErrorPageFilter  : Cannot forward to error page for request [/tasks/v1/retrieve] as the response has already been committed. As a result, the response may have the wrong status code. If your application is running on WebSphere Application Server you may be able to resolve this problem by setting com.ibm.ws.webcontainer.invokeFlushAfterService to false

com.ibm.ws.webcontainer.webapp.WebAppErrorReport: java.io.IOException: Stream is closed
    at com.ibm.ws.http.channel.internal.outbound.HttpOutputStreamImpl.validate(HttpOutputStreamImpl.java:223)
    at com.ibm.ws.http.channel.internal.outbound.HttpOutputStreamImpl.flush(HttpOutputStreamImpl.java:648)
    at com.ibm.wsspi.http.ee7.HttpOutputStreamEE7.flush(HttpOutputStreamEE7.java:299)
    at com.ibm.ws.webcontainer.osgi.response.WCOutputStream.flush(WCOutputStream.java:233)
    at com.ibm.ws.webcontainer31.osgi.response.WCOutputStream31.flush(WCOutputStream31.java:598)
    at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:277)
    at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219)
    at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:123)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:686)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:729)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:426)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:182)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1001)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1139)
    at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5057)
    at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:314)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1005)
    at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:279)
    at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1134)
    at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:415)
    at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:374)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:577)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:359)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:326)
    at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:167)
    at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:75)
    at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
    at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
    at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:958)
    at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1047)
    at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Stream is closed
    ... 71 common frames omitted

Steps to Reproduce Write a Spring boot 2.3.5 application. Deploy it in Openliberty (20.0.0.10) as WAR file. Access an endpoint in a client like Postman. The output from the RESTful service is 200 OK and the response is obtained correctly, but in the logs we see the above stack trace for every single request.

I have tried using <webContainer throwExceptionWhenUnableToCompleteOrDispatch="false" com.ibm.ws.webcontainer.invokeflushafterservice="false"></webContainer> but the error still occurs.

Expected behavior The logs should not be printing this ERROR.

Diagnostic information:

<?xml version="1.0" encoding="UTF-8"?>
<server description="Open Liberty Server Configuration">

    <featureManager>
        <feature>servlet-4.0</feature>
        <feature>springBoot-2.0</feature>
        <feature>jndi-1.0</feature>
        <feature>jms-2.0</feature>
        <feature>jdbc-4.1</feature>
    </featureManager>
    <!-- Variables -->
    <variable name="default_http_port" defaultValue="9080" />
    <variable name="default_https_port" defaultValue="9443" />
    <!-- Improved startup times, only CDI scan wars with beans.xml files  -->
    <cdi12 enableImplicitBeanArchives="false" />

    <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="${default_http_port}" httpsPort="${default_https_port}" />

    <applicationManager autoExpand="true" />
    <logging consoleLogLevel="INFO"/>
    <webContainer throwExceptionWhenUnableToCompleteOrDispatch="false" com.ibm.ws.webcontainer.invokeflushafterservice="false"></webContainer>
</server>
anjumfatima90 commented 3 years ago

Hi, I have few questions like

  1. I know that you are deploying the application as a WAR, I want to know where the application is deployed? To dropins/spring or just directly to the dropins folder or as an application tag in serverl.xml using webApplication or springApplication.
  2. I deployed a simple application war but wasn't successful in reproducing the error. Could you please point me to a sample application which gives this error?

Thanks!

Ajz316 commented 3 years ago

I am deploying the WAR Using in dropins folder. The error is logged when you hit an endpoint. My controller returns something like ResponseEntity.ok(aCustomObject) Thats where it prints the error in log.

On Tue, 26 Jan 2021, 07:27 ANJUM FATIMA, notifications@github.com wrote:

Hi, I have few questions like

  1. I know that you are deploying the application as a WAR, I want to know where the application is deployed? To dropins/spring or just directly to the dropins folder or as an application tag in serverl.xml using webApplication or springApplication.
  2. I deployed a simple application war but wasn't successful in reproducing the error. Could you please point me to a sample application which gives this error?

Thanks!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenLiberty/open-liberty/issues/15647#issuecomment-767091861, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2W364H5VLFRM7V5II7R5LS3XH2PANCNFSM4WQ7CZXA .

anjumfatima90 commented 3 years ago

You won't need to set<springBootApplication> in you server.xml if you are placing your application in dropins. This is how you can deploy the application

Deploy an application with either the server.xml application configuration or with the dropins folder as shown in the following options.

You can find more information on https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_dep_springboot.html

Ajz316 commented 3 years ago

Hi Anjum While I appreciate the inputs you have provided for deployments, my problem is coming at runtime. The deployment looks fine to me. Is this something you could help me with?

On Wed, 27 Jan 2021, 03:10 ANJUM FATIMA, notifications@github.com wrote:

You won't need to set in you server.xml if you are placing your application in dropins. This is how you can deploy the application

Deploy an application with either the server.xml application configuration or with the dropins folder as shown in the following options.

-

Define the application with a springBootApplication element or an application element. . or

-

If you choose not to update the server.xml file, use Liberty dropins support instead.

Copy the hellospringboot.jar file to the /usr/servers/helloserver/dropins/spring/hellospringboot.jar directory.

The spring subdirectory within the dropins subdirectory keeps the existing .jar or .war extension.

Use the .spring extension when you copy the application directly to the dropins subdirectory.

You can find more information on [( https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_dep_springboot.html )]

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenLiberty/open-liberty/issues/15647#issuecomment-767649996, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2W363TMXRZ2UJT2E2AIJ3S33SORANCNFSM4WQ7CZXA .

anjumfatima90 commented 3 years ago

I am trying to understand how you are deploying exactly. I need to know the exact folder where you deployed your application and the server.xml config. Also, it would be very helpful if could put up a sample application on github which gives this error, because i wasn't able to reproduce the error. I want to know, what is that in your application which is causing this error to appear.

Ajz316 commented 3 years ago

Ok the application is a bit sensitive. How about I try to use the app you are using locally and see if i xan reproduce steps? Will that be easier as we both will be working on common code base? By the way, just so we are clear, when you say you can't reproduce the error- you have deployed the app and you are accessing a controller's endpoint through postman or something and you are not getting any error is logs?

On Wed, 27 Jan 2021, 07:59 ANJUM FATIMA, notifications@github.com wrote:

I am trying to understand how you are deploying exactly. I need to know the exact folder where you deployed your application and the server.xml config. Also, it would be very helpful if could put up a sample application on github which gives this error, because i wasn't able to reproduce the error. I want to know, what is that in your application which is causing this error to appear.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenLiberty/open-liberty/issues/15647#issuecomment-767823747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2W36ZGM36B5FU5RGRNFH3S34UKDANCNFSM4WQ7CZXA .

anjumfatima90 commented 3 years ago

ok, let me work on the sample application. I will let you know the link as soon as possible.

Ajz316 commented 3 years ago

Thanks but you did not answer the second part of my question. Are you trying to access the end point and you do not see any error log?

On Wed, 27 Jan 2021, 08:25 ANJUM FATIMA, notifications@github.com wrote:

ok, let me work on the sample application. I will let you know the link as soon as possible.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenLiberty/open-liberty/issues/15647#issuecomment-767837426, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2W36ZQAXO7VJRCKQBRGJ3S34XK5ANCNFSM4WQ7CZXA .

anjumfatima90 commented 3 years ago

Yes, I do not see the error when I try to access the end point from browser and the POSTMAN.

Ajz316 commented 3 years ago

Hi Anjum Found this one out https://github.com/openliberty/open-liberty/issues/14875

Seems like it could be an issue with Spring. But is it possible for you to find the version it is fixed in? I have commented on that issue too to check with author if he can provide some inputs.

Thanks Ajit

On Wed, 27 Jan 2021, 08:37 ANJUM FATIMA, notifications@github.com wrote:

Yes, I do not see the error when I try to access the end point from browser and also from POSTMAN.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenLiberty/open-liberty/issues/15647#issuecomment-767843612, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2W362AEXEWEJQGFQ56JQDS34YYBANCNFSM4WQ7CZXA .

anjumfatima90 commented 3 years ago

The issue has been tagged 5.2.11 milestone. May be you could try the next version of spring boot which would be 2.3.6.RELEASE or later.

Ajz316 commented 3 years ago

Yes, it seems it is a Spring issue. I could not try 5.2.11, but tried 5.2.12 and that seems to work fine. No errors in log. Now, that is an option for sure. But I wonder if there is a flag or something in Openliberty that I can use to disable this error message? That would be easier for my team, given that there is a lot of interdependencies of different modules we have.

anjumfatima90 commented 3 years ago

I am not aware of any such flag. Although, could you try this workaround?

In the mean time as a workaround you could disable the JsonGenerator.Feature.AUTO_CLOSE_TARGET on the ObjectMapper. 

For example in Spring Boot:

@Bean
public Jackson2ObjectMapperBuilderCustomizer om() {
    return builder -> builder.featuresToDisable(JsonGenerator.Feature.AUTO_CLOSE_TARGET);
}

This was mentioned in https://github.com/spring-projects/spring-framework/issues/25987#issuecomment-719731353.

The issue was linked as backport in https://github.com/spring-projects/spring-framework/issues/25989