reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response，springboot2.1.6

[kinsey-jian]

springboot2.1.6.RELEASE and Spring Cloud Greenwich.SR1

application.yml

spring: application: name: gateway-service cloud: gateway: default-filters:

• StripPrefix=1 routes:
• id: provider_api_route uri: lb://provider-service predicates:
  • Path=/web/** discovery: locator: enabled: true consul: enabled: true discovery: instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}

    prefer-ip-address: true

    retry: initial-interval: 10 multiplier: 1 max-interval: 30000 max-attempts: 15

my pom.xml

`

org.springframework.boot spring-boot-starter-parent 2.1.6.RELEASE

<groupId>com.example</groupId>
<artifactId>demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>demo</name>
<description>Demo project for Spring Boot</description>

<properties>
    <java.version>1.8</java.version>
    <spring-cloud.version>Greenwich.SR1</spring-cloud.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-gateway</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-consul-discovery</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

`

[spencergibb]

Please try with Greenwich.SR2

[kinsey-jian]

@spencergibb Now I have changed the version to Greenwich.SR2，but it is still the same error。

my pom.xml

Dependencies

[spencergibb]

Please, don't use screenshots.

How do we recreate the problem?

[kinsey-jian]

@spencergibb I'm so sorry

this is error

2019-07-04 09:51:02.955 ERROR 1276 --- [ctor-http-nio-2] a.w.r.e.AbstractErrorWebExceptionHandler : [ee3c8217] 500 Server Error for HTTP GET "/web/provider/test"

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

2019-07-04 09:51:02.955 ERROR 1276 --- [tor-http-nio-11] a.w.r.e.AbstractErrorWebExceptionHandler : [64f0c49b] 500 Server Error for HTTP GET "/web/provider/test"

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

2019-07-04 09:51:07.772 WARN 1276 --- [tor-http-nio-10] r.netty.http.client.HttpClientConnect : [id: 0xa7c1cd87, L:0.0.0.0/0.0.0.0:52844 ! R:/172.16.61.149:9010] The connection observed an error

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

this is application.yml

spring: application: name: gateway-service cloud: gateway: default-filters:

• StripPrefix=1 routes:
• id: provider_api_route uri: lb://provider-service predicates:
  • Path=/web/** discovery: locator: enabled: true consul: enabled: true discovery: instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}

    prefer-ip-address: true

    retry: initial-interval: 10 multiplier: 1 max-interval: 30000 max-attempts: 15

this is my pom.xml `

org.springframework.boot spring-boot-starter-parent 2.1.6.RELEASE

<groupId>com.example</groupId>
<artifactId>demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>demo</name>
<description>Demo project for Spring Boot</description>

<properties>
    <java.version>1.8</java.version>
    <spring-cloud.version>Greenwich.SR2</spring-cloud.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-gateway</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-consul-discovery</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

`

[spencergibb]

How do we recreate the problem?

We need step by step instructions and maybe a project that recreates the problem (not pasted code).

[kinsey-jian]

@spencergibb ok

this is gateway project https://github.com/kinsey-jian/gateway.git

this is projecthttps://github.com/kinsey-jian/cloud-k8s.git provider

you should install consul Launch provider-service and gateway, respectively

In the gateway projiect you can use wrk -t4 -c1024 -d6s -T5 --script=./wrk.lua --latency http://localhost:8080/web/provider/test command

[Ziemowit]

Hey @kinsey-jian. Did not check very deeply the description. I just remember that we had very similar problem with "Connection prematurely closed BEFORE response".

It was caused by Apache through which traffic was flowing. Apache have reached the limits and was unable to handle next connection. But have no idea if your gateway redirects to services which are hidden behind the Apache.

[spiritme1984]

Any update? I also have this issue with my project.

[Ziemowit]

Any update? I also have this issue with my project.

Since 0.9.0.RELEASE of reactor netty there is a possibility to configure the pooled connection maxIdleTime.

Netty ConnectionProvider.class:

@param maxIdleTime the {@link Duration} after which the channel will be closed (resolution: ms), if {@code NULL} there is no max idle time

Without it as I understand connections on Netty site were not closed but kept with no limits. In our case the connection is kept with no limits whereas after 2 seconds Apache on server site closes the connection if not used. It leads to

Connection prematurely closed BEFORE response

when after 2 seconds not used connection is trying to be used again by Gateway. At least it is my assumption after our investigation.

I have added commit which allows to define such maxIdleTime here https://github.com/spring-cloud/spring-cloud-gateway/pull/1411 but it is "waiting for triage".

[spiritme1984]

Any update? I also have this issue with my project.

Since 0.9.0.RELEASE of reactor netty there is a possibility to configure the pooled connection maxIdleTime.

Netty ConnectionProvider.class:

@param maxIdleTime the {@link Duration} after which the channel will be closed (resolution: ms), if {@code NULL} there is no max idle time

Without it as I understand connections on Netty site were not closed but kept with no limits. In our case the connection is kept with no limits whereas after 2 seconds Apache on server site closes the connection if not used. It leads to

Connection prematurely closed BEFORE response

when after 2 seconds not used connection is trying to be used again by Gateway. At least it is my assumption after our investigation.

I have added commit which allows to define such maxIdleTime here #1411 but it is "waiting for triage".

Thanks @Ziemowit. As per your reply, I set my tomcat server never close the connection like this: <Connector port="8080" URIEncoding="utf-8" protocol="org.apache.coyote.http11.Http11NioProtocol" maxThreads="200" connectionTimeout="-1" redirectPort="8443" /> But this issue doesn't go away... That makes me very confused. Did I miss something?

[Ziemowit]

I saw for sure that maxIdleTime is missing for spring-gateway and it may cause the problems.

Also do not know if it is your case but AFAIK on Apache level (so before Tomcat) there are also configuration which defines how long Apache keeps the connection open.

It is our case because for us all traffic goes additionally via Apache. If it does not help then sorry. Had hope that manage to help.

Here we are waiting for approve our commit and ship it with HOXTON release. If it does not help we will be searching further as it impacts us too :/

[spencergibb]

Tomcat properties have no impact since gateway uses Netty

[ifrozenice]

@spencergibb It seems to be a reactor netty problem. Please check the reactor-netty issue:Connection Closed Prematurely #413 . how to avoid this problem in gateway?

[JianJang2017]

@ifrozenice do you have some solution about this issue?

[sgf1205]

reactor netty problem. Please check the reactor-ne you can try it:

@Bean
public HttpClient getHttpClient(){
ConnectionProvider connectionProvider= ConnectionProvider.elastic("gw connection pool(60s time out)", Duration.ofSeconds(60l));
return HttpClient.create(connectionProvider);
}

i argreen with @Ziemowit , but spring gateway current GA (2.2.1) can't config maxIdleTime in yml, i find the next verion can do it .so you can config your maxIdleTime in code as above.

[spencergibb]

Closed via 069f24d568678d3698bd9ac3a8ea7d8a1c0f1943

[creaton60]

@spencergibb

Hello, I'm still having this problem.

This is my release version

SpringCloud : Hoxton.SR3
reactor-netty : 0.9.5 RELEASE
reactor-core : 3.3.3 RELEASE

This is my routeLocator

@Bean
    public RouteLocator customRouteLocator(RouteLocatorBuilder builder, AuthFilter authFilter, AccessFilter accessFilter, ActuatorFilter actuatorFilter) {
        return builder.routes()
                .route("comments.api", p -> p
                        .host(host.get("api"))
                        .and().path("/comments/**")
                        .filters(f -> f.rewritePath("^/comments/(?<path>.+)", "/comments/${path}")
                                .filter(authFilter))
                        .uri(uri.get("comment")))
                .route("op.api", p -> p
                        .host(host.get("api"))
                        .and().path("/op/**")
                        .filters(f -> f.rewritePath("^/op/(?<path>.+)", "/op/${path}")
                                .filter(authFilter))
                        .uri(uri.get("op")))
                .route("campaign.api", p -> p
                        .host(host.get("api"))
                        .and().path("/campaigns/**")
                        .filters(f -> f.rewritePath("^/campaigns/(?<path>.+)", "/campaigns/${path}")
                                .filter(authFilter))
                        .uri(uri.get("campaign")))
                .route("trevi.apply.api", p -> p
                        .host(host.get("api"))
                        .and().path("/ad/trevi/apply", "/ad/trevi/notify/installed")
                        .filters(f -> f.rewritePath("^/ad/trevi/(?<path>.+)", "/${path}")
                        .hystrix(c -> c.setName("trevi-apply-cmd").setFallbackUri("forward:/fallback/trevi")))
                        .uri(uri.get("trevi.apply")))
                .route("trevi.track.api", p -> p
                        .host(host.get("api"))
                        .and().path("/ad/trevi/track")
                        .filters(f -> f.rewritePath("^/ad/trevi/(?<path>.+)", "/${path}")
                        .hystrix(c -> c.setName("trevi-track-cmd").setFallbackUri("forward:/fallback/trevi")))
                        .uri(uri.get("trevi.track")))
                .route("trevi.imp.api", p -> p
                        .host(host.get("api"))
                        .and().path("/ad/trevi/reward/**")
                        .filters(f -> f.rewritePath("^/ad/trevi/reward/(?<path>.+)", "/reward/${path}")
                        .hystrix(c -> c.setName("trevi-imp-cmd").setFallbackUri("forward:/fallback/trevi")))
                        .uri(uri.get("trevi.imp")))
                .route("iapi.api", p -> p
                        .host(host.get("api"))
                        .and().path("/ad/coupon-products/**")
                        .filters(f -> f.addRequestHeader("Service", "trevi")
                                .rewritePath("^/ad/coupon-products(?<segment>.*)", "/vp/coupon-products${segment}")
                        .filter(accessFilter)
                        .hystrix(c -> c.setName("coupon-api-cmd").setFallbackUri("forward:/fallback/coupon")))
                        .uri(uri.get("legacy-iapi")))
                .route("iapi.dev.api", p -> p
                        .host(host.get("api"))
                        .and().path("/dev/ad/coupon-products/**")
                        .filters(f -> f.addRequestHeader("Service", "trevi")
                                .rewritePath("^/dev/ad/coupon-products(?<segment>.*)", "/vp/coupon-products${segment}")
                                .filter(accessFilter))
                        .uri(uri.get("legacy-dev-iapi")))
                .route("api", p -> p
                        .order(999)
                        .host(host.get("api"))
                        .filters(f -> f.addRequestHeader("Service", "legacy-api")
                        .hystrix(c -> c.setName("legacy-cmd").setFallbackUri("forward:/fallback/legacy")))
                        .uri(uri.get("legacy-api")))
                .build();
    }

This is my application.yaml

spring:
  profiles: sandbox
  application:
    name: webtoon-gateway-sandbox
  cloud:
    gateway:
      globalcors:
        cors-configurations:
          '[/**]':
            allowed-origins:
              - "*"
            allowed-methods:
              - "*"
            allowed-headers:
              - "*"
            allow-credentials: true
      httpclient:
        pool:
          type: fixed
          max-connections: 500
          acquire-timeout: 75000
  boot:
    admin:
      client:
        url: http://webtoon-cloud-admin.dev.daum.net
        instance:
          prefer-ip: true

hystrix:
  threadpool:
    default:
      coreSize: 128
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMillseconds: 5000
      circuitBreaker:
        requestVolumeThreshold: 20
        errorThresholdPercentage: 50
        sleepWindowInMilliseconds: 5000
        enabled: true

Is there anything I missed?

[spencergibb]

Please open a new issue and provide the stack trace

[Ziemowit]

@creaton60 have you tried to set the maxIdleTime property?

spring:
    gateway:
      httpclient:
        pool:
          max-idle-time: <PUT_YOUR_VALUE_HERE>ms

AFAIR by default, it is NULL so your connection will be waiting forever without closing it on the gateway and it may be closed on server site.

[creaton60]

@Ziemowit Thanks I'll try it.

[iceqing]

spring.cloud.gateway.httpclient.pool.max-idle-time: Time in millis after which the channel will be closed. If NULL, there is no max.

server.connection-timeout or connectionTimeout: The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any). keepAliveTimeout: The number of milliseconds this Connector will wait for another HTTP request before closing the connection. The default value is to use the value that has been set for the connectionTimeout attribute. Use a value of -1 to indicate no (i.e. infinite) timeout.

@Ziemowit Thank you very mutch. It solved my problem. The Spring Cloud Gateway project spring.cloud.gateway.httpclient.pool.max-idle-time defalut value was null (there is no max). my proxy target server was tomcat, server.connection-timeout default was 20000ms. My server was set to 100ms, 100ms was to short, it cause many PrematureCloseException.

My solution:

server.connection-timeout=5000
spring.cloud.gateway.httpclient.pool.max-idle-time=2000ms

Notice: spring.cloud.gateway.httpclient.pool.max-idle-time must be less than or server.connection-timeout

https://github.com/reactor/reactor-netty/issues/1092

[iceqing]

@spencergibb Can we set a default value? https://github.com/spring-cloud/spring-cloud-gateway/pull/1787

This is an example project that reproduce the problem of PrematureCloseException: https://github.com/iceqing/spring-cloud-gateway-example

[Lovnx]

https://blog.csdn.net/rickiyeat/article/details/107900585

[spencergibb]

Please stop posting the same thing in multiple issues