Open michaldo opened 4 years ago
This is seriously very weird behaviour. We have spring cloud gateway in production environment and for clients which send Expect: 100-continue
are not getting proper response.
I don't know how is this possible but during peak times the response for the route is also wrong.
For ex : Two routes A and B and they both call expecting 100. Route A gets 100 status code and the the payload response is sent to route B.
I believe this is blocked by the reactor netty issue listed above.
@spencergibb Reactor Netty will respond automatically with 100 Continue
ONLY when the IO handler asks for the body but there was no response with 100 Continue
before that.
In Spring Gateway I see the following:
In the implementation the incoming data is requested always without checking whether there is Expect
header
To document, request.getBody()
is called in webflux initialization very early on before gateway can respond to any header here https://github.com/spring-projects/spring-framework/blob/master/spring-web/src/main/java/org/springframework/web/server/adapter/DefaultServerWebExchange.java#L131
So it still looks blocked but at the framework level now @rstoyanchev
That is not an actual call but deferred initialization in case of a call to exchange.getFormData()
. To trigger that something must call getFormData()
.
I'm able to send a 100 Continue
from gateway (or let the downstream service send it), but it seems the request is terminated and the client never sends the body. I'm not sure where in the stack the problem is.
Is there any other work around in springboot to handle the Expect: 100-continue header.
Is there any other work around in springboot to handle the Expect: 100-continue header.
From issue description:
Worth to notice that workaround is filter our header 'Expect 100-continue` when gateway forwards request to target server
So you can simply add route filter that removes Expect
header
.filters(f -> f.removeRequestHeader("Expect"))
or
filters:
- RemoveRequestHeader=Expect
@spencergibb gateway sends 100 Continue, but also for the final response. I have Wiremock configured to return 201. In case of direct call to Wiremock:
$ curl localhost:8082/huge-request -X POST -H 'Content-Type: text/plain' --data "@10KB.txt" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8082 (#0)
> POST /huge-request HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: text/plain
> Content-Length: 10240
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Matched-Stub-Id: 9cd25804-832d-45bf-9b25-91990ac9c4c5
< Matched-Stub-Name: for testing
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked
< Server: Jetty(9.4.20.v20190813)
<
* Connection #0 to host localhost left intact
* Closing connection 0
In case of call through gateway, two 100 Continue are returned and connection timeouts:
$ curl localhost:8080/huge-request -X POST -H 'Content-Type: text/plain' --data "@10KB.txt" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /huge-request HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: text/plain
> Content-Length: 10240
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 100 Continue
I just updated spring-boot to version 2.4.3 which depends on reactor-netty version 1.0.4 and netty version 4.1.59 The RemoveRequestHeader work around does not work any more. I tried to create a gatewayFilter that sets response status to 100 and complete the response but I get:
Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:137)
That is strange, because my experiment shows that spring-boot version 2.4.3 & spring cloud version 2020.0.1 fixes the problem
The problem is still visible for spring-boot version 2.3.9.RELEASE and spring cloud Hoxton.SR9
I vote for close the problem
I reckon I haven't given enough details about what I'm trying to do.
We have several micro-services behind a spring-cloud-gateway which is accessed by mobile games.
We want the gateway to handle Expect 100-continue headers and respond with 100.
We don't want the gateway to pass the request to downstream services in this case.
Before spring-boot 2.4.3, we added RemoveRequestHeader and the gateway worked as required.
Since upgrading to spring-boot 2.4.3 it passes the request downstream which we don't want it to do.
I tried to implement a gateway filter responsible to intercept Expect 100-continue requests, set the response to 100 and complete the response.
I can't make it work; I alwys the following exception:
Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:137)
@paskos I'm going to say your issue is different than the original intent of this issue. Can you post your filter?
ExpectContinueGatewayFilterFactory I changed the package but the code is exactly the same.
I configured it in the gateway's default filters:
spring:
cloud:
gateway:
default-filters:
- ExpectContinue
I found the changes in behaviour thanks to an integration test in my gateway that uses wiremock to simulate a downtstream service. WebTestClient sends a http request with the Expect 100-continue header and then the tests calls wiremock verify() to make sure that no requests were received by it.
When my gateway depends on spring-boot version 2.4.2 and spring-cloud 2020.0.1 my test passes. When I upgrade to spring-boot version 2.4.3 the test fails because wiremock did receive the request.
This might be related to this fix https://github.com/reactor/reactor-netty/pull/1492
Thank you for the link. @violetagg @spencergibb I'm far from proficient with Netty, spring-cloud-gateway and reactor or reactive programming. Could you, please, point me in the direction on how to implement Expect 100-continue at the gateway level where the request is not sent downstream ? Thanks a lot in advance
Currently we are working on a project using Spring Cloud Gateway with dynamic routing and custom load balancing.
We faced the same issue and after some investigation we realized that as michalod said in this comment: https://github.com/spring-cloud/spring-cloud-gateway/issues/1337#issuecomment-790928638
The 100-continue feature works perfectly in Spring Cloud Gateway with the latest version of related dependencies. (for us: spring-cloud-starter-gateway v3.0.4 spring-cloud-gateway-webflux v3.0.4 spring-boot v2.5.5)
So I think this issue can be closed.
And a late response to paskos:
We want the gateway to handle Expect 100-continue headers and respond with 100. We don't want the gateway to pass the request to downstream services in this case.
This is wrong behaviour because if the mobile game requests your gateway with Expect: 100-continue header without Body, then it keeps alive the connection and waits for a response. If the status of the response is 100 Continue, then the mobile game tries to send the rest of the request (actually the Body) in the same TCP connection then waits for a final response (for example: 200 Ok). If the gateway do not pass the request, then the mobile game will hang until timeout.
If you would like to reject the requests from mobile games which contains Expect: 100-continue header then in accordance with the RFC standards your gateway has to respond 417 Expectation Failed. If the mobile game implemented the RFC properly then it will send the full request again without Expect header.
Here is the code you can use in your custom ExpectContinueGatewayFilterFactory if you want to implement this way.
String expectHeader = exchange.getRequest().getHeaders().get("Expect").get(0);
if (expectHeader != null && "100-continue".equals(expectHeader)) {
exchange.getResponse().setStatusCode(EXPECTATION_FAILED); // Reject with Status Code 417
return exchange.getResponse().setComplete();
}
this code is also a workaround for Spring Cloud Gateway apps using spring-boot version before 2.4.3 if Client hangs in the 100-continue flow.
I reckon I haven't given enough details about what I'm trying to do. We have several micro-services behind a spring-cloud-gateway which is accessed by mobile games. We want the gateway to handle Expect 100-continue headers and respond with 100. We don't want the gateway to pass the request to downstream services in this case. Before spring-boot 2.4.3, we added RemoveRequestHeader and the gateway worked as required. Since upgrading to spring-boot 2.4.3 it passes the request downstream which we don't want it to do. I tried to implement a gateway filter responsible to intercept Expect 100-continue requests, set the response to 100 and complete the response. I can't make it work; I alwys the following exception:
Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:137)
This clearly violates https://datatracker.ietf.org/doc/html/rfc7231#section-5.1.1. If the proxy cannot handle the expect handle it must ask the origin. So I expect the API gateway to forward all request headers to the origin server if it cannot make a decision based on the headers.
Steps to reproduce problem
Target server: Spring Boot 2.1.8 (Tomcat)
Cloud gateway: Greenwich SR3
When target server is called directly:
curl localhost:8080/put -X PUT -H 'Expect: 100-continue' -d x=foo1 -v
communication flow is OK
Expect: 100-continue
100-continue
hello
When target server is called through gateway:
curl localhost:8082/put -X PUT -H 'Expect: 100-continue' -d x=foo1 -v
communication flow is not OK. My wireshark investigation
Expect: 100-continue
to gateway100-continue
100-continue
to gatewayhello
100-continue
to clienthello
but it is not forwardedResponse
100 Continue
is received 2 times - once generated by gateway itself, second forwarded from target server, but true response never reach client.Worth to notice that workaround is filter our header 'Expect 100-continue` when gateway forwards request to target server