search

spring-cloud / spring-cloud-gateway

An API Gateway built on Spring Framework and Spring Boot providing routing and more.
http://cloud.spring.io
Apache License 2.0
4.55k stars 3.33k forks source link

Document that `SimpleClientHttpRequestFactory` (and `URLConnection` which it uses) is not suitable for the WebMVC gateway server. #3451

Open haeuserd opened 4 months ago

haeuserd commented 4 months ago

Describe the bug

With the changes of https://github.com/spring-cloud/spring-cloud-gateway/issues/3405 (introduced in version 4.1.4) the error handling does not work properly any more when using DefaultRestClient.

When the client returns any error status code (e.g. 400), the DefaultRestClient raises an org.springframework.web.client.ResourceAccessException when trying to read the response body which happens RestClientProxyExchange.

However the ResourceAccessException does not provide any accessible information about the status code other than the message text. Therefore we cannot handle it properly and only respond with 500 Internal Server Error by default.

A workaround is to use a different http client. But actually I would expect Spring Cloud Gateway to work out of the box with Spring's default client. However, you may have other opinions on this.

Sample

I'm about half an hour before my three week holiday starts, so unfortunately I'm not able to provide a reproducable example any more. I can do that when I get back, if that helps. All I can do for now is to provide the error stack trace:

2024-07-11T16:00:53.232+02:00  WARN 249503 --- [api-outages-proxy] [omcat-handler-0] d.e.a.p.e.ErrorEntityExceptionHandler    : I/O error on POST request for "http://localhost:11112/api/outages": Server returned HTTP response code: 400 for URL: http://localhost:11112/api/outages

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:11112/api/outages": Server returned HTTP response code: 400 for URL: http://localhost:11112/api/outages
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.createResourceAccessException(DefaultRestClient.java:575) ~[spring-web-6.1.10.jar:6.1.10]
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:498) ~[spring-web-6.1.10.jar:6.1.10]
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchange(DefaultRestClient.java:465) ~[spring-web-6.1.10.jar:6.1.10]
    at org.springframework.cloud.gateway.server.mvc.handler.RestClientProxyExchange.exchange(RestClientProxyExchange.java:42) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
    at org.springframework.cloud.gateway.server.mvc.handler.ProxyExchangeHandlerFunction.handle(ProxyExchangeHandlerFunction.java:120) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
    at org.springframework.cloud.gateway.server.mvc.handler.HandlerFunctions$LookupProxyExchangeHandlerFunction.handle(HandlerFunctions.java:107) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
jordanjennings commented 4 months ago

We have hit this bug also - we have a backend we expect to be returning a 404, but after upgrading to 4.1.4 that 404 produces a 500 error to the client instead.

Following the flow of the code it looks like HttpURLConnection throws a FileNotFound exception on 404, which due to this line of code is now throwing in a different place than it previously would have, and the exception isn't handled from here correctly:

https://github.com/spring-cloud/spring-cloud-gateway/blame/656ab5aba4a25738e2e7af1a2bb7846667667e3b/spring-cloud-gateway-server-mvc/src/main/java/org/springframework/cloud/gateway/server/mvc/handler/RestClientProxyExchange.java#L51

This is a blocker for upgrading for us, and I hope it can be addressed soon. For now we'll stay on the older version.

haeuserd commented 3 months ago

Here is a minimal reproducable example:

application.properties:

spring.cloud.gateway.mvc.http-client.type=autodetect

Gateway Application:

@SpringBootApplication
public class GatewayDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(GatewayDemoApplication.class, args);
    }

    @Bean
    public RouterFunction<ServerResponse> getRoute() {
        return route().GET("/status/*", http("https://httpbin.org/")).build();
    }
}

Example request resulting in http status code 500 instead of correct response code:

curl -i http://localhost:8080/status/400

Stacktrace:

2024-07-30T14:33:35.442+02:00 ERROR 201951 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://httpbin.org/status/400": Server returned HTTP response code: 400 for URL: https://httpbin.org/status/400] with root cause

java.io.IOException: Server returned HTTP response code: 400 for URL: https://httpbin.org/status/400
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1998) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
    at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
    at org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:88) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:492) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchange(DefaultRestClient.java:465) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.cloud.gateway.server.mvc.handler.RestClientProxyExchange.exchange(RestClientProxyExchange.java:42) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
    at org.springframework.cloud.gateway.server.mvc.handler.ProxyExchangeHandlerFunction.handle(ProxyExchangeHandlerFunction.java:120) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
    at org.springframework.cloud.gateway.server.mvc.handler.HandlerFunctions$LookupProxyExchangeHandlerFunction.handle(HandlerFunctions.java:107) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
    at org.springframework.web.servlet.function.support.HandlerFunctionAdapter.handle(HandlerFunctionAdapter.java:108) ~[spring-webmvc-6.1.11.jar:6.1.11]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.11.jar:6.1.11]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.11.jar:6.1.11]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.11.jar:6.1.11]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.11.jar:6.1.11]
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.11.jar:6.1.11]
    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.springframework.cloud.gateway.server.mvc.filter.WeightCalculatorFilter.doFilter(WeightCalculatorFilter.java:229) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.springframework.cloud.gateway.server.mvc.filter.FormFilter.doFilter(FormFilter.java:93) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.11.jar:6.1.11]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
    at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
mendiCap commented 3 weeks ago

Any update on this? Am stuck here for some time, maybe downgrading to an older version?

spencergibb commented 3 weeks ago

But actually I would expect Spring Cloud Gateway to work out of the box with Spring's default client.

The default client in Spring Cloud Gateway WebMVC is the jdk HttpClient, not HttpURLConnection

mendiCap commented 3 weeks ago

@spencergibb I've forked the latest version of branch 4.1.x to test the changes so that I can continue testing my gateway. However, seems like the issue isn't quite fixed yet. The ResourceAccessException gets thrown after an IO error. Which occurs here:

    try {
            InputStream body = clientResponse.getBody(); // ---------HERE
            // put the body input stream in a request attribute so filters can read it.
            MvcUtils.putAttribute(request.getServerRequest(), MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR, body);
        }
        catch (FileNotFoundException e) {
            // if using SimpleClientHttpRequestFactory
            return ServerResponse.notFound().build();
        }

So the catch doesn't get called. Which causes a ResourceAccessException later and returning the default TomCat error HTML page in the API response.

Maybe the following might help: Before the program executes clientResponse.getBody() I do it manually in the debugger, which causes: First click

When executing it again in the debugger console: It does work second click

I can even parse the error message: Parsed JSON error message

Is there any alternative till this issue is resolved?

spencergibb commented 3 weeks ago

@mendiCap I don't get a ResourceAccessException, I get a FileNotFoundException. Can you tell me how to recreate your specific situation?

spencergibb commented 3 weeks ago

Ah, I'm testing 404 specifically.

mendiCap commented 3 weeks ago

Currently I'm testing a 409.

Do you know of a previous version this would work?

mendiCap commented 3 weeks ago

@spencergibb I managed to fix it with the following added catch method:

    try {
            InputStream body = clientResponse.getBody();
            // put the body input stream in a request attribute so filters can read it.
            MvcUtils.putAttribute(request.getServerRequest(), MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR, body);
        }
        catch (FileNotFoundException e) {
            // if using SimpleClientHttpRequestFactory
            return ServerResponse.notFound().build();
        }
        catch (IOException e) {
            return ServerResponse.status(clientResponse.getStatusCode())
                .body(new String(clientResponse.getBody().readAllBytes(), StandardCharsets.UTF_8));
        }

This returns the correct status code and error body for the downstream backend.

However, I'm not sure it's a good fix because I couldn't use an after-filter, the headers become read-only.

P.s: I used spring-cloud-starter-gateway-mvc, the initial changes for this bug were on spring-cloud-server-gateway-mvc. Is there a difference or should it be changed on both?

spencergibb commented 3 weeks ago

I don't think we can arbitrarily catch IOException here. Other filters, such as circuitbreaker and retry rely on exceptions to function correctly. I'm worried that this is a rabbit hole I don't want to go down and URLConnection is not a very good http client for a proxy. Is there a reason you are not using the jdk http client?

Mendistern commented 3 weeks ago

This is the default config, I haven't explicity used that. How can I use the jdk one?

spencergibb commented 3 weeks ago

@Mendistern it is not the default. The jdk http client is the default unless you set spring.cloud.gateway.mvc.http-client.type=autodetect

mendiCap commented 3 weeks ago

@spencergibb Some context: I've an Angular frontend, Spring Backend, Oauth, and the gateway. The gateway handles the Oauth flow and routes requests to the frontend or backend.

I've removed the autodetect line. But without this, it doesn't forward requests and the browser stays on loading state. Even though the filter was found:

 Predicate "/**" matches against "HTTP GET /my-deposits/new-modification-deposit"
2024-10-30T10:18:22.999+01:00 TRACE 17312 --- [edepot-api-gateway] [nio-7081-exec-6] o.s.w.s.f.support.RouterFunctionMapping  : Mapped to org.springframework.web.servlet.function.HandlerFilterFunction$$Lambda/0x000002331da8ce48@7f60d7ac
2024-10-30T10:18:22.999+01:00 DEBUG 17312 --- [edepot-api-gateway] [nio-7081-exec-6] o.j.s.OpenEntityManagerInViewInterceptor : Opening JPA EntityManager in

I tried setting http-client.type=jdk, but this also doesn't help. I tried manually setting an HttpClient bean:

   @Bean
    public HttpClient httpClient() {
        return HttpClient.newBuilder()
                .version(HttpClient.Version.HTTP_2)
                .build();
    }

Also doesn't work.

Here's my application.properties:

spring.application.name=api-gateway
server.port=${SERVER_PORT}

#### FILTERS

#BFF
spring.cloud.gateway.mvc.routes[0].id=api
spring.cloud.gateway.mvc.routes[0].uri=${backend-uri}
spring.cloud.gateway.mvc.routes[0].predicates[0]=Path=/api/**
spring.cloud.gateway.mvc.routes[0].filters[0].name=DedupeResponseHeader
spring.cloud.gateway.mvc.routes[0].filters[0].args[name]=Access-Control-Allow-Credentials Access-Control-Allow-Origin
spring.cloud.gateway.mvc.routes[0].filters[1].name=TokenRelay
#spring.cloud.gateway.mvc.routes[0].filters[2]=AddResponseHeader=Content-Type, application/json

#Back-channel logout
spring.cloud.gateway.mvc.routes[1].id=auth-route
spring.cloud.gateway.mvc.routes[1].uri=${scheme}://${hostname}:${bff-port}
spring.cloud.gateway.mvc.routes[1].predicates[0]=Path=/bff/**
#spring.cloud.gateway.mvc.routes[1].filters[0]=StripPrefix=1

#Frontend routes
spring.cloud.gateway.mvc.routes[2].id=frontend
spring.cloud.gateway.mvc.routes[2].uri=${frontend-uri}
spring.cloud.gateway.mvc.routes[2].predicates=Path=/**
# Forwarding support
#spring.cloud.gateway.mvc.http-client.type=jdk
#spring.cloud.gateway.mvc.http-client.connect-timeout=60s
#spring.cloud.gateway.mvc.http-client.read-timeout=60s
#spring.cloud.mvc.discovery.enabled=true

# Spring JDBC Session
spring.session.store-type=jdbc
server.servlet.session.timeout=600
spring.session.jdbc.initialize-schema=never
spring.session.jdbc.table-name=AUTH_SESSION
spring.datasource.url=jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?currentSchema=${DB_SCHEMA}
spring.datasource.username=${DB_USER}
spring.datasource.password=${DB_PASSWORD}
# Flyway
spring.flyway.enabled=true
spring.flyway.locations=classpath:db/migration
spring.flyway.table=flyway_spring_session
spring.flyway.baselineOnMigrate=true
spring.flyway.baselineVersion=0

#spring.cloud.gateway.httpserver.wiretap=true
#spring.cloud.gateway.httpclient.wiretap=true

Other than that I haven't customized anything besides the Oauth

Mendistern commented 3 weeks ago

I did manage to make it work with the apache http 5 module. Is this compatible with the gateway?

spencergibb commented 3 weeks ago

@Mendistern yes it is.

mendiCap commented 3 weeks ago

Alright! Thank you for your help

spencergibb commented 3 weeks ago

After discussion with the team, I've reverted the original change for the FileNotFoundException 7a41f6ad1bf3a3e01e6d86aeca905803326c84f4 and won't be adding any other workarounds for SimpleClientHttpRequestFactory. We can use this issue to document that SimpleClientHttpRequestFactory (and URLConnection which it uses) is not suitable for the WebMVC gateway server.

haeuserd commented 1 week ago

Okay, thank you very much for clarifying this matter.

I misunderstood the actual purpose and impact of spring.cloud.gateway.mvc.http-client.type=autodetect. I think with https://github.com/spring-cloud/spring-cloud-gateway/issues/3571 this gets much more intuitive :+1: