Unable to Stream Chunked Response

[zirota]

Hello,

I am using spring cloud gateway mvc: 4.1.5 When making requests that expects a streaming chunked response - where each chunk is returned to the api caller as soon as a chunk is ready. The gateway returns the response in different batches, each with multiple chunks at the same time.

I found a similar issue raised here: https://github.com/spring-cloud/spring-cloud-gateway/issues/2868

I don't think its that streaming is not supported as curl is return the response from the gateway in batches - instead of the entire response being returned all at once. It could be that the gateway is somehow caching the response chunks. Is there configuration that can be done on the gateway to resolve it?

Sample I have tried a variety of different approaches. Including the examples in the linked issue.

My original approach using just handler functions:

defaultRoute = route("some-route")
                .route(path(this.targetApiPrefix), https(uri))
                .build();

Attempt to use of StreamingResponseBody with ProxyExchange

@PostMapping("/api/v1/chat")
public ResponseEntity<?> proxyChatApi(
        @RequestHeader(HttpHeaders.AUTHORIZATION) String authorizationHeader,
        @RequestBody Map body,
        ProxyExchange<?> proxyExchange) throws Exception {

    proxyExchange.header(HttpHeaders.AUTHORIZATION, authorizationHeader);
    proxyExchange.body(body);
    var response = proxyExchange.uri(this.uri).post();

    StreamingResponseBody stream = outputStream -> {
        StreamUtils.copy(response.getBody().getContentAsByteArray(), outputStream);
    };
    return new ResponseEntity(stream, response.getStatusCode());
}

Attempt to return a Flux<?> with ProxyExchange - I am not very familiar with the reactive way so pardon me if this seems incorrect:

@PostMapping("/api/v1/chat")
public Flux<?> proxyChatApi(
        @RequestHeader(HttpHeaders.AUTHORIZATION) String authorizationHeader,
        @RequestBody Map body,
        ProxyExchange<?> proxyExchange) throws Exception {
    var uri = new URI(this.uri);
    proxyExchange.header(HttpHeaders.AUTHORIZATION, authorizationHeader);
    proxyExchange.body(body);
    var response = proxyExchange.uri(uri.toString()).post();
    return Flux.from(response);
}

[zirota]

I found that the behaviour of the streaming response differs across different embedded servers.

The above behaviour comes from the default Tomcat server.

Jetty & Undertow buffers all chunked responses from the remote server and returns the entire response in full to the api caller.

However, can't seem to figure out how to disable or configure this buffer.

[iLaoke]

I have the same issue, use spring-cloud-gateway 4.0.7. The returned content does not exactly match the proxy's third-party interface, such as the proxy gemini interface