4.x: Invalid response payload returned when client requests 'Accept-Encoding: gzip' and server handler writes directly to response OutputStream

Environment Details

Helidon Version: 4.0.10
Helidon SE

Problem Description

An invalid response payload is returned by the server when the client requests 'Accept-Encoding: gzip' and the server handler writes the content directly to the response OutputStream. The following exception is received (as per the reproducer attached) when the client tries to decompress the response payload :-

java.io.EOFException: Unexpected end of ZLIB input stream at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:266) at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:175) at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:128) at java.base/java.io.InputStream.readNBytes(InputStream.java:412) at java.base/java.io.InputStream.readAllBytes(InputStream.java:349)

Calling outputStream.flush() immediately after outputStream.write() doesn't seem to help fix the problem. Although calling outputStream.close(), after all the content is written to the outputStream, solves the problem, it doesn't feel like a risk-free solution since it involves the server handler prematurely closing the OutputStream instead of allowing Helidon to close the OutputStream when it is ready. The OutputStream in this case is an instance of io.helidon.webserver.observe.tracing.TracingObserver.TracingStreamOutputDelegate, in which GZIPOutputStream is the delegate.

While the reproducer involves a simplified scenario, in the actual usecase it is not possible to set the CONTENT_LENGTH header, prior to writing to the OutputStream, since it requires buffering the content in memory and also the actual content length really depends on the data that is to be compressed. Even if it was possible to set the CONTENT_LENGTH, the GzipEncoding class used in ServerResponse#ouputStream() removes the header given that it is a streaming encoder. Since there is no CONTENT_LENGTH header in the response, chunked encoding is used to transfer the response. The understanding here is that the terminating chunk(s), which Helidon seems to be writing to the OutputStream before closing it out, could be causing problems when the client tries to decompress. This understanding also seems to explain why the problem is not seen when the server handler prematurely closes the OutputStream after writing the content.

Steps to reproduce

import io.helidon.webserver.WebServer;
import org.junit.jupiter.api.Test;

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.zip.GZIPInputStream;

import static org.junit.jupiter.api.Assertions.assertEquals;

public class ContentEncodingTest2 {
    @Test
    void invalidResponseIfContentWrittenUsingOutputStream() throws IOException {
        String strInResponse = "This is the content in the response";
        String content[]  = { "This is the ", "content in the response"};
        var server = WebServer.builder()
            .port(8080)
            .routing(it -> it.get("/hello", (req, res) -> {
                try {
//                    var outputStream = req.context().get(OutputStream.class).orElseThrow();
                    var outputStream = res.outputStream();
                    Arrays.stream(content).forEach(s -> {
                        try {
                            outputStream.write(s.getBytes(StandardCharsets.UTF_8));
                            outputStream.flush();
                        } catch (IOException e) {
                            throw new RuntimeException(e);
                        }
                    });
                } catch (Exception ex) {
                    throw ex;
                }
            }))
            .build();

        server.start();

        var url = new URL("http://localhost:8080/hello");
        var conn = (HttpURLConnection) url.openConnection();
        conn.setRequestProperty("Accept-Encoding", "gzip");
        assertEquals(200, conn.getResponseCode());
        var compressedInputStream = conn.getInputStream();
        var decompressedResponse = new String(new GZIPInputStream(compressedInputStream).readAllBytes());
        assertEquals(strInResponse, decompressedResponse);

        server.stop();
    }
}

helidon-io / helidon