OutOfMemoryError when returning a large InputStream which is consumed slowly

Toseban commented 3 months ago

Describe the bug

When returning an InputStream with a size larger than the configured -XX:MaxDirectMemorySize, there is an OutOfMemoryError when the client does not consume the content fast enough.

In VertxOutputStream the following code just sends all data to netty, which then buffers it in memory:

 private boolean awaitWriteable() throws IOException {
        if (Vertx.currentContext() == ((HttpServerRequestInternal) request).context()) {
            return false; // we are on the (right) event loop, so we can write - Netty will do the right thing.
        }
        [...]

Expected behavior

If a client is consuming data slowly, it should not be loaded into memory fully and no OutOfMemoryError should occur

Actual behavior

2024-07-03 09:27:14,692 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /hello failed, error id: 3639e8fd-4281-47b0-82c7-43a46485cf5a-1: java.lang.RuntimeException: java.lang.OutOfMemoryError: Cannot reserve 65536 bytes of direct buffer memory (allocated: 104796533, limit: 104857600)
    at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:255)
    at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:185)
    at org.jboss.resteasy.reactive.server.core.serialization.FixedEntityWriterArray.write(FixedEntityWriterArray.java:31)
    at org.jboss.resteasy.reactive.server.handlers.ResponseWriterHandler.handle(ResponseWriterHandler.java:34)
    at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:147)
    at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
    at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:599)
    at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
    at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
    at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.OutOfMemoryError: Cannot reserve 65536 bytes of direct buffer memory (allocated: 104796533, limit: 104857600)
    at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
    at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
    at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
    at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:705)
    at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:680)
    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:212)
    at io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:177)
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:134)
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
    at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:397)
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
    at io.quarkus.vertx.utils.AppendBuffer.append(AppendBuffer.java:136)
    at io.quarkus.vertx.utils.VertxOutputStream.write(VertxOutputStream.java:178)
    at java.base/java.io.InputStream.transferTo(InputStream.java:797)
    at org.jboss.resteasy.reactive.common.providers.serialisers.InputStreamMessageBodyHandler.writeTo(InputStreamMessageBodyHandler.java:38)
    at org.jboss.resteasy.reactive.server.providers.serialisers.ServerInputStreamMessageBodyHandler.writeResponse(ServerInputStreamMessageBodyHandler.java:46)
    at org.jboss.resteasy.reactive.server.providers.serialisers.ServerInputStreamMessageBodyHandler.writeResponse(ServerInputStreamMessageBodyHandler.java:17)
    at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:217)
    ... 13 more

How to Reproduce?

Run the QuarkusTest in the attached example project with -XX:MaxDirectMemorySize=100m quarkus-oom.zip

Output of `uname -a` or `ver`

Microsoft Windows [Version 10.0.19045.4529]

Output of `java -version`

openjdk version "21.0.1" 2023-10-17 OpenJDK Runtime Environment (build 21.0.1+12-29) OpenJDK 64-Bit Server VM (build 21.0.1+12-29, mixed mode, sharing)

Quarkus version or git rev

3.12.0

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) Maven home: C:\Users\redacted\Java\apache-maven-3.9.6-bin\apache-maven-3.9.6 Java version: 21.0.1, vendor: Oracle Corporation, runtime: C:\Users\redacted.jdks\openjdk-21.0.1 Default locale: en_US, platform encoding: UTF-8 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

Additional information

The issue occurs both on Windows and on Linux

quarkus-bot[bot] commented 3 months ago

/cc @FroMage (resteasy-reactive), @stuartwdouglas (resteasy-reactive)

geoand commented 3 months ago

@franz1981 this will interest you

franz1981 commented 3 months ago

I will take a look later today but...

When returning an InputStream with a size larger than the configured -XX:MaxDirectMemorySize

The max direct memory size of Netty, if not configured, uses the one configured in the JVM args and the pooled allocator too has some limits of what is going to be pooled i.e. big buffers are not pooled but allocated and removed on the flight. So, in general, is a red flag to allocate beyond the configured direct memory capacity. If the allocated buffers are used from a Netty thread, there is no way to block it, so we can just keep on accumulate it and let Netty to provide some feedback that we are overallocating (suspending reading from vertx, at connection level), but that means that if the overallocating is happening from a different connection, we always risk to react too late and the only option is to close forcibly the connection while throwing an exception back to the user code. Wdyt @geoand ?

Toseban commented 3 months ago

Just as an additional note, when we saw this happen in our application we did not have an explicit -XX:MaxDirectMemorySize configured (it defaulted to 1GB in that scenario)

franz1981 commented 3 months ago

we did not have an explicit -XX:MaxDirectMemorySize configured

in that case by default Netty uses the max heap size (configured or implied) as max direct memory limit.

quarkusio / quarkus