Some Netty features are not enabled by default

[franz1981]

Description

Currently there are (after the introduction of Java modules), few Netty features, which are not enabled because the "right" JVM args are not present by default on quarkus applications:

1. allocating direct buffer(s) without Cleaners: this is used on off-heap pooled arena(s) backed NIO ByteBuffers. It can be enabled adding -Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.nio=ALL-UNNAMED to the JVM args.
2. allocating "big enough" (> 1024 by default) byte[] without zeroing them, using the "hidden" feature explained at https://shipilev.net/jvm/anatomy-quarks/7-initialization-costs/#_experiment. It can be enabled via -Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED

I'm not proposing to add both, but I think is important to be aware that these features are not available in the current default configuration.

The first problem (related the cleaner option) is pretty important, because:

• without configurations as it is now: it implies that the practical (direct) memory limits are the ones specified by Java by default, which means == the heap size (it means that a container should have *2max Java heap size** available TOTAL memory: max java heap capacity + direct memory limits). Additionally, the JVM is responsible to expose via its metrics (yeeeee!) these direct allocations, and will take care of issuing OutOfDirectMemory (which is not "yay", the opposite: see https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/java.base/share/classes/java/nio/Bits.java#L145-L178 how ugly can go!)
• with the proposed configuration: the direct memory limits are still (if nothing is configured, see https://github.com/netty/netty/blob/b0e43847a8c78325ea7dd8bb02934ed6c690cb9c/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L151-L158) the same as the max Java heap BUT the JDK NIO direct ByteBuffers allocations are tracked separately by OpenJDK, while the Netty direct allocations would use Unsafe::allocateMemory, instead, and its limit are enforced by Netty, instead. This means that the TOTAL memory required (in theory) to a container is: *3max java heap**(!!!! ie heap + direct NIO memory limit + Netty direct memory limit). Additionally, there won't be any JVM metrics (maybe just native memory tracking AFAIK), which can report the total number of allocated direct memory (including the one from Netty, via Unsafe). Thanks to @alesj we can still use the Netty memory metrics, anyway, for this purpose.

So, why this second option could be appealing?

1. the allocated NIO ByteBuffers won't contain any Cleaner saving some memory
2. the default heuristic for the platform, while going OOM, is not great (see https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/java.base/share/classes/java/nio/Bits.java#L145-L178) and can cause an application to become unresponsive

Implementation ideas

Configured the proposed option by default

[franz1981]

@zakkak @cescoffier

[cescoffier]

Hum, we cannot really recommend using --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED.

[zakkak]

I agree we should only resolve to opening core modules only if strictly necessary.

[franz1981]

To me is fine to:

• -Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.nio=ALL-UNNAMED and see if it bring evident benefits (on native image)
• just don't do anything and just remember about this note: in the past Wildfly people didn't have such note and even sizing memory on Netty (which is part of this) has been my "most answered question" of the last 4 years, so, still worthy to be aware of it

[franz1981]

The issue related zeroing of heap buffers (i.e. byte[]) can be verified (in native image as well) by enabling debug/trace logging and search for these log messages: https://github.com/netty/netty/blob/afb475dabd5ca9eaa94ccf07f0ecd8431db008ba/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L507-L517

@zakkak

[cescoffier]

We would need alternatives to enable these options without requiring an add-open.

[franz1981]

The alternative for the one impacting SSL is already in; by having SSL heap buffer pooling we don't need to care about allocate uninitialized arrays, cause we won't allocate them anymore 😁 For the cleaner one, I have to think 🤔