search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.53k stars 2.61k forks source link

Default Jackson thread local pool uses Soft references: can we do better? #38860

Open franz1981 opened 6 months ago

franz1981 commented 6 months ago

Description

The default Jackson thread local pool is using https://github.com/FasterXML/jackson-core/blob/2.17/src/main/java/com/fasterxml/jackson/core/util/JsonRecyclerPools.java#L125 which, under the hood, uses https://github.com/FasterXML/jackson-core/blob/2.17/src/main/java/com/fasterxml/jackson/core/util/JsonRecyclerPools.java#L132 to reuse a BufferRecycler stored in a soft reference's ThreadLocal at https://github.com/FasterXML/jackson-core/blob/28fce9b4b7c48d2f70f244536ad284f1f52345ab/src/main/java/com/fasterxml/jackson/core/util/BufferRecyclers.java#L60

Despite it works great to avoid buffer leaks in dev mode or for the Quarkus blocking thread pool ThreadLocal(s), soft reference management can be very dangerous and unstable in the container world, where the (scarse) available MBs of heap can cause unexpected soft reference collections, causing new (big) jackson recyclers allocations, harming runtime performance and, again, the available memory bandwidth to the rest of application (instead of saving it).

An example of this is the json benchmark at Techempower, which, for some weird reason (test duration vs available MB in the heap), was triggering re-allocation of soft references and recycler, as by this flamegraph

image

FYI the soft reference collection policy works based on the default SoftRefLRUPolicyMSPerMB values as explained very well at https://www.michaelpollmeier.com/2019/04/09/understanding-jvm-soft-references-for-great-good-and-building-a-cache: have fun and be horrified by this diabolic behaviour.

In order to reproduce it, just setup a small enough max heap capacity eg 20 MB, occupy 19 MB of it (in JMH just retaining byte[]s for that), and leaving 1 MB available: the default value of SoftRefLRUPolicyMSPerMB will keep the soft reference of the recycler alive for 1 second per MB of available heap (which is 1, here), then it will free it up.

Implementation ideas

No response

quarkus-bot[bot] commented 6 months ago

/cc @geoand (jackson), @gsmet (jackson)

franz1981 commented 6 months ago

@mariofusco ^^

geoand commented 3 months ago

Is this still an issue?